Amazon Quick Suite is an enterprise AI platform that transforms how organizations work with their data and knowledge, combining generative AI-powered search, deep research capabilities, intelligent agents and automations, and comprehensive business intelligence to help teams make faster, data-driven decisions while maintaining enterprise-grade security and governance.
La información a continuación detalla los requisitos del puesto, la experiencia esperada del candidato y las cualificaciones correspondientes.
We are seeking a Data Scientist II to join our Quick Data team, focusing on evaluation and benchmarking data development for Quick Suite features, with particular emphasis on Research and other generative AI capabilities. Our mission is to engineer high-quality datasets that are essential to the success of Amazon Quick Suite. From human evaluations and Responsible AI safeguards to Retrieval-Augmented Generation and beyond, our work ensures that Generative AI is enterprise-ready, safe, and effective for users at scale.
Key job responsibilities
* Design and develop comprehensive evaluation and benchmarking datasets for Quick Suite AI‐powered features.
* Leverage LLMs for synthetic data corpora generation; conduct data evaluation and quality assessment using LLM‐as‐a‐judge settings.
* Create ground truth datasets with high‐quality question‐answer pairs across diverse domains and use cases.
* Lead human annotation initiatives and model evaluation audits to ensure data quality and relevance.
* Develop and refine annotation guidelines and quality frameworks for evaluation tasks.
* Conduct statistical analysis to measure model performance, identify failure patterns, and guide improvement strategies.
* Collaborate with ML scientists and engineers to translate evaluation insights into actionable product improvements.
* Build scalable data pipelines and tools to support continuous evaluation and benchmarking efforts.
* Contribute to Responsible AI initiatives by developing safety and fairness evaluation datasets.
Basic Qualifications
* 2+ years of data scientist experience.
* 3+ years of data querying languages (e.g., SQL), scripting languages (e.g., Python) or statistical/mathematical software (e.g., R, SAS, Matlab) experience.
* 3+ years of machine learning/statistical modeling data analysis tools and techniques experience, including parameters that affect performance.
* 1+ year of working with or evaluating AI systems.
* 1+ year of creating or contributing to mathematical textbooks, research papers, or educational content.
* Master's degree in Science, Technology, Engineering, or Mathematics (STEM), or relevant STEM experience.
* Experience applying theoretical models in an applied environment.
Preferred Qualifications
* Ph.D. in Science, Technology, Engineering, or Mathematics (STEM).
* Knowledge of machine learning concepts and their application to reasoning and problem‐solving.
* Experience in a ML or data scientist role with a large technology company.
* Experience defining and creating benchmarks for assessing GenAI model performance.
* Experience working on multi‐team, cross‐disciplinary projects.
* Experience applying quantitative analysis to solve business problems and make data‐driven business decisions.
* Experience effectively communicating complex concepts through written and verbal communication.
Amazon is an equal opportunity employer and does not discriminate on the basis of protected veteran status, disability, or other legally protected status.
Los Angeles County applicants: Job duties for this position include working safely and cooperatively with other employees, supervisors, and staff; adhering to standards of excellence despite stressful conditions; communicating effectively and respectfully with employees, supervisors, and staff to ensure exceptional customer service; and following all federal, state, and local laws and Company policies. xohynlm Criminal history may have a direct, adverse, and negative relationship with some of the material job duties of this position.