Technical Program Manager, AI Data Quality

CLICK HERE TO APPLY

Job Description 

OverviewAs an AI Data Quality Technical Program Manager, you will work at the nexus of data creation, AI model development, and third-party partners. You will be responsible for ensuring that our research data needs are being met and that our data creation vendors are providing high-quality, high-value and compliant data to support our research priorities.  
We are seeking a highly analytical and detail-oriented AI training data quality assurance (QA) specialist with a strong background in coding STEM, phenomenal attention to detail, and program management experience to ensure the quality and integrity of data creation and annotation work performed by Microsoft partners. This individual will play a key role in defining data collection requirements, reviewing and validating complex annotations, ensuring alignment with project guidelines, building instructions and guidelines to annotators, actively enhancing coordination with vendors, giving feedback to vendors, and supporting the continuous improvement of data quality processes that result in improved model training and evaluation. This role will involve deep interaction with senior AI scientists across Microsoft AI. 
Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond. 
By applying to this role based in U.S. Mountain View, CA. You are required to be local to the San Francisco area and be in office 3 days a week.  
Responsibilities

• Help drive overall data creation and vendor management strategy. 
• Perform quality assurance checks on vendor-provided coding, STEM, RLHF, and other data collection types. 
• Collaborate with internal teams and vendors to define quality metrics and resolve discrepancies. 
• Develop and maintain data annotation instructions, guidelines and templates.  
• Provide detailed feedback and coaching to vendors to uphold high-quality standards. 
• Identify systemic quality issues and drive root-cause analysis and resolution plans. 
• Support program management of annotation workflows, timelines, and priorities. 
• Monitor progress and help track key performance indicators (KPIs) related to QA. 
• Participate in designing and improving annotation tooling and platform for QA features.  
• Help ensure the safe, compliant, responsible, and agile use of data to support our research priorities. 
• Help other Microsoft AI building teams access data for training as needed. Embody our Culture and Values.  

Qualifications
Required Qualifications: 

• Bachelor’s Degree in Computer Science, Mathematics, Engineering, or a related technical field AND at least 6 years' experience in STEM and Code data creation efforts that involve complex QA and program management
• OR Master’s Degree in Computer Science, Mathematics, Engineering, or a related technical field AND at least 4 years' experience in STEM and Code data creation efforts that involve complex QA and program management, or equivalent experience 
• OR equivalent experience 
• 4 years of experience in leading multi-disciplinary projects, defining requirements, developing project plans and working with technical teams to execute them 
• Excellent attention to detail and critical thinking skills 
• Solid understanding of programming concepts, data structures, and algorithms. 
• Experience in managing and goal tracking data collection or other equivalent programs  
• Experience in coding, or technical program management. 
• Strong LLM practitioner, who frequently uses LLMs and understands how training data is used to improve them 
• Solid communication and collaboration skills, with the ability to give clear, constructive feedback. 

Preferred Qualifications 

• PhD. in a relevant field AND at least 6 years' experience in STEM and Code data creation efforts that involve complex QA and program management, or equivalent experience. 
• 6 years' experience managing cross-functional and/or cross-team projects. 
• Familiarity with Python, SQL, JS or other programming languages used in data annotation. 
• Background in educational content, coding assessments, or STEM domains. 
• Experience working with data labeling platforms and quality monitoring tools. 

Technical Program Management IC5 - The typical base pay range for this role across the U.S. is USD $137,600 - $267,000 per year. There is a different range applicable to specific work locations, within the San Francisco Bay area and New York City metropolitan area, and the base pay range for this role in those locations is USD $180,400 - $294,000 per year. Certain roles may be eligible for benefits and other compensation. Find additional benefits and pay information here: https://careers.microsoft.com/us/en/us-corporate-pay    Microsoft will accept applications and processes offers for these roles on an ongoing basis.
#MicrosoftAI #copilot

CLICK HERE TO APPLY