Junior AI Search Evaluation Analyst

About the role

Responsibilities

• Design relevance rating guidelines for new query categories and domains
• Manage rater recruitment, training, and quality calibration
• Build annotation tooling for efficient and consistent relevance judgment
• Run inter-annotator agreement analysis and flag quality degradation
• Analyze rating data to generate relevance signals for search model evaluation
• Report evaluation results to the search engineering team with statistical context

Requirements

• 1+ year of experience in data analysis, quality assurance, or research
• Strong attention to detail and comfort with ambiguous quality judgments
• Basic Python and SQL skills for data manipulation and analysis
• Good written communication for designing clear rating guidelines
• Interest in search systems, NLP, or information retrieval

Benefits

• Fully remote with flexible hours
• Health and dental insurance
• $1,500 annual learning budget
• Structured mentorship from senior search engineers
• Clear progression path

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.