AI Long-Context Faithfulness Evaluation Developer

About the role

Responsibilities

• Design long-context faithfulness evaluation tasks covering unsupported claims, wrong attribution, and source contradiction
• Build automated faithfulness judge using LLM-based evaluation calibrated against human faithfulness assessments
• Implement context length ablation evaluation measuring how faithfulness degrades as context grows
• Develop document type sensitivity analysis identifying which source types trigger most faithfulness failures
• Create faithfulness evaluation dataset curation pipeline ensuring diverse failure type coverage
• Build evaluation analysis dashboards supporting research team interpretation of faithfulness benchmark results

Requirements

• 3+ years in NLP engineering with LLM evaluation, benchmark design, or faithfulness and hallucination research
• Strong Python skills with LLM evaluation pipeline development and statistical analysis
• Understanding of faithfulness evaluation methodology, attribution assessment, and NLI-based evaluation approaches
• Experience building automated evaluation systems calibrated against human judgment
• Ability to design evaluation datasets that provide persistent signal as model capabilities improve

Benefits

• Fully remote with flexible schedule
• Health and dental insurance
• $3,000 annual professional development budget
• 401(k) with match
• Equity options

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.