AI Retrieval-Augmented Generation Quality Engineer

About the role

Responsibilities

• Build automated test suites for retrieval quality, hallucination detection, and attribution accuracy
• Design per-deployment quality benchmarks from representative query samples
• Implement regression testing that catches quality degradation after model or index updates
• Create quality dashboards per deployment with configurable alert thresholds
• Develop synthetic evaluation datasets for knowledge base coverage testing
• Collaborate with LLM engineers on quality-driven architecture improvements

Requirements

• 3+ years in NLP engineering with RAG or information retrieval system experience
• Experience designing evaluation frameworks for generative AI systems
• Familiarity with LangChain, LlamaIndex, or equivalent RAG frameworks
• Strong Python skills and analytical mindset for quality metric design
• Understanding of hallucination patterns and retrieval failure modes in LLMs

Benefits

• Fully remote
• Health, dental, and vision
• $3,000 professional development budget
• Equity participation
• Quarterly team retreats

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.