Profile cover
O

AI Retrieval-Augmented Generation Quality Engineer

Emrex AI

Remote · $100k – $130k / year · Full-time · Posted 4 months ago

No longer accepting applications

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

RAG systems fail quietly. The retrieval returns plausible-looking chunks that don't actually answer the question. The generation produces confident text that contradicts the retrieved context. The attribution cites a document that says the opposite of the generated claim. None of these failures trigger an error — they just produce wrong answers that users might trust. We build LLM infrastructure for enterprise clients. Our RAG quality engineering practice is what separates our system reliability from off-the-shelf RAG demos. You'll own the quality evaluation framework: building automated test suites that detect retrieval failures, hallucination patterns, and attribution errors before they reach users. The system processes 4 million queries per day across 28 enterprise deployments. Each has different knowledge bases, different user populations, and different accuracy requirements. Your evaluation framework needs to be configurable per deployment while maintaining consistent quality standards across all of them.

Fully remote.

Responsibilities

  • Build automated test suites for retrieval quality, hallucination detection, and attribution accuracy
  • Design per-deployment quality benchmarks from representative query samples
  • Implement regression testing that catches quality degradation after model or index updates
  • Create quality dashboards per deployment with configurable alert thresholds
  • Develop synthetic evaluation datasets for knowledge base coverage testing
  • Collaborate with LLM engineers on quality-driven architecture improvements

Requirements

  • 3+ years in NLP engineering with RAG or information retrieval system experience
  • Experience designing evaluation frameworks for generative AI systems
  • Familiarity with LangChain, LlamaIndex, or equivalent RAG frameworks
  • Strong Python skills and analytical mindset for quality metric design
  • Understanding of hallucination patterns and retrieval failure modes in LLMs

Benefits

  • Fully remote
  • Health, dental, and vision
  • $3,000 professional development budget
  • Equity participation
  • Quarterly team retreats

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

NLP & Prompt Engineering MLOps & AI Infrastructure

Skills

python rag langchain hugging face fastapi

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Materials Lead

Lambda

Full-time Lead

Sr. Solutions Architect - Travel and Hospitality

Databricks

Full-time Senior

Revenue Operations Manager

Handshake

Full-time

Senior Security Engineer - Detection & Response

Lambda

Full-time Senior

Lead/ Senior Machine Learning Scientist, Integrative Phenotyping

insitro

Full-time Lead

AWS Alliances Lead

LangChain

Full-time Lead

Job Type

Full-time

Level

Mid-Level

Language

English

Salary Range

$100k – $130k / year

AI Expertise

NLP & Prompt Engineering MLOps & AI Infrastructure

Skills

python rag langchain hugging face fastapi sql

Ready to apply for this role?

Create a free talent account.

  • Find AI roles in one place
  • Get AI-matched job recommendations
  • Reply to companies that message you
  • Build your public AI talent profile
Create free account & apply Log in

This role is closed. Create your free profile to apply to similar open roles.

Don't miss the next one: get matched to new roles like this.

Create free profile