AI Dialogue Quality Scoring Engine Developer

Mevra Labs

Remote · $80k – $100k / year · Full-time · Posted 4 months ago

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

Our conversational AI platform powers customer service bots for 95 enterprise clients. Every bot has tens of thousands of conversations per week. How do you know which conversations went well? Right now, clients rely on CSAT surveys that get responses from 3% of users and human QA that reviews 1% of conversations. That means 96%+ of conversations are evaluated by no one. You'll build the automated quality scoring engine that evaluates every conversation: task completion, response appropriateness, tone consistency with brand guidelines, factual accuracy against the knowledge base, and escalation appropriateness. The models need to agree with human QA reviewers at least 85% of the time to be trusted — which requires careful calibration and a feedback loop that learns from disagreements. The NLP team is four engineers. You'll own the quality scoring system end-to-end: model design, training pipeline, calibration framework, and the scoring API consumed by client analytics dashboards. When a bot starts producing lower-quality responses after a knowledge base update, your system catches it — before the client's customers do.

Fully remote.

Responsibilities

Build multi-dimensional conversation quality scoring across task completion and tone dimensions
Design calibration framework aligning automated scores with human QA reviewer judgments
Implement factual accuracy checking against client knowledge base content
Create feedback loop incorporating QA reviewer disagreements into model improvement
Build scoring API serving quality metrics to client analytics dashboards in real time
Monitor score distribution drift as bot knowledge bases and conversation patterns evolve

Requirements

3+ years in NLP engineering with dialogue or conversation analysis experience
Experience building evaluation or quality scoring systems for text or dialogue
Strong Python and PyTorch skills with HuggingFace familiarity
Understanding of calibration techniques for ML classifiers
Ability to build systems that earn trust through demonstrated alignment with human judgment