Profile cover
H

AI Dialogue Quality Scoring Engine Developer

Henry Eriksson

Full-time · Job · Berlin

About the role

Our conversational AI platform powers customer service bots for 95 enterprise clients. Every bot has tens of thousands of conversations per week. How do you know which conversations went well? Right now, clients rely on CSAT surveys that get responses from 3% of users and human QA that reviews 1% of conversations. That means 96%+ of conversations are evaluated by no one. You'll build the automated quality scoring engine that evaluates every conversation: task completion, response appropriateness, tone consistency with brand guidelines, factual accuracy against the knowledge base, and escalation appropriateness. The models need to agree with human QA reviewers at least 85% of the time to be trusted — which requires careful calibration and a feedback loop that learns from disagreements. The NLP team is four engineers. You'll own the quality scoring system end-to-end: model design, training pipeline, calibration framework, and the scoring API consumed by client analytics dashboards. When a bot starts producing lower-quality responses after a knowledge base update, your system catches it — before the client's customers do.

Responsibilities

  • Build multi-dimensional conversation quality scoring across task completion and tone dimensions
  • Design calibration framework aligning automated scores with human QA reviewer judgments
  • Implement factual accuracy checking against client knowledge base content
  • Create feedback loop incorporating QA reviewer disagreements into model improvement
  • Build scoring API serving quality metrics to client analytics dashboards in real time
  • Monitor score distribution drift as bot knowledge bases and conversation patterns evolve

Requirements

  • 3+ years in NLP engineering with dialogue or conversation analysis experience
  • Experience building evaluation or quality scoring systems for text or dialogue
  • Strong Python and PyTorch skills with HuggingFace familiarity
  • Understanding of calibration techniques for ML classifiers
  • Ability to build systems that earn trust through demonstrated alignment with human judgment

Benefits

  • Fully remote across EU time zones
  • 30 days PTO plus local holidays
  • Private health insurance
  • €2,000 annual learning budget
  • Equity participation

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

NLP & Prompt Engineering AI & Machine Learning Engineers

Skills

python pytorch hugging face langchain sql

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior AI Researcher- Pre-training (f/m/d)

RemoteAI company

Full-time Senior

Technical Writer

RemoteAI company

Part-time

AI Resident UK/EMEA

RemoteAI company

Full-time

Senior AI R&D Engineer – Model Training (f/m/d)

RemoteAI company

Full-time Senior

Senior Machine Learning Engineer (Clinical Team)

RemoteAI company

Full-time Senior

Conversational Modelling Research Engineer

RemoteAI company

Full-time

Job Type

Full-time

Level

Mid-Level

Language

English

Salary Range

$80k – $100k / year

AI Expertise

NLP & Prompt Engineering AI & Machine Learning Engineers

Skills

python pytorch hugging face langchain sql fastapi

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile