Profile cover
S

AI Conversation Quality Scoring Engine Developer

Scarlett Wang

Full-time · Job · New_York

About the role

Chatbot conversations are easy to generate and hard to evaluate. BLEU scores and perplexity tell you nothing about whether the bot actually solved the user's problem, maintained a coherent persona, or gave accurate information. Our quality scoring engine evaluates every chatbot conversation across 8 dimensions — task completion, factual accuracy, tone consistency, response relevance, escalation appropriateness, and three more — producing scores that correlate with human quality judgments at 0.89 Spearman rho. You'll own the scoring engine: designing new quality dimensions as our clients' needs evolve, calibrating scorers against human judgment panels, building the real-time scoring pipeline that evaluates conversations as they happen, and creating the analytics layer that surfaces quality trends to product teams. Every client chatbot generates a quality score distribution that their team monitors like a North Star metric. The conversational AI team is seven engineers. You'll be the senior engineer who defines what "quality" means and builds the system that measures it. Your scores determine whether new chatbot versions ship or get rejected — that's significant responsibility.

Responsibilities

  • Design and calibrate multi-dimensional conversation quality scoring
  • Build real-time scoring pipeline evaluating conversations as they occur
  • Maintain 0.89+ correlation between automated scores and human judgments
  • Create analytics surfacing quality trends for product teams
  • Define new quality dimensions as client needs evolve
  • Set quality thresholds that gate chatbot version releases

Requirements

  • 6+ years in NLP with conversation evaluation or dialogue quality focus
  • Experience building evaluation systems that correlate with human judgment
  • Strong PyTorch and Hugging Face skills for scorer development
  • Understanding of dialogue quality dimensions beyond simple metrics
  • Track record of shipping evaluation systems used in production decisions

Benefits

  • Fully remote with async-first culture
  • Top-tier compensation plus equity
  • Comprehensive health coverage
  • $5,000 annual conference and research budget
  • Define quality standards for enterprise chatbots

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

NLP & Prompt Engineering AI & Machine Learning Engineers

Skills

python pytorch hugging face sql docker

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior AI Researcher- Pre-training (f/m/d)

RemoteAI company

Full-time Senior

Senior Product Security Engineer

RemoteAI company

Full-time Senior

AI Resident UK/EMEA

RemoteAI company

Full-time

Senior AI R&D Engineer – Model Training (f/m/d)

RemoteAI company

Full-time Senior

Senior Machine Learning Engineer (Clinical Team)

RemoteAI company

Full-time Senior

Conversational Modelling Research Engineer

RemoteAI company

Full-time

Job Type

Full-time

Level

Senior

Language

English

Salary Range

$160k – $200k / year

AI Expertise

NLP & Prompt Engineering AI & Machine Learning Engineers

Skills

python pytorch hugging face sql docker fastapi

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile