Profile cover
B

AI Voice Agent Evaluation Lead

Brandon Kim

Full-time · Job · Los_Angeles

About the role

Voice AI agents handle 3 million customer service calls per month for our clients. But how do you measure whether a voice agent is actually good? Call completion rate misses the nuance — a call that technically resolves but leaves the customer frustrated is a failure. Satisfaction surveys capture sentiment but not capability gaps. You'll design the evaluation framework that measures voice agent quality across every dimension that matters: task completion accuracy, conversational naturalness, error recovery gracefully, emotional tone appropriateness, and the ability to know when to escalate to a human. You'll build automated evaluation pipelines, design A/B testing frameworks for agent improvements, and set the quality standards that all agent deployments must meet. This is a leadership role in evaluation methodology. You'll define what 'good' means for voice AI — and your definitions will determine which improvements ship to production and which get sent back for iteration.

Responsibilities

  • Design multi-dimensional evaluation frameworks for voice AI agent quality
  • Build automated evaluation pipelines scoring conversational performance at scale
  • Create A/B testing frameworks for voice agent improvement measurement
  • Set quality standards and deployment gates for agent releases
  • Lead evaluation methodology development across the voice AI team

Requirements

  • 5+ years in conversational AI with evaluation or quality assurance focus
  • Deep understanding of dialogue systems and voice interaction design
  • Strong statistical skills for experiment design and metric validation
  • Experience with LLM-based agents and prompt engineering
  • Leadership experience setting quality standards for AI products

Benefits

  • Fully remote — US West Coast hours
  • Senior compensation with equity
  • $3,500 conference budget
  • Defining quality standards for millions of AI conversations

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

NLP & Prompt Engineering AI Product & Strategy

Skills

python openai api langchain statistics sql

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Technical Writer

RemoteAI company

Part-time

Product Manager, Developer Experience

RemoteAI company

Full-time

Software Engineer, Full Stack - UK

RemoteAI company

Full-time

Software Engineer, Applications (Scientific Development Solutions) (High Seniority)

RemoteAI company

Full-time Senior

PhD Autonomy Engineer Intern - Computer Vision / Deep Learning Summer 2027

RemoteAI company

Full-time Intern

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

Job Type

Full-time

Level

Senior

Language

English

Salary Range

$160k – $200k / year

AI Expertise

NLP & Prompt Engineering AI Product & Strategy

Skills

python openai api langchain statistics sql pandas

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile