Profile cover
S

AI Dialogue Evaluation Specialist

Scarlett Wang

Full-time · Job · Berlin

About the role

Chatbots are easy to build and hard to evaluate. Does the response actually answer the question? Is the tone appropriate? Did it hallucinate facts? Our evaluation platform scores conversational AI quality across 14 dimensions — and we need someone to maintain the evaluation datasets, run quality audits, and improve the scoring rubrics. As a dialogue evaluation specialist, you'll review thousands of chatbot conversations weekly, scoring them against our quality framework. You'll identify failure patterns — the types of questions that consistently produce bad answers — and build test datasets that specifically target those weaknesses. This role teaches you how production AI systems fail. After a year of studying chatbot failures systematically, you'll understand conversational AI quality better than most engineers who build them. It's meticulous, pattern-driven work that directly improves the products our clients ship.

Responsibilities

  • Score chatbot conversations across 14 quality dimensions
  • Identify systematic failure patterns in conversational AI responses
  • Build targeted test datasets for known weakness areas
  • Maintain and improve evaluation rubrics for dialogue quality
  • Generate weekly quality reports with failure trend analysis

Requirements

  • Python proficiency for data analysis and scripting
  • Strong reading comprehension and analytical attention to detail
  • Basic understanding of LLMs and chatbot systems
  • SQL skills for querying conversation databases
  • Interest in conversational AI quality and evaluation methodology

Benefits

  • Fully remote — European time zones
  • 28 days PTO
  • Mentorship from senior NLP engineers
  • $1,500 learning budget

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

NLP & Prompt Engineering

Skills

python openai api pandas sql prompt engineering

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Software Engineer, Full Stack - UK

RemoteAI company

Full-time

Technical Writer

RemoteAI company

Part-time

Software Engineer, Applications (Scientific Development Solutions) (High Seniority)

RemoteAI company

Full-time Senior

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

PhD Autonomy Engineer Intern - Computer Vision / Deep Learning Summer 2027

RemoteAI company

Full-time Intern

Backend Engineer, Platform

RemoteAI company

Full-time

Job Type

Full-time

Level

Junior

Language

English

Salary Range

$60k – $80k / year

AI Expertise

NLP & Prompt Engineering

Skills

python openai api pandas sql prompt engineering

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile