Profile cover
S

AI Dialogue Evaluation Specialist

Brenvo Tech

Remote · $60k – $80k / year · Full-time · Posted 5 months ago

No longer accepting applications

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

Chatbots are easy to build and hard to evaluate. Does the response actually answer the question? Is the tone appropriate? Did it hallucinate facts? Our evaluation platform scores conversational AI quality across 14 dimensions — and we need someone to maintain the evaluation datasets, run quality audits, and improve the scoring rubrics. As a dialogue evaluation specialist, you'll review thousands of chatbot conversations weekly, scoring them against our quality framework. You'll identify failure patterns — the types of questions that consistently produce bad answers — and build test datasets that specifically target those weaknesses. This role teaches you how production AI systems fail. After a year of studying chatbot failures systematically, you'll understand conversational AI quality better than most engineers who build them. It's meticulous, pattern-driven work that directly improves the products our clients ship.

Fully remote.

Responsibilities

  • Score chatbot conversations across 14 quality dimensions
  • Identify systematic failure patterns in conversational AI responses
  • Build targeted test datasets for known weakness areas
  • Maintain and improve evaluation rubrics for dialogue quality
  • Generate weekly quality reports with failure trend analysis

Requirements

  • Python proficiency for data analysis and scripting
  • Strong reading comprehension and analytical attention to detail
  • Basic understanding of LLMs and chatbot systems
  • SQL skills for querying conversation databases
  • Interest in conversational AI quality and evaluation methodology

Benefits

  • Fully remote — European time zones
  • 28 days PTO
  • Mentorship from senior NLP engineers
  • $1,500 learning budget

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

NLP & Prompt Engineering

Skills

python openai api pandas sql prompt engineering

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Materials Lead

Lambda

Full-time Lead

Sr. Solutions Architect - Travel and Hospitality

Databricks

Full-time Senior

Revenue Operations Manager

Handshake

Full-time

Senior Security Engineer - Detection & Response

Lambda

Full-time Senior

Lead/ Senior Machine Learning Scientist, Integrative Phenotyping

insitro

Full-time Lead

Solutions Engineer, Canada

Glean

Full-time

Job Type

Full-time

Level

Junior

Language

English

Salary Range

$60k – $80k / year

AI Expertise

NLP & Prompt Engineering

Skills

python openai api pandas sql prompt engineering

Ready to apply for this role?

Create a free talent account.

  • Find AI roles in one place
  • Get AI-matched job recommendations
  • Reply to companies that message you
  • Build your public AI talent profile
Create free account & apply Log in

This role is closed. Create your free profile to apply to similar open roles.

Don't miss the next one: get matched to new roles like this.

Create free profile