Profile cover
S

AI Evaluator Agreement Calibration Engineer

Sofia Garcia

Full-time · Job · New_York

About the role

Every time we add a new LLM-as-judge to our evaluation stack, its scores disagree with our existing judges just enough to make trend lines meaningless. We publish our benchmark findings internally and expect engineers here to read papers on evaluator calibration and implement what actually works, not just what sounds good in a blog post. In this role you'll build the calibration layer that aligns judge models against human-rated gold sets, so a capability score means the same thing whether it came from judge A or judge C, and report findings back to the evaluation team in a form they can act on.

Responsibilities

  • Build calibration pipelines aligning LLM judges against human gold sets
  • Maintain a growing library of test cases covering judge disagreement patterns
  • Investigate and document root causes of evaluator score drift
  • Share findings on evaluator calibration with the wider research team

Requirements

  • 2+ years working hands-on with LLM evaluation or benchmarking
  • Comfortable reading research on evaluation methodology and applying it
  • Experience with LangChain or similar orchestration frameworks

Benefits

  • Fully remote with flexible hours
  • Internal research time allocated each month
  • Health, dental, and vision coverage

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

NLP & Prompt Engineering

Skills

python langchain prompt engineering openai api

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Software Engineer, Full Stack - UK

RemoteAI company

Full-time

Team Lead, Actors

RemoteAI company

Full-time Lead

PhD Autonomy Engineer Intern - Computer Vision / Deep Learning Summer 2027

RemoteAI company

Full-time Intern

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Senior AI R&D Engineer – Model Training (f/m/d)

RemoteAI company

Full-time Senior

Senior AI Researcher- Pre-training (f/m/d)

RemoteAI company

Full-time Senior

Job Type

Full-time

Level

Mid-Level

Language

English

Salary Range

$100k – $130k / year

AI Expertise

NLP & Prompt Engineering

Skills

python langchain prompt engineering openai api

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile