Profile cover
S

AI Evaluator Agreement Calibration Engineer

Zenora Labs

Remote · $100k – $130k / year · Full-time · Posted 3 months ago

No longer accepting applications

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

Every time we add a new LLM-as-judge to our evaluation stack, its scores disagree with our existing judges just enough to make trend lines meaningless. We publish our benchmark findings internally and expect engineers here to read papers on evaluator calibration and implement what actually works, not just what sounds good in a blog post. In this role you'll build the calibration layer that aligns judge models against human-rated gold sets, so a capability score means the same thing whether it came from judge A or judge C, and report findings back to the evaluation team in a form they can act on.

Fully remote.

Responsibilities

  • Build calibration pipelines aligning LLM judges against human gold sets
  • Maintain a growing library of test cases covering judge disagreement patterns
  • Investigate and document root causes of evaluator score drift
  • Share findings on evaluator calibration with the wider research team

Requirements

  • 2+ years working hands-on with LLM evaluation or benchmarking
  • Comfortable reading research on evaluation methodology and applying it
  • Experience with LangChain or similar orchestration frameworks

Benefits

  • Fully remote with flexible hours
  • Internal research time allocated each month
  • Health, dental, and vision coverage

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

NLP & Prompt Engineering

Skills

python langchain prompt engineering openai api

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Security Engineer - Detection & Response

Lambda

Full-time Senior

Lead/ Senior Machine Learning Scientist, Integrative Phenotyping

insitro

Full-time Lead

AWS Alliances Lead

LangChain

Full-time Lead

AWS Alliances Lead

LangChain

Full-time Lead

AWS Alliances Lead

LangChain

Full-time Lead

Solutions Engineer, Canada

Glean

Full-time

Job Type

Full-time

Level

Mid-Level

Language

English

Salary Range

$100k – $130k / year

AI Expertise

NLP & Prompt Engineering

Skills

python langchain prompt engineering openai api

Ready to apply for this role?

Create a free talent account.

  • Find AI roles in one place
  • Get AI-matched job recommendations
  • Reply to companies that message you
  • Build your public AI talent profile
Create free account & apply Log in

This role is closed. Create your free profile to apply to similar open roles.

Don't miss the next one: get matched to new roles like this.

Create free profile