Profile cover
S

AI Evaluation Dataset Curation Engineer

Zenora Labs

Remote · $60k – $80k / year · Full-time · Posted 4 months ago

No longer accepting applications

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

Every benchmark starts clean and becomes contaminated. Training data leaks into test sets. Models get fine-tuned on benchmark questions. Leaderboards lose meaning. We spend significant effort keeping our evaluation datasets fresh, diverse, and free of contamination — and that effort needs to scale faster than we're currently managing. As a junior curation engineer, you'll maintain and expand our evaluation dataset library: adding new test cases in underrepresented categories, running contamination detection against major training corpora, validating question quality against our taxonomy standards, and maintaining the versioning and metadata that makes each dataset traceable. Senior researchers define what good evaluation looks like; you execute the curation pipeline that produces it. This is an excellent role for someone methodical, detail-oriented, and genuinely interested in understanding how AI systems are evaluated. You'll learn the evaluation literature, understand why benchmark design matters, and contribute to a library used by external researchers who cite our datasets in papers.

Fully remote.

Responsibilities

  • Curate new test cases for underrepresented categories in evaluation datasets
  • Run contamination detection against major training corpora for new additions
  • Validate question quality against taxonomy and difficulty calibration standards
  • Maintain versioning and metadata for all evaluation datasets
  • Build and maintain the pipeline that ingests, validates, and publishes new dataset versions
  • Document curation methodology for external researchers who use the datasets

Requirements

  • 1+ year of experience in data analysis, quality assurance, or research
  • Strong attention to detail and comfort with structured evaluation criteria
  • Basic Python and SQL skills for data manipulation
  • Interest in AI evaluation, benchmarks, or research methodology
  • Clear written communication for documentation and taxonomy descriptions

Benefits

  • Fully remote with flexible schedule
  • Health and dental insurance
  • $1,500 annual learning budget
  • Exposure to cutting-edge AI evaluation research
  • Mentorship from senior researchers with strong publication records

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

Data Science & Analytics AI Product & Strategy

Skills

python sql pandas statistics openai api

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Revenue Operations Manager

Handshake

Full-time

Materials Lead

Lambda

Full-time Lead

Sr. Solutions Architect - Travel and Hospitality

Databricks

Full-time Senior

AI Product Engineer Intern (Summer 2027)

Brainco

Full-time Intern

Senior Security Engineer - Detection & Response

Lambda

Full-time Senior

Lead/ Senior Machine Learning Scientist, Integrative Phenotyping

insitro

Full-time Lead

Job Type

Full-time

Level

Junior

Language

English

Salary Range

$60k – $80k / year

AI Expertise

Data Science & Analytics AI Product & Strategy

Skills

python sql pandas statistics openai api

Ready to apply for this role?

Create a free talent account.

  • Find AI roles in one place
  • Get AI-matched job recommendations
  • Reply to companies that message you
  • Build your public AI talent profile
Create free account & apply Log in

This role is closed. Create your free profile to apply to similar open roles.

Don't miss the next one: get matched to new roles like this.

Create free profile