Profile cover
S

AI Evaluation Dataset Curation Engineer

Sofia Garcia

Full-time · Job · Los_Angeles

About the role

Every benchmark starts clean and becomes contaminated. Training data leaks into test sets. Models get fine-tuned on benchmark questions. Leaderboards lose meaning. We spend significant effort keeping our evaluation datasets fresh, diverse, and free of contamination — and that effort needs to scale faster than we're currently managing. As a junior curation engineer, you'll maintain and expand our evaluation dataset library: adding new test cases in underrepresented categories, running contamination detection against major training corpora, validating question quality against our taxonomy standards, and maintaining the versioning and metadata that makes each dataset traceable. Senior researchers define what good evaluation looks like; you execute the curation pipeline that produces it. This is an excellent role for someone methodical, detail-oriented, and genuinely interested in understanding how AI systems are evaluated. You'll learn the evaluation literature, understand why benchmark design matters, and contribute to a library used by external researchers who cite our datasets in papers.

Responsibilities

  • Curate new test cases for underrepresented categories in evaluation datasets
  • Run contamination detection against major training corpora for new additions
  • Validate question quality against taxonomy and difficulty calibration standards
  • Maintain versioning and metadata for all evaluation datasets
  • Build and maintain the pipeline that ingests, validates, and publishes new dataset versions
  • Document curation methodology for external researchers who use the datasets

Requirements

  • 1+ year of experience in data analysis, quality assurance, or research
  • Strong attention to detail and comfort with structured evaluation criteria
  • Basic Python and SQL skills for data manipulation
  • Interest in AI evaluation, benchmarks, or research methodology
  • Clear written communication for documentation and taxonomy descriptions

Benefits

  • Fully remote with flexible schedule
  • Health and dental insurance
  • $1,500 annual learning budget
  • Exposure to cutting-edge AI evaluation research
  • Mentorship from senior researchers with strong publication records

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

data science & analytics AI Product & Strategy

Skills

python sql pandas statistics openai api

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Technical Writer

RemoteAI company

Part-time

Product Manager, Developer Experience

RemoteAI company

Full-time

Software Engineer, Applications (Scientific Development Solutions) (High Seniority)

RemoteAI company

Full-time Senior

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

PhD Autonomy Engineer Intern - Computer Vision / Deep Learning Summer 2027

RemoteAI company

Full-time Intern

Backend Engineer, Platform

RemoteAI company

Full-time

Job Type

Full-time

Level

Junior

Language

English

Salary Range

$60k – $80k / year

AI Expertise

data science & analytics AI Product & Strategy

Skills

python sql pandas statistics openai api

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile