Profile cover
S

AI Benchmark Contamination Detection Engineer

Sofia Garcia

Full-time · Job · Los_Angeles

About the role

When a model scores 95% on a benchmark, does it actually understand the task or did it memorize the test set during training? Benchmark contamination — training data that overlaps with evaluation data — is one of the biggest credibility problems in ML evaluation. Models that appear state-of-the-art on contaminated benchmarks fail dramatically on genuinely novel test cases. Our contamination detection system identifies when evaluation data has leaked into training sets. You'll build the detection methods: n-gram overlap analysis at scale, embedding similarity detection that catches paraphrased contamination, memorization probing that tests whether models can reproduce benchmark examples verbatim, and statistical tests that compare performance on likely-contaminated versus clean subsets. The evaluation integrity team is four engineers. You'll be the senior engineer who designs the detection methodology, working with our benchmark curation team who create the clean evaluation datasets and our client-facing team who explains contamination findings to model developers who may not appreciate being told their scores are inflated.

Responsibilities

  • Design contamination detection methods for ML evaluation benchmarks
  • Build n-gram overlap and embedding similarity analysis at training-data scale
  • Implement memorization probing tests for model-level contamination detection
  • Create statistical tests comparing performance on contaminated vs clean subsets
  • Produce contamination reports for model developers and benchmark users
  • Research new detection approaches as contamination methods evolve

Requirements

  • 5+ years in ML evaluation, benchmarking, or model analysis
  • Deep understanding of training data dynamics and memorization
  • Strong Python, PyTorch, and statistical analysis skills
  • Experience with large-scale text analysis and embedding comparison
  • Ability to communicate sensitive findings diplomatically

Benefits

  • Fully remote with flexible schedule
  • Competitive salary plus equity
  • Health, dental, and vision insurance
  • $4,000 annual conference budget
  • Shape the integrity of ML evaluation worldwide

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

AI & Machine Learning Engineers data science & analytics

Skills

python pytorch hugging face statistics docker

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Senior AI Researcher- Pre-training (f/m/d)

RemoteAI company

Full-time Senior

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

AI Resident UK/EMEA

RemoteAI company

Full-time

Senior AI R&D Engineer – Model Training (f/m/d)

RemoteAI company

Full-time Senior

Senior Machine Learning Engineer (Clinical Team)

RemoteAI company

Full-time Senior

Job Type

Full-time

Level

Senior

Language

English

Salary Range

$130k – $160k / year

AI Expertise

AI & Machine Learning Engineers data science & analytics

Skills

python pytorch hugging face statistics docker aws

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile