Profile cover
S

AI Benchmark Contamination Detection Engineer

Zenora Labs

Remote · $130k – $160k / year · Full-time · Posted 5 months ago

No longer accepting applications

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

When a model scores 95% on a benchmark, does it actually understand the task or did it memorize the test set during training? Benchmark contamination — training data that overlaps with evaluation data — is one of the biggest credibility problems in ML evaluation. Models that appear state-of-the-art on contaminated benchmarks fail dramatically on genuinely novel test cases. Our contamination detection system identifies when evaluation data has leaked into training sets. You'll build the detection methods: n-gram overlap analysis at scale, embedding similarity detection that catches paraphrased contamination, memorization probing that tests whether models can reproduce benchmark examples verbatim, and statistical tests that compare performance on likely-contaminated versus clean subsets. The evaluation integrity team is four engineers. You'll be the senior engineer who designs the detection methodology, working with our benchmark curation team who create the clean evaluation datasets and our client-facing team who explains contamination findings to model developers who may not appreciate being told their scores are inflated.

Fully remote.

Responsibilities

  • Design contamination detection methods for ML evaluation benchmarks
  • Build n-gram overlap and embedding similarity analysis at training-data scale
  • Implement memorization probing tests for model-level contamination detection
  • Create statistical tests comparing performance on contaminated vs clean subsets
  • Produce contamination reports for model developers and benchmark users
  • Research new detection approaches as contamination methods evolve

Requirements

  • 5+ years in ML evaluation, benchmarking, or model analysis
  • Deep understanding of training data dynamics and memorization
  • Strong Python, PyTorch, and statistical analysis skills
  • Experience with large-scale text analysis and embedding comparison
  • Ability to communicate sensitive findings diplomatically

Benefits

  • Fully remote with flexible schedule
  • Competitive salary plus equity
  • Health, dental, and vision insurance
  • $4,000 annual conference budget
  • Shape the integrity of ML evaluation worldwide

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

AI & Machine Learning Engineers Data Science & Analytics

Skills

python pytorch hugging face statistics docker

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Sr Technical Program Manager, Data

Hyperscience

Full-time Senior

Principal Engineer Team Lead, Behavior Safety & Metrics

Motional

Full-time Principal

AI Researcher - Pre-training (f/m/d)

Aleph Alpha

Full-time

Senior Software Engineer - Full Stack - Portal Platform

Abnormal Security

Full-time Senior

Senior Security Engineer - Detection & Response

Lambda

Full-time Senior

Solutions Engineer, Canada

Glean

Full-time

Job Type

Full-time

Level

Senior

Language

English

Salary Range

$130k – $160k / year

AI Expertise

AI & Machine Learning Engineers Data Science & Analytics

Skills

python pytorch hugging face statistics docker aws

Ready to apply for this role?

Create a free talent account.

  • Find AI roles in one place
  • Get AI-matched job recommendations
  • Reply to companies that message you
  • Build your public AI talent profile
Create free account & apply Log in

This role is closed. Create your free profile to apply to similar open roles.

Don't miss the next one: get matched to new roles like this.

Create free profile