Profile cover
S

AI Benchmark Design Engineer

Sofia Garcia

Contract · Job · London

About the role

Everyone claims their model beats the benchmark. But benchmarks themselves are full of contamination, metric gaming, and evaluation artifacts that make comparisons meaningless. Our evaluation lab builds clean, rigorous benchmarks for enterprise AI capabilities — and the testing infrastructure that produces trustworthy results. This 6-month contract focuses on benchmark dataset construction: identifying evaluation gaps in existing benchmarks, curating test cases that probe specific capabilities, validating that test items aren't contaminated in training data, and documenting evaluation methodology with enough rigor that results are reproducible. You'll learn evaluation science — the discipline of measuring AI capabilities accurately. It's one of the most important and least glamorous parts of AI development. If you're the kind of person who reads a leaderboard and immediately asks 'but how did they measure that?', this role is for you.

Responsibilities

  • Curate benchmark datasets targeting specific enterprise AI capability gaps
  • Validate test items for training data contamination
  • Design evaluation protocols that resist metric gaming and artifacts
  • Document benchmark methodology for reproducibility and peer review
  • Analyze evaluation results and identify systematic model weaknesses

Requirements

  • Python proficiency with pandas for data analysis
  • Basic understanding of ML evaluation methodology
  • Strong statistical reasoning for test design and result interpretation
  • Interest in AI evaluation science and benchmark quality
  • Meticulous attention to detail for dataset curation

Benefits

  • 6-month contract with renewal option
  • Fully remote — UK and EU time zones
  • Competitive contract rate
  • Work at the frontier of AI evaluation methodology

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

data science & analytics AI Product & Strategy

Skills

python pandas sql hugging face statistics

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Technical Writer

RemoteAI company

Part-time

Product Manager, Developer Experience

RemoteAI company

Full-time

Software Engineer, Applications (Scientific Development Solutions) (High Seniority)

RemoteAI company

Full-time Senior

PhD Autonomy Engineer Intern - Computer Vision / Deep Learning Summer 2027

RemoteAI company

Full-time Intern

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

Backend Engineer, Platform

RemoteAI company

Full-time

Job Type

Contract

Level

Junior

Language

English

Salary Range

$60k – $80k / year

AI Expertise

data science & analytics AI Product & Strategy

Skills

python pandas sql hugging face statistics

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile