Profile cover
S

AI Model Capability Evaluation Researcher

Sofia Garcia

Full-time · Job · New_York

About the role

Benchmarks lie. A model that scores 90% on MMLU might fail at tasks that MMLU wasn't designed to measure, and the gap between benchmark performance and real-world capability is where AI products break. Building evaluations that actually predict real-world usefulness — that's an open research problem, and it's the problem we work on every day. We build evaluation infrastructure for AI labs and enterprise AI teams. Our platform runs capability assessments across reasoning, instruction following, tool use, factual accuracy, and safety behaviors. More importantly, we design novel evaluations for capabilities that existing benchmarks miss. When a client says their model fails at contract review but benchmarks say it's great at legal reasoning, we build the eval that explains why. You'll lead the evaluation research team, designing new benchmarks, running capability studies on frontier models, and publishing findings that advance the field's understanding of where current models succeed and fail.

Responsibilities

  • Design novel benchmarks for capability gaps that existing evaluations miss
  • Run rigorous capability studies on frontier models across reasoning and instruction following
  • Build automated evaluation pipelines that scale across model families
  • Collaborate with AI lab clients to diagnose specific real-world capability failures
  • Publish evaluation research at NeurIPS, ICML, ACL, and related venues
  • Lead a team of two junior researchers and one evaluation engineer

Requirements

  • 6+ years in ML research with publications in NLP evaluation or AI capabilities
  • Deep understanding of evaluation methodology and benchmark design principles
  • Experience with frontier model evaluation across multiple capability dimensions
  • Strong statistical reasoning for evaluation result interpretation
  • Track record of publishing influential evaluation research

Benefits

  • Fully remote
  • Top-of-market researcher compensation
  • Unlimited research budget
  • 30 days PTO
  • Equity participation

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

AI & Machine Learning Engineers AI Product & Strategy

Skills

python pytorch statistics hugging face pandas

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior AI Researcher- Pre-training (f/m/d)

RemoteAI company

Full-time Senior

Senior Product Security Engineer

RemoteAI company

Full-time Senior

AI Resident UK/EMEA

RemoteAI company

Full-time

Senior AI R&D Engineer – Model Training (f/m/d)

RemoteAI company

Full-time Senior

Senior Machine Learning Engineer (Clinical Team)

RemoteAI company

Full-time Senior

Multimodal AI Model Optimization Research Engineer

RemoteAI company

Full-time

Job Type

Full-time

Level

Senior

Language

English

Salary Range

$160k – $200k / year

AI Expertise

AI & Machine Learning Engineers AI Product & Strategy

Skills

python pytorch statistics hugging face pandas docker

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile