Profile cover
S

AI Model Capability Evaluation Researcher

Zenora Labs

Remote · $160k – $200k / year · Full-time · Posted 4 months ago

No longer accepting applications

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

Benchmarks lie. A model that scores 90% on MMLU might fail at tasks that MMLU wasn't designed to measure, and the gap between benchmark performance and real-world capability is where AI products break. Building evaluations that actually predict real-world usefulness — that's an open research problem, and it's the problem we work on every day. We build evaluation infrastructure for AI labs and enterprise AI teams. Our platform runs capability assessments across reasoning, instruction following, tool use, factual accuracy, and safety behaviors. More importantly, we design novel evaluations for capabilities that existing benchmarks miss. When a client says their model fails at contract review but benchmarks say it's great at legal reasoning, we build the eval that explains why. You'll lead the evaluation research team, designing new benchmarks, running capability studies on frontier models, and publishing findings that advance the field's understanding of where current models succeed and fail.

Fully remote.

Responsibilities

  • Design novel benchmarks for capability gaps that existing evaluations miss
  • Run rigorous capability studies on frontier models across reasoning and instruction following
  • Build automated evaluation pipelines that scale across model families
  • Collaborate with AI lab clients to diagnose specific real-world capability failures
  • Publish evaluation research at NeurIPS, ICML, ACL, and related venues
  • Lead a team of two junior researchers and one evaluation engineer

Requirements

  • 6+ years in ML research with publications in NLP evaluation or AI capabilities
  • Deep understanding of evaluation methodology and benchmark design principles
  • Experience with frontier model evaluation across multiple capability dimensions
  • Strong statistical reasoning for evaluation result interpretation
  • Track record of publishing influential evaluation research

Benefits

  • Fully remote
  • Top-of-market researcher compensation
  • Unlimited research budget
  • 30 days PTO
  • Equity participation

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

AI & Machine Learning Engineers AI Product & Strategy

Skills

python pytorch statistics hugging face pandas

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

AI Researcher - Pre-training (f/m/d)

Aleph Alpha

Full-time

Research Engineer / Scientist, Pre-training

Anthropic

Full-time

AI Security Engineer Intern (Summer 2027)

Brainco

Full-time Intern

Lead/ Senior Machine Learning Scientist, Integrative Phenotyping

insitro

Full-time Lead

Senior Security Engineer - Detection & Response

Lambda

Full-time Senior

Solutions Engineer, Canada

Glean

Full-time

Job Type

Full-time

Level

Senior

Language

English

Salary Range

$160k – $200k / year

AI Expertise

AI & Machine Learning Engineers AI Product & Strategy

Skills

python pytorch statistics hugging face pandas docker

Ready to apply for this role?

Create a free talent account.

  • Find AI roles in one place
  • Get AI-matched job recommendations
  • Reply to companies that message you
  • Build your public AI talent profile
Create free account & apply Log in

This role is closed. Create your free profile to apply to similar open roles.

Don't miss the next one: get matched to new roles like this.

Create free profile