Profile cover
C

AI Model Performance Comparison Dashboard Developer

Connor Leclerc

Full-time · Job · Chicago

About the role

An ML team running 40 experiments needs to answer one question: which model should we deploy? Answering it requires comparing performance across held-out sets, production traffic samples, latency benchmarks, and cost profiles simultaneously — and doing it in a UI that doesn't require an engineering degree to interpret. Most experiment tracking tools show you the numbers. They don't help you make the decision. Our ML platform serves 55 enterprise teams, and the model comparison interface is consistently the feature users cite when they switch from Weights & Biases or MLflow. We have a working comparison view. It's not good enough: it handles metric comparison well but falls apart on model behavior comparison — where do the two models disagree, on which input types, and what does that pattern suggest about which is better for the deployment use case? You'll own the comparison dashboard: building behavioral disagreement analysis that surfaces where two models differ on real inputs, implementing sliced evaluation views that compare performance across demographic and feature segments, and designing the recommendation interface that synthesizes comparison signals into a deployment recommendation with explicit reasoning.

Responsibilities

  • Build behavioral disagreement analysis surfacing where candidate models diverge on real production inputs
  • Implement sliced evaluation views comparing model performance across user-defined feature segments
  • Design recommendation interface synthesizing comparison signals into reasoned deployment suggestions
  • Create cost-vs-performance trade-off visualization for models with different latency and accuracy profiles
  • Build comparison sharing and annotation features for collaborative model review
  • Develop comparison dashboard APIs consumed by CI/CD deployment gates

Requirements

  • 3+ years in ML platform engineering, data visualization, or developer tooling
  • Strong Python and TypeScript skills with dashboard or analytics product development experience
  • Understanding of ML evaluation methodology and experiment tracking concepts
  • Experience designing interfaces that help non-expert users make complex technical decisions
  • Familiarity with MLflow, Weights & Biases, or similar experiment tracking tools

Benefits

  • Fully remote with flexible schedule
  • Health, dental, and vision insurance
  • $3,000 annual learning budget
  • 401(k) with match
  • Equity options

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure AI Product & Strategy

Skills

python fastapi docker sql aws

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Principal Engineer

RemoteAI company

Full-time Principal

Staff Application Security Engineer

RemoteAI company

Full-time Staff

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

Backend Engineer, Platform

RemoteAI company

Full-time

Technical Writer

RemoteAI company

Part-time

Job Type

Full-time

Level

Mid-Level

Language

English

Salary Range

$100k – $130k / year

AI Expertise

MLOps & AI Infrastructure AI Product & Strategy

Skills

python fastapi docker sql typescript aws

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile