Profile cover
N

AI Model Shadow Testing Framework Developer

Nathan Yamamoto

Full-time · Job · Chicago

About the role

You can't know how a new model will behave in production until it's in production — unless you have a shadow testing framework. Shadow mode runs candidate models against real production traffic without affecting users, comparing outputs to the live model and flagging behavioral differences before they cause incidents. We build deployment infrastructure for ML teams at 140 enterprise clients. Shadow testing is one of the most-requested features — and the hardest to get right. The challenges: capturing representative traffic without privacy violations, handling models with different output schemas, measuring behavioral difference beyond simple accuracy, and doing all of this without adding latency to production requests. You'll own the shadow testing feature from backend to client SDK. The infrastructure exists; what's missing is the comparison layer — the behavioral diff engine that tells ML engineers what actually changed between the old model and the new one.

Responsibilities

  • Build the behavioral diff engine comparing shadow model outputs to production model
  • Design metrics capturing qualitative output differences beyond accuracy
  • Implement traffic capture that anonymizes PII before shadow replay
  • Create SDK integrations so client teams can configure shadow tests in minutes
  • Build shadow test dashboards visualizing behavioral change across traffic slices
  • Optimize shadow execution to zero production latency impact

Requirements

  • 3+ years in MLOps or platform engineering with model serving experience
  • Experience with traffic shadowing, canary releases, or A/B testing infrastructure
  • Strong Python and Docker/Kubernetes skills
  • Understanding of behavioral testing beyond simple accuracy metrics
  • Ability to build developer-facing tooling with good DX

Benefits

  • Fully remote
  • Health and dental from day one
  • $2,500 annual learning budget
  • Stock options
  • Flexible hours

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure Automation & AI Workflows

Skills

python docker kubernetes fastapi aws

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Staff Application Security Engineer

RemoteAI company

Full-time Staff

AI Resident UK/EMEA

RemoteAI company

Full-time

Devops Engineer, US

RemoteAI company

Full-time

Infrastructure Engineer

RemoteAI company

Full-time

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

Job Type

Full-time

Level

Mid-Level

Language

English

Salary Range

$80k – $100k / year

AI Expertise

MLOps & AI Infrastructure Automation & AI Workflows

Skills

python docker kubernetes fastapi aws mlflow

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile