Profile cover
A

AI Model Traffic Shadowing Engineer

Aurora Vestergaard

Full-time · Job · Los_Angeles

About the role

Traffic shadowing is how you validate a new model against production traffic before it touches users: mirror live requests to the candidate model, record its responses, compare them against the production model's responses without affecting the user experience. It's the deployment safety technique that lets you catch regressions on real traffic rather than test sets — and most ML teams implement it badly, incompletely, or not at all. We build deployment reliability infrastructure for 45 ML teams across 18 enterprise clients. Shadowing is our most-requested missing capability: teams want to validate new models on their actual traffic patterns before switching routing, but they don't want to build the shadowing infrastructure themselves. You'll design and build the traffic shadowing system: request mirroring with configurable sampling rates, asynchronous response comparison with configurable divergence metrics, and the analysis dashboards that help teams understand where the candidate model behaves differently from production and whether those differences are improvements or regressions. The system needs to handle 50K requests per second at the highest-volume client with sub-millisecond impact on production request latency. The role is hybrid — 2 days per week in our SF office.

Responsibilities

  • Design traffic shadowing system supporting configurable request mirroring at 50K+ RPS
  • Implement asynchronous candidate model invocation with zero latency impact on production path
  • Build response comparison framework with configurable divergence metrics per model type
  • Create shadow analysis dashboards helping teams identify regression and improvement regions
  • Develop sampling strategies targeting high-value request subsets for comparison efficiency
  • Integrate shadowing into existing blue-green and canary deployment workflows

Requirements

  • 6+ years in ML infrastructure, platform engineering, or distributed systems
  • Deep understanding of high-throughput request routing and traffic mirroring architectures
  • Experience with Kubernetes, service mesh, or load balancing at production scale
  • Ability to design systems with strict latency constraints on the production request path
  • Track record building reliable deployment safety infrastructure for ML systems

Benefits

  • Hybrid: 2 days SF office, 3 days remote
  • Comprehensive health benefits
  • $4,000 annual professional development budget
  • Equity participation
  • Performance bonus

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure AI & Machine Learning Engineers

Skills

python docker kubernetes aws fastapi

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Senior AI R&D Engineer – Model Training (f/m/d)

RemoteAI company

Full-time Senior

AI Resident UK/EMEA

RemoteAI company

Full-time

Devops Engineer, US

RemoteAI company

Full-time

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

Technical Writer

RemoteAI company

Part-time

Job Type

Full-time

Level

Senior

Language

English

Salary Range

$130k – $160k / year

AI Expertise

MLOps & AI Infrastructure AI & Machine Learning Engineers

Skills

python docker kubernetes aws fastapi sql

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile