Profile cover
A

AI Canary Analysis Automation Engineer

Aurora Vestergaard

Full-time · Job · Los_Angeles

About the role

Canary deployments exist to catch problems before they affect 100% of traffic. The problem is that most canary analysis is manual: an engineer watches dashboards for 30 minutes and decides whether the metrics look okay. That works when you have 5 deployments per day. At 200 deployments per day — which we do — it creates a bottleneck and introduces human error into the most critical safety gate in the deployment pipeline. We're building fully automated canary analysis: a system that continuously monitors key metrics during a deployment, applies statistical tests to distinguish signal from noise, detects regression patterns before they become incidents, and makes automated rollout or rollback decisions with documented reasoning. When an engineer deploys a new model version at 2 AM, the system handles the canary phase without waking anyone up — unless it actually needs to. This is a staff-level IC role requiring deep expertise in both statistical inference and production infrastructure. You'll design the statistical framework, build the monitoring integrations, and set the standards that 12 model deployment teams rely on every day.

Responsibilities

  • Design the statistical analysis framework for automated canary deployment decisions
  • Build monitoring integrations for business metrics, error rates, and model performance signals
  • Implement automated rollout and rollback logic with documented decision reasoning
  • Develop anomaly detection that distinguishes deployment-caused regressions from background noise
  • Create the audit trail that lets engineers review automated deployment decisions post-hoc
  • Define canary analysis standards and train deployment teams on the framework

Requirements

  • 7+ years in MLOps, SRE, or platform engineering
  • Deep expertise in statistical inference and hypothesis testing
  • Experience building automated deployment safety systems
  • Strong Python and Kubernetes skills
  • Track record of reducing deployment risk without slowing release velocity

Benefits

  • Fully remote with flexible schedule
  • Top-of-market total compensation
  • Significant equity package
  • Comprehensive health benefits
  • Annual $6,000 professional development budget

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure Automation & AI Workflows

Skills

python kubernetes docker aws statistics

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Staff Application Security Engineer

RemoteAI company

Full-time Staff

AI Resident UK/EMEA

RemoteAI company

Full-time

Devops Engineer, US

RemoteAI company

Full-time

Infrastructure Engineer

RemoteAI company

Full-time

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

Job Type

Full-time

Level

Senior

Language

English

Salary Range

$200k+ / year

AI Expertise

MLOps & AI Infrastructure Automation & AI Workflows

Skills

python kubernetes docker aws statistics fastapi

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile