Profile cover
A

AI Multi-Region Failover Orchestration Engineer

Aiden Ferrari

Full-time · Job · New_York

About the role

Failover for a stateless web service is a solved problem. Failover for an ML serving cluster is not. When a model serving region goes down, you need to reroute inference traffic without violating latency SLAs, without saturating the surviving regions, and without dropping requests for models that only exist in the failed region. At our scale — 12 model serving clusters across three cloud providers — uncoordinated failover causes cascading failures more often than the original outage. You'll design and build the intelligent failover orchestration layer: traffic health monitoring that detects regional degradation before it becomes a total failure, predictive load shedding that preemptively redistributes traffic when a region shows early stress signals, cross-region model replication policies that ensure critical models are always available in at least two regions, and rollback automation for failover decisions that made things worse. This is a high-stakes infrastructure engineering role. You'll be on the hot path for our reliability roadmap and will carry an on-call pager for incidents. The platform SRE team is eight engineers; this role's scope covers the full failover orchestration system.

Responsibilities

  • Design intelligent failover orchestration for ML serving clusters across 3 cloud providers
  • Build regional health monitoring with early degradation detection
  • Implement predictive load shedding based on leading stress indicators
  • Develop cross-region model replication policies for availability guarantees
  • Create rollback automation for failover decisions that worsen the situation
  • Define and monitor regional SLAs with automated escalation on SLA breach

Requirements

  • 7+ years in platform engineering or SRE with multi-region distributed systems experience
  • Deep expertise in traffic management, load balancing, and failover strategies
  • Experience with ML serving infrastructure is a strong plus
  • Strong Python and Kubernetes skills
  • Comfort with on-call responsibilities for a production platform

Benefits

  • Fully remote with flexible schedule
  • Top-of-market compensation
  • Equity package
  • Comprehensive health benefits including mental health coverage
  • Annual $6,000 professional development budget

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure Automation & AI Workflows

Skills

python kubernetes docker aws fastapi

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Staff Application Security Engineer

RemoteAI company

Full-time Staff

AI Resident UK/EMEA

RemoteAI company

Full-time

Devops Engineer, US

RemoteAI company

Full-time

Infrastructure Engineer

RemoteAI company

Full-time

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

Job Type

Full-time

Level

Senior

Language

English

Salary Range

$160k – $200k / year

AI Expertise

MLOps & AI Infrastructure Automation & AI Workflows

Skills

python kubernetes docker aws fastapi statistics

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile