AI Multi-Region Failover Orchestration Engineer

Covex Cloud

Remote · $160k – $200k / year · Full-time · Posted 4 months ago

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

Failover for a stateless web service is a solved problem. Failover for an ML serving cluster is not. When a model serving region goes down, you need to reroute inference traffic without violating latency SLAs, without saturating the surviving regions, and without dropping requests for models that only exist in the failed region. At our scale — 12 model serving clusters across three cloud providers — uncoordinated failover causes cascading failures more often than the original outage. You'll design and build the intelligent failover orchestration layer: traffic health monitoring that detects regional degradation before it becomes a total failure, predictive load shedding that preemptively redistributes traffic when a region shows early stress signals, cross-region model replication policies that ensure critical models are always available in at least two regions, and rollback automation for failover decisions that made things worse. This is a high-stakes infrastructure engineering role. You'll be on the hot path for our reliability roadmap and will carry an on-call pager for incidents. The platform SRE team is eight engineers; this role's scope covers the full failover orchestration system.

Fully remote.

Responsibilities

Design intelligent failover orchestration for ML serving clusters across 3 cloud providers
Build regional health monitoring with early degradation detection
Implement predictive load shedding based on leading stress indicators
Develop cross-region model replication policies for availability guarantees
Create rollback automation for failover decisions that worsen the situation
Define and monitor regional SLAs with automated escalation on SLA breach

Requirements

7+ years in platform engineering or SRE with multi-region distributed systems experience
Deep expertise in traffic management, load balancing, and failover strategies
Experience with ML serving infrastructure is a strong plus
Strong Python and Kubernetes skills
Comfort with on-call responsibilities for a production platform