Profile cover
E

Senior AI Training Job Preemption Recovery Engineer

Ellie Petrov

Full-time · Job · Los_Angeles

About the role

Our customers run multi-day training jobs on spot and preemptible GPUs to save money, and they lose hours of progress every time a node disappears. Xivio Solutions already handles capacity planning and scheduling; the missing piece is fast, automatic recovery. You'll build it: asynchronous checkpointing that doesn't stall training, elastic resumption on a different number of GPUs, and smart rescheduling across AWS and Azure capacity pools. You'll work with a team of five infra engineers and be the technical owner of recovery end to end.

Responsibilities

  • Design asynchronous, sharded checkpointing for large distributed training jobs
  • Build elastic resume so jobs restart on whatever capacity is available
  • Own recovery time and wasted-GPU-hours metrics across all customer clusters
  • Drive adoption with customer ML teams through docs, examples and office hours

Requirements

  • Significant experience with distributed training (FSDP, DeepSpeed or similar)
  • Strong Kubernetes and cloud infrastructure background on AWS or Azure
  • Comfortable debugging NCCL, storage throughput and scheduler issues
  • History of owning reliability-critical systems

Benefits

  • Remote in the US
  • Salary at the top of market plus equity
  • Unlimited PTO with a 4-week minimum
  • Premium health coverage
  • Annual hardware refresh

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure

Skills

python kubernetes pytorch aws azure

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Solutions Architect

RemoteAI company

Full-time

Infrastructure Systems Engineer – Linux, Storage & Automation

RemoteAI company

Full-time

Staff Software Engineer, Identity and Authorization

RemoteAI company

Full-time Staff

Staff Software Engineer, Identity and Authorization

RemoteAI company

Full-time Staff

Senior Forward Deployed Engineer, Sr FDE (L5)

RemoteAI company

Full-time Senior

Job Type

Full-time

Level

Senior

Language

English

Salary Range

$200k+ / year

AI Expertise

MLOps & AI Infrastructure

Skills

python kubernetes pytorch aws azure docker

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to AI roles in one place
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile