Profile cover
D

AI Model Training Checkpoint Recovery Engineer

David Dubois

Full-time · Job · Los_Angeles

About the role

Training a large model for 14 days and losing everything to a hardware fault on day 13 is not a theoretical scenario — it happens regularly on GPU clusters operating at scale. The difference between a training platform that loses that work and one that recovers and resumes within 30 minutes is entirely in checkpoint engineering: how frequently checkpoints are written, how efficiently they're stored, how quickly the cluster can reconstruct distributed training state from a checkpoint, and how gracefully the system handles the partial state left by mid-iteration failures. Our training infrastructure runs 400 concurrent model training jobs for enterprise ML teams. Checkpoint reliability is one of the top three causes of job failure and retry cost. We have checkpoint writing. We don't have intelligent checkpoint management: adaptive frequency based on failure probability, compressed incremental checkpoints that write only what changed, and distributed state reconstruction that restores multi-node training state without manual intervention. You'll own the checkpoint recovery system end to end: designing the adaptive frequency strategy, implementing incremental checkpoint compression, building the distributed state reconstruction pipeline, and creating the failure detection that triggers automatic recovery rather than requiring an engineer to notice and intervene.

Responsibilities

  • Design adaptive checkpoint frequency strategy based on hardware fault probability and job value estimation
  • Implement incremental checkpoint compression reducing storage I/O without sacrificing recovery completeness
  • Build distributed state reconstruction restoring multi-node training from checkpoints with no manual intervention
  • Develop failure detection and automatic recovery triggering integrated with cluster health monitoring
  • Create checkpoint lifecycle management expiring old checkpoints while preserving recovery-critical history
  • Monitor checkpoint recovery success rate and mean time to resume as primary reliability metrics

Requirements

  • 6+ years in ML infrastructure with distributed training and fault tolerance engineering experience
  • Deep understanding of PyTorch distributed training, gradient synchronization, and checkpointing internals
  • Strong Python and Kubernetes skills with large-scale GPU cluster operations
  • Experience designing fault-tolerant distributed systems that recover from partial state
  • Track record reducing training job retry rates through infrastructure reliability improvements

Benefits

  • Fully remote with flexible schedule
  • Comprehensive health benefits
  • $4,000 annual professional development budget
  • Equity participation
  • Performance bonus

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure

Skills

python pytorch kubernetes aws docker

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

AI Resident UK/EMEA

RemoteAI company

Full-time

Devops Engineer, US

RemoteAI company

Full-time

Infrastructure Engineer

RemoteAI company

Full-time

Senior AI R&D Engineer – Model Training (f/m/d)

RemoteAI company

Full-time Senior

Senior AI Researcher- Pre-training (f/m/d)

RemoteAI company

Full-time Senior

Job Type

Full-time

Level

Senior

Language

English

Salary Range

$130k – $160k / year

AI Expertise

MLOps & AI Infrastructure

Skills

python pytorch kubernetes aws docker sql

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile