AI Model Training Checkpoint Recovery Engineer

Mivex Cloud

Remote · $130k – $160k / year · Full-time · Posted 4 months ago

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

Training a large model for 14 days and losing everything to a hardware fault on day 13 is not a theoretical scenario — it happens regularly on GPU clusters operating at scale. The difference between a training platform that loses that work and one that recovers and resumes within 30 minutes is entirely in checkpoint engineering: how frequently checkpoints are written, how efficiently they're stored, how quickly the cluster can reconstruct distributed training state from a checkpoint, and how gracefully the system handles the partial state left by mid-iteration failures. Our training infrastructure runs 400 concurrent model training jobs for enterprise ML teams. Checkpoint reliability is one of the top three causes of job failure and retry cost. We have checkpoint writing. We don't have intelligent checkpoint management: adaptive frequency based on failure probability, compressed incremental checkpoints that write only what changed, and distributed state reconstruction that restores multi-node training state without manual intervention. You'll own the checkpoint recovery system end to end: designing the adaptive frequency strategy, implementing incremental checkpoint compression, building the distributed state reconstruction pipeline, and creating the failure detection that triggers automatic recovery rather than requiring an engineer to notice and intervene.

Fully remote.

Responsibilities

Design adaptive checkpoint frequency strategy based on hardware fault probability and job value estimation
Implement incremental checkpoint compression reducing storage I/O without sacrificing recovery completeness
Build distributed state reconstruction restoring multi-node training from checkpoints with no manual intervention
Develop failure detection and automatic recovery triggering integrated with cluster health monitoring
Create checkpoint lifecycle management expiring old checkpoints while preserving recovery-critical history
Monitor checkpoint recovery success rate and mean time to resume as primary reliability metrics

Requirements

6+ years in ML infrastructure with distributed training and fault tolerance engineering experience
Deep understanding of PyTorch distributed training, gradient synchronization, and checkpointing internals
Strong Python and Kubernetes skills with large-scale GPU cluster operations
Experience designing fault-tolerant distributed systems that recover from partial state
Track record reducing training job retry rates through infrastructure reliability improvements