AI Distributed Training Fault Tolerance Engineer

Mivex Cloud

Remote · $200k+ / year · Full-time · Posted 4 months ago

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

Have you ever debugged a distributed training job that failed at step 47,832 of 50,000 and lost 40 hours of gradient accumulation? We build the infrastructure that makes that scenario impossible. Mivex runs one of the largest AI training cluster scheduling systems outside the hyperscalers. Our customers are mid-market ML teams who can't afford an in-house infrastructure team but need GPU-efficient, fault-tolerant training at scale. Fault tolerance is the feature they ask about most. You'd join our core platform team of 6 and own the fault detection and recovery subsystem. This means designing checkpoint strategies, implementing failure classification, and building automatic restart logic that preserves training state with minimal compute waste. The work is low-level, high-impact, and requires deep knowledge of how PyTorch distributed and NCCL actually behave under failure conditions.

Fully remote.

Responsibilities

Design and implement checkpoint strategies for distributed training workloads
Build failure classification systems that distinguish recoverable from non-recoverable faults
Implement automatic restart and step recovery logic across multi-node training setups
Benchmark fault recovery overhead against training throughput across job sizes
Collaborate with cluster scheduling engineers to integrate fault signals into scheduling decisions

Requirements

6+ years in ML infrastructure or systems engineering
Deep knowledge of PyTorch distributed (DDP, FSDP) and NCCL
Experience with Kubernetes-based GPU cluster management
Ability to debug low-level distributed training failures
Prior work on checkpoint systems or fault-tolerant distributed systems preferred