AI Spot Instance Interruption Predictor

Xivio Solutions

Remote · $100k – $130k / year · Full-time · Posted 4 months ago

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

Spot instances can be 70-90% cheaper than on-demand compute, but they get terminated with 2 minutes notice when AWS needs the capacity back. For ML training workloads, an unexpected spot interruption on hour 47 of a 50-hour training run is catastrophic unless checkpointing is handled correctly. For batch inference workloads, it causes queue backup and latency spikes. For stateful services, it can cause data loss. Our infrastructure optimization platform manages compute for 35 ML-heavy clients. Spot interruption prediction — estimating the probability and likely timing of interruption for active spot instances — is the signal that drives intelligent workload management: when to checkpoint aggressively, when to pre-warm on-demand fallback capacity, and when to migrate workloads to a different instance pool before interruption hits. You'll build the interruption prediction model: engineering features from instance type, availability zone, current pool utilization signals, historical interruption patterns, and market pricing trends, training prediction models on our historical interruption dataset, and deploying the prediction service that feeds workload management decisions in real time.

Fully remote.

Responsibilities

Engineer prediction features from instance pool utilization, historical interruption rates, and pricing signals
Train interruption probability models with calibrated confidence outputs per instance type and AZ
Build time-to-interruption distribution estimates enabling checkpoint timing decisions
Implement real-time prediction service serving interruption risk to workload management system
Develop monitoring tracking prediction calibration against realized interruption events
Evaluate cost savings from interruption-aware workload management versus baseline

Requirements

3+ years in cloud infrastructure, ML engineering, or platform data science
Strong Python and SQL skills with AWS spot market and EC2 experience
Experience with calibrated probabilistic forecasting models
Understanding of ML training workloads and checkpoint strategies
Ability to design prediction systems that drive real-time infrastructure decisions