Profile cover
E

AI Spot Instance Interruption Predictor

Ellie Petrov

Full-time · Job · Los_Angeles

About the role

Spot instances can be 70-90% cheaper than on-demand compute, but they get terminated with 2 minutes notice when AWS needs the capacity back. For ML training workloads, an unexpected spot interruption on hour 47 of a 50-hour training run is catastrophic unless checkpointing is handled correctly. For batch inference workloads, it causes queue backup and latency spikes. For stateful services, it can cause data loss. Our infrastructure optimization platform manages compute for 35 ML-heavy clients. Spot interruption prediction — estimating the probability and likely timing of interruption for active spot instances — is the signal that drives intelligent workload management: when to checkpoint aggressively, when to pre-warm on-demand fallback capacity, and when to migrate workloads to a different instance pool before interruption hits. You'll build the interruption prediction model: engineering features from instance type, availability zone, current pool utilization signals, historical interruption patterns, and market pricing trends, training prediction models on our historical interruption dataset, and deploying the prediction service that feeds workload management decisions in real time.

Responsibilities

  • Engineer prediction features from instance pool utilization, historical interruption rates, and pricing signals
  • Train interruption probability models with calibrated confidence outputs per instance type and AZ
  • Build time-to-interruption distribution estimates enabling checkpoint timing decisions
  • Implement real-time prediction service serving interruption risk to workload management system
  • Develop monitoring tracking prediction calibration against realized interruption events
  • Evaluate cost savings from interruption-aware workload management versus baseline

Requirements

  • 3+ years in cloud infrastructure, ML engineering, or platform data science
  • Strong Python and SQL skills with AWS spot market and EC2 experience
  • Experience with calibrated probabilistic forecasting models
  • Understanding of ML training workloads and checkpoint strategies
  • Ability to design prediction systems that drive real-time infrastructure decisions

Benefits

  • Fully remote with flexible schedule
  • Health and dental insurance
  • $3,000 annual professional development budget
  • 401(k) with match
  • Equity options

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure data science & analytics

Skills

python scikit-learn sql aws spark

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

Technical Writer

RemoteAI company

Part-time

Staff Application Security Engineer

RemoteAI company

Full-time Staff

Software Engineer, Applications (Scientific Development Solutions) (High Seniority)

RemoteAI company

Full-time Senior

PhD Autonomy Engineer Intern - Computer Vision / Deep Learning Summer 2027

RemoteAI company

Full-time Intern

Job Type

Full-time

Level

Mid-Level

Language

English

Salary Range

$100k – $130k / year

AI Expertise

MLOps & AI Infrastructure data science & analytics

Skills

python scikit-learn sql aws spark statistics

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile