AI Inference Model Warm Pool Manager

About the role

Responsibilities

• Build a warm pool management system predicting near-term per-model inference demand
• Balance latency improvements against the GPU cost of maintaining idle warm capacity
• Coordinate warm pool policy with existing autoscaling and batching systems
• Run controlled experiments measuring p99 latency impact of warm pool changes
• Own cost reporting showing warm pool spend against latency gains delivered

Requirements

• 6+ years in infrastructure engineering, ideally ML serving systems
• Strong Kubernetes and Docker experience in production
• Comfortable working within AWS-based GPU infrastructure
• Experience with latency-sensitive, cost-constrained systems

Benefits

• Fully remote position
• Competitive salary and equity
• Health insurance
• Access to internal GPU clusters for personal experimentation

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.