AI Auto-Scaling Policy Learning Engineer

Xivio Solutions

Remote · $160k – $200k / year · Full-time · Posted 4 months ago

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

Senior Machine Learning Engineer (Tech Lead), Robot Learning, Loco- Manipulation Path Robotics View →
Senior Machine Learning Engineer (Reinforcement Learning/World Model) Path Robotics View →
Machine Learning Engineer, Perception Path Robotics View →

See all open AI jobs

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

Autoscaling policies for GPU inference clusters are typically reactive: scale up when utilization exceeds a threshold, scale down when it drops below one. Reactive autoscaling introduces latency spikes at scale-up time, wastes capacity during predictable low-demand periods, and fails to anticipate demand patterns from incoming request queues. Learned autoscaling policies — trained from historical demand patterns and response latency requirements — can pre-emptively provision capacity before demand arrives and more aggressively reclaim capacity during predictable low-demand windows. We manage GPU inference capacity for 45 enterprise ML teams, with 280 inference endpoints running across AWS and GCP. Autoscaling inefficiency is our biggest cost driver: over-provisioning adds 25-35% to GPU costs, and under-provisioning creates the latency violations that affect customer SLAs. We've profiled the demand patterns extensively. The next step is replacing rule-based autoscaling with learned policies that optimize for the cost-latency trade-off specific to each endpoint's traffic profile. You'll lead autoscaling policy learning: designing the RL-based policy learning framework, training policies on historical demand data and latency feedback, and implementing the policy deployment system that runs learned policies in production while monitoring against the reactive fallback.

Fully remote.

Responsibilities

Design RL-based autoscaling policy learning framework using historical demand patterns and latency feedback
Train endpoint-specific policies optimizing cost-latency trade-offs for each traffic profile type
Implement look-ahead scaling incorporating request queue depth and demand prediction signals
Build policy deployment system running learned policies with reactive fallback monitoring
Develop policy performance evaluation comparing learned versus rule-based scaling on cost and SLA metrics
Create policy adaptation updating policies as traffic patterns shift due to product changes or user growth

Requirements

6+ years in ML infrastructure or systems engineering with autoscaling, resource optimization, or RL application experience
Deep understanding of Kubernetes HPA/VPA, GPU cluster management, and inference serving latency dynamics
Strong Python and RL implementation skills with real system control problems
Experience deploying learned policies in production systems with safety constraints
Track record reducing infrastructure costs while maintaining SLA compliance through optimization