Profile cover
E

AI Inference Model Warm Pool Manager

Aevix Cloud

Remote · $160k – $200k / year · Full-time · Posted 3 months ago

No longer accepting applications

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

The models you'll help serve process millions of inference requests a day, and right now a meaningful share of our p99 latency comes from cold starts on infrequently used models. You will build a warm pool management system that predicts near-term demand per model and keeps the right instances pre-warmed, balancing latency improvement against the GPU cost of idle capacity. You'll work closely with the team that owns our autoscaling and batching infrastructure, since warm pool decisions need to stay coordinated with their scaling policies rather than fight against them.

Fully remote.

Responsibilities

  • Build a warm pool management system predicting near-term per-model inference demand
  • Balance latency improvements against the GPU cost of maintaining idle warm capacity
  • Coordinate warm pool policy with existing autoscaling and batching systems
  • Run controlled experiments measuring p99 latency impact of warm pool changes
  • Own cost reporting showing warm pool spend against latency gains delivered

Requirements

  • 6+ years in infrastructure engineering, ideally ML serving systems
  • Strong Kubernetes and Docker experience in production
  • Comfortable working within AWS-based GPU infrastructure
  • Experience with latency-sensitive, cost-constrained systems

Benefits

  • Fully remote position
  • Competitive salary and equity
  • Health insurance
  • Access to internal GPU clusters for personal experimentation

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure

Skills

python kubernetes aws docker

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Technical Support Engineer - APAC

Hyperbolic

Contract

Senior Engineer, Exposure & Vulnerability Management

Recursion

Full-time Senior

Software Engineer

Turing

Full-time

Principal AI Security Engineer

Cerebras

Full-time Principal

Member of Technical Staff, GPU Infrastructure

Cohere

Full-time Staff

Member of Technical Staff, ML Workload Platform

Cohere

Full-time Staff

Job Type

Full-time

Level

Senior

Language

English

Salary Range

$160k – $200k / year

AI Expertise

MLOps & AI Infrastructure

Skills

python kubernetes aws docker

Ready to apply for this role?

Create a free talent account.

  • Find AI roles in one place
  • Get AI-matched job recommendations
  • Reply to companies that message you
  • Build your public AI talent profile
Create free profile and apply Log in

This role is closed. Create your free profile to apply to similar open roles.

Don't miss the next one: get matched to new roles like this.

Create free profile