Profile cover
D

AI Model Serving Load Balancer Engineer

David Dubois

Full-time · Job · Los_Angeles

About the role

Mivex Cloud runs ML inference infrastructure for 60+ enterprise clients across healthcare, finance, and e-commerce. Our cluster scheduling stack is mature — the next bottleneck is model serving load balancing: routing inference requests to the right model replica, managing autoscaling under bursty load, and maintaining tail latency SLAs across heterogeneous GPU hardware. In this senior role, you'll design and implement the intelligent load balancing layer for our model serving infrastructure. That means building request routing algorithms that account for model replica state, hardware affinity, and latency history, alongside the autoscaling policies that spin replicas up and down in response to real-time demand signals. This is infrastructure ownership at scale: your load balancer sits in the critical path of every inference call across the platform. We run a rigorous on-call rotation and expect this engineer to drive the reliability roadmap for model serving. If you've built high-throughput distributed systems before and want to apply that expertise to ML infrastructure, this is the role.

Responsibilities

  • Design and implement intelligent request routing algorithms for model serving replicas
  • Build autoscaling policies responsive to real-time inference demand signals
  • Own tail latency SLA monitoring and incident response for model serving
  • Collaborate with hardware team to optimize routing for heterogeneous GPU configurations
  • Drive the reliability roadmap for the model serving layer

Requirements

  • 5+ years infrastructure or distributed systems engineering experience
  • Deep Kubernetes expertise for stateful workload management
  • Experience building high-throughput load balancers or proxy systems
  • Strong Python and FastAPI skills
  • Background in ML infrastructure or model serving platforms is preferred

Benefits

  • Remote-first culture
  • Equity in a high-growth ML infrastructure company
  • Health, dental, and vision
  • $4,000 annual professional development budget
  • On-call compensation above base

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure AI & Machine Learning Engineers

Skills

python kubernetes docker aws fastapi

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Senior AI R&D Engineer – Model Training (f/m/d)

RemoteAI company

Full-time Senior

AI Resident UK/EMEA

RemoteAI company

Full-time

Devops Engineer, US

RemoteAI company

Full-time

AI Engineer

RemoteAI company

Full-time

Infrastructure Engineer

RemoteAI company

Full-time

Job Type

Full-time

Level

Senior

Language

English

Salary Range

$130k – $160k / year

AI Expertise

MLOps & AI Infrastructure AI & Machine Learning Engineers

Skills

python kubernetes docker aws fastapi

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile