Profile cover
D

AI Model Serving Load Balancer Engineer

Mivex Cloud

Remote · $130k – $160k / year · Full-time · Posted 4 months ago

No longer accepting applications

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

Mivex Cloud runs ML inference infrastructure for 60+ enterprise clients across healthcare, finance, and e-commerce. Our cluster scheduling stack is mature — the next bottleneck is model serving load balancing: routing inference requests to the right model replica, managing autoscaling under bursty load, and maintaining tail latency SLAs across heterogeneous GPU hardware. In this senior role, you'll design and implement the intelligent load balancing layer for our model serving infrastructure. That means building request routing algorithms that account for model replica state, hardware affinity, and latency history, alongside the autoscaling policies that spin replicas up and down in response to real-time demand signals. This is infrastructure ownership at scale: your load balancer sits in the critical path of every inference call across the platform. We run a rigorous on-call rotation and expect this engineer to drive the reliability roadmap for model serving. If you've built high-throughput distributed systems before and want to apply that expertise to ML infrastructure, this is the role.

Fully remote.

Responsibilities

  • Design and implement intelligent request routing algorithms for model serving replicas
  • Build autoscaling policies responsive to real-time inference demand signals
  • Own tail latency SLA monitoring and incident response for model serving
  • Collaborate with hardware team to optimize routing for heterogeneous GPU configurations
  • Drive the reliability roadmap for the model serving layer

Requirements

  • 5+ years infrastructure or distributed systems engineering experience
  • Deep Kubernetes expertise for stateful workload management
  • Experience building high-throughput load balancers or proxy systems
  • Strong Python and FastAPI skills
  • Background in ML infrastructure or model serving platforms is preferred

Benefits

  • Remote-first culture
  • Equity in a high-growth ML infrastructure company
  • Health, dental, and vision
  • $4,000 annual professional development budget
  • On-call compensation above base

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure AI & Machine Learning Engineers

Skills

python kubernetes docker aws fastapi

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Solutions Engineer, Observe

Snowflake

Full-time Senior

Senior Release Manager

Agility Robotics

Full-time Senior

Senior AI R&D Engineer – Model Training (f/m/d)

Aleph Alpha

Full-time Senior

Solutions Engineer

Pinecone

Full-time

Solutions Architect - Digital Native Business (Geo)

Databricks

Full-time

Staff Data Scientist

Gradientai

Full-time Staff

Job Type

Full-time

Level

Senior

Language

English

Salary Range

$130k – $160k / year

AI Expertise

MLOps & AI Infrastructure AI & Machine Learning Engineers

Skills

python kubernetes docker aws fastapi

Ready to apply for this role?

Create a free talent account.

  • Find AI roles in one place
  • Get AI-matched job recommendations
  • Reply to companies that message you
  • Build your public AI talent profile
Create free profile and apply Log in

This role is closed. Create your free profile to apply to similar open roles.

Don't miss the next one: get matched to new roles like this.

Create free profile