Profile cover
E

AI Inference Pipeline Architect

Evan Ferrari

Full-time · Job · Chicago

About the role

Our GPU clusters handle 2.3 billion inference requests per month, and that number doubles every quarter. The bottleneck isn't compute — it's orchestration. We need an architect who can redesign our inference serving layer from the ground up, moving from a monolithic gateway to a distributed mesh that routes requests based on model size, latency SLA, and cost. You'll own the full inference stack: model packaging, container orchestration, autoscaling policies, and real-time monitoring. Your decisions will directly impact our cost-per-inference metric, which the board tracks monthly. The current system wastes 34% of GPU cycles on idle warm-up — fixing that alone would save $1.2M annually. This is a staff-level IC role reporting to the VP of Infrastructure. No people management required, but you'll mentor two mid-level engineers and set technical direction for the inference team.

Responsibilities

  • Redesign the inference serving architecture to support multi-model routing
  • Build autoscaling policies that balance latency SLAs against GPU cost
  • Own model packaging standards from training output to production container
  • Instrument the full inference path with latency, throughput, and error metrics
  • Partner with the ML team to define model deployment contracts
  • Evaluate and integrate new serving frameworks as they mature

Requirements

  • 7+ years building production ML infrastructure at scale
  • Deep expertise in Kubernetes, Docker, and GPU scheduling
  • Hands-on experience with model serving frameworks like Triton, vLLM, or TGI
  • Strong understanding of networking, load balancing, and gRPC
  • Track record of reducing infrastructure costs by 20%+ through architecture changes

Benefits

  • Fully remote — work from anywhere with reliable internet
  • $5,000 annual learning budget for conferences and courses
  • Equity package with 4-year vesting
  • Unlimited PTO with 3-week minimum encouraged
  • Home office setup stipend of $2,500

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure AI & Machine Learning Engineers

Skills

python kubernetes docker aws pytorch

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

AI Resident UK/EMEA

RemoteAI company

Full-time

Senior AI Researcher- Pre-training (f/m/d)

RemoteAI company

Full-time Senior

Senior AI R&D Engineer – Model Training (f/m/d)

RemoteAI company

Full-time Senior

Devops Engineer, US

RemoteAI company

Full-time

AI Engineer

RemoteAI company

Full-time

Job Type

Full-time

Level

Senior

Language

English

Salary Range

$160k – $200k / year

AI Expertise

MLOps & AI Infrastructure AI & Machine Learning Engineers

Skills

python kubernetes docker aws pytorch mlflow

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile