AI Inference Pipeline Architect

Aevix Cloud

Remote · $160k – $200k / year · Full-time · Posted 5 months ago

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

Our GPU clusters handle 2.3 billion inference requests per month, and that number doubles every quarter. The bottleneck isn't compute — it's orchestration. We need an architect who can redesign our inference serving layer from the ground up, moving from a monolithic gateway to a distributed mesh that routes requests based on model size, latency SLA, and cost. You'll own the full inference stack: model packaging, container orchestration, autoscaling policies, and real-time monitoring. Your decisions will directly impact our cost-per-inference metric, which the board tracks monthly. The current system wastes 34% of GPU cycles on idle warm-up — fixing that alone would save $1.2M annually. This is a staff-level IC role reporting to the VP of Infrastructure. No people management required, but you'll mentor two mid-level engineers and set technical direction for the inference team.

Fully remote.

Responsibilities

Redesign the inference serving architecture to support multi-model routing
Build autoscaling policies that balance latency SLAs against GPU cost
Own model packaging standards from training output to production container
Instrument the full inference path with latency, throughput, and error metrics
Partner with the ML team to define model deployment contracts
Evaluate and integrate new serving frameworks as they mature

Requirements

7+ years building production ML infrastructure at scale
Deep expertise in Kubernetes, Docker, and GPU scheduling
Hands-on experience with model serving frameworks like Triton, vLLM, or TGI
Strong understanding of networking, load balancing, and gRPC
Track record of reducing infrastructure costs by 20%+ through architecture changes