AI Inference Throughput Benchmarking Engineer

Aevix Cloud

Remote · $130k – $160k / year · Full-time · Posted 4 months ago

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

GPU vendors publish benchmark numbers. Reality disagrees. A model that achieves 1,200 tokens per second on a pristine A100 in a controlled benchmark might deliver 400 tokens per second in production under concurrent load, batching constraints, memory pressure from co-tenanted workloads, and the specific attention pattern distributions of real user queries. Understanding the gap is how you optimize it. We operate GPU inference infrastructure for 60 enterprise clients serving 2 billion monthly inference requests. Our internal benchmarking has gaps: we measure throughput on standardized test sets, not under the actual load profiles our clients generate. The result is that optimization work that looks impactful in benchmark runs sometimes has negligible real-world effect. You'll build the production-realistic inference benchmarking system: capturing real traffic distributions from client workloads, designing benchmark scenarios that reproduce those distributions, and building the continuous benchmark pipeline that measures how infrastructure changes and model updates affect real-world throughput. The output is the evidence that drives hardware purchasing, capacity planning, and optimization prioritization decisions worth tens of millions of dollars annually.

Fully remote.

Responsibilities

Design production-realistic inference benchmarks capturing real client traffic distribution characteristics
Build continuous benchmarking pipeline measuring throughput impact of infrastructure and model changes
Implement load generation system reproducing concurrent query patterns, batch sizes, and sequence length distributions
Create comparison reporting isolating the effect of specific changes on real-world inference throughput
Develop hardware efficiency metrics comparing cost-per-token across GPU configurations
Drive capacity planning decisions with throughput projections under modeled load growth

Requirements

6+ years in ML infrastructure, GPU systems, or performance engineering
Deep understanding of transformer inference mechanics, batching strategies, and GPU utilization
Experience designing rigorous benchmarking methodology for distributed ML systems
Strong Python skills with profiling and performance analysis tooling
Track record using benchmarking evidence to drive infrastructure investment decisions