AI Speculative Decoding Optimization Engineer

Emrex AI

Remote · $160k – $200k / year · Full-time · Posted 4 months ago

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

LLM inference latency is the core product metric for every API we serve. When a legal research tool waits 8 seconds for a response, lawyers switch tabs. When a coding assistant takes 4 seconds to autocomplete, developers lose the flow state. We've deployed speculative decoding, and it's delivering 2.1x throughput improvement on our standard workloads. The theoretical maximum for our draft model configuration is 3.8x. The gap is where you come in. Speculative decoding is still research-grade in production: acceptance rates vary dramatically by prompt type, draft model quality degrades on domain-specific vocabulary, and the batching strategy that's optimal for spec-dec is different from the one that maximizes standard autoregressive throughput. You'll own the optimization work: tuning draft model selection per workload type, implementing dynamic speculation depth based on real-time acceptance rate feedback, and building the infrastructure that switches between spec-dec and standard decoding based on request characteristics. The inference team is four engineers. This is a senior IC role requiring deep knowledge of transformer inference internals — not a general MLOps role.

Fully remote.

Responsibilities

Optimize speculative decoding acceptance rates from 2.1x toward theoretical 3.8x maximum
Design workload-adaptive draft model selection based on prompt domain classification
Implement dynamic speculation depth using real-time acceptance rate feedback
Build infrastructure for seamless switching between spec-dec and standard decoding
Develop batching strategies optimized for speculative decoding workloads
Benchmark improvements against latency, throughput, and cost per token metrics

Requirements

6+ years in ML infrastructure with LLM inference optimization focus
Deep expertise in speculative decoding, KV cache management, and transformer inference
Strong Python and PyTorch skills with C++/CUDA experience
Experience with vLLM, TGI, TensorRT-LLM, or equivalent serving frameworks
Track record improving LLM serving performance by >50% through systems changes