AI Speculative Decoding Implementation Engineer

Lontex Tech

Remote · $160k – $200k / year · Full-time · Posted 4 months ago

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

Speculative decoding is one of the most impactful LLM serving optimizations of the past two years: use a small draft model to generate candidate tokens, verify them in parallel with the large model, and achieve 2-4x throughput improvement without any accuracy loss. The papers make it look straightforward. The production implementation is not. Our serving infrastructure handles 140 million LLM API calls per month across 12 different model families. We want to implement speculative decoding across our entire serving fleet — but the implementation details are genuinely hard: draft model selection per request type, acceptance criteria tuning, handling of streaming responses where rejected tokens affect perceived latency, and the monitoring needed to detect when the draft model is hurting rather than helping. This is a senior IC role requiring both deep ML systems knowledge and production serving experience. You'll implement the speculative decoding system, design the experiments that tune its parameters, and build the monitoring that tracks its impact across request types. The target: 2.5x throughput improvement at constant quality on our top-5 served models.

Fully remote.

Responsibilities

Implement speculative decoding across production LLM serving infrastructure
Design draft model selection strategy per request type and model family
Build acceptance criteria tuning pipeline with continuous quality validation
Handle streaming response edge cases where token rejection affects perceived latency
Create monitoring tracking throughput gains, quality impact, and draft acceptance rate
Run A/B experiments validating speedup vs. quality trade-offs across model families

Requirements

6+ years in ML systems or LLM inference engineering
Deep understanding of transformer inference, KV caching, and batching strategies
Experience with speculative decoding or other inference acceleration techniques
Strong Python and PyTorch skills with GPU optimization knowledge
Track record implementing low-level serving optimizations in production