AI Speculative Decoding Implementation Engineer

About the role

Responsibilities

• Implement speculative decoding across production LLM serving infrastructure
• Design draft model selection strategy per request type and model family
• Build acceptance criteria tuning pipeline with continuous quality validation
• Handle streaming response edge cases where token rejection affects perceived latency
• Create monitoring tracking throughput gains, quality impact, and draft acceptance rate
• Run A/B experiments validating speedup vs. quality trade-offs across model families

Requirements

• 6+ years in ML systems or LLM inference engineering
• Deep understanding of transformer inference, KV caching, and batching strategies
• Experience with speculative decoding or other inference acceleration techniques
• Strong Python and PyTorch skills with GPU optimization knowledge
• Track record implementing low-level serving optimizations in production

Benefits

• Fully remote with flexible schedule
• Top-of-market compensation
• Comprehensive health benefits
• $5,000 annual equipment and learning budget
• Equity participation

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.