Profile cover
O

AI Speculative Decoding Optimization Engineer

Owen Goldberg

Full-time · Job · Los_Angeles

About the role

LLM inference latency is the core product metric for every API we serve. When a legal research tool waits 8 seconds for a response, lawyers switch tabs. When a coding assistant takes 4 seconds to autocomplete, developers lose the flow state. We've deployed speculative decoding, and it's delivering 2.1x throughput improvement on our standard workloads. The theoretical maximum for our draft model configuration is 3.8x. The gap is where you come in. Speculative decoding is still research-grade in production: acceptance rates vary dramatically by prompt type, draft model quality degrades on domain-specific vocabulary, and the batching strategy that's optimal for spec-dec is different from the one that maximizes standard autoregressive throughput. You'll own the optimization work: tuning draft model selection per workload type, implementing dynamic speculation depth based on real-time acceptance rate feedback, and building the infrastructure that switches between spec-dec and standard decoding based on request characteristics. The inference team is four engineers. This is a senior IC role requiring deep knowledge of transformer inference internals — not a general MLOps role.

Responsibilities

  • Optimize speculative decoding acceptance rates from 2.1x toward theoretical 3.8x maximum
  • Design workload-adaptive draft model selection based on prompt domain classification
  • Implement dynamic speculation depth using real-time acceptance rate feedback
  • Build infrastructure for seamless switching between spec-dec and standard decoding
  • Develop batching strategies optimized for speculative decoding workloads
  • Benchmark improvements against latency, throughput, and cost per token metrics

Requirements

  • 6+ years in ML infrastructure with LLM inference optimization focus
  • Deep expertise in speculative decoding, KV cache management, and transformer inference
  • Strong Python and PyTorch skills with C++/CUDA experience
  • Experience with vLLM, TGI, TensorRT-LLM, or equivalent serving frameworks
  • Track record improving LLM serving performance by >50% through systems changes

Benefits

  • Fully remote with flexible schedule
  • Top-of-market compensation
  • Equity package
  • Comprehensive health benefits
  • Access to frontier model weights for benchmarking

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure AI & Machine Learning Engineers

Skills

python pytorch docker kubernetes aws

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

AI Resident UK/EMEA

RemoteAI company

Full-time

Senior AI Researcher- Pre-training (f/m/d)

RemoteAI company

Full-time Senior

Senior AI R&D Engineer – Model Training (f/m/d)

RemoteAI company

Full-time Senior

Devops Engineer, US

RemoteAI company

Full-time

AI Engineer

RemoteAI company

Full-time

Job Type

Full-time

Level

Senior

Language

English

Salary Range

$160k – $200k / year

AI Expertise

MLOps & AI Infrastructure AI & Machine Learning Engineers

Skills

python pytorch docker kubernetes aws fastapi

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile