AI Inference Request Batching Optimizer

About the role

Responsibilities

• Design adaptive batch size selection based on real-time request arrival rate and per-request latency budgets
• Implement per-endpoint batching policies learned from historical traffic patterns and SLA requirements
• Build dynamic batching window control balancing batch assembly time against tail latency
• Develop batching strategy evaluation measuring throughput-latency Pareto improvement against fixed-window baseline
• Create batching simulation framework for testing policy changes against production traffic replay
• Monitor GPU utilization and p99 latency as primary optimization target metrics

Requirements

• 6+ years in ML infrastructure or systems engineering with inference serving optimization experience
• Deep understanding of GPU batching dynamics, CUDA execution, and inference serving framework internals
• Strong Python skills with TensorRT, Triton Inference Server, or similar inference stack experience
• Experience with adaptive systems that learn optimal parameters from streaming operational feedback
• Track record demonstrating GPU utilization or throughput improvements through inference optimization

Benefits

• Fully remote with flexible schedule
• Comprehensive health benefits
• $4,000 annual professional development budget
• Equity participation
• Performance bonus tied to GPU utilization improvement metrics

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.