Profile cover
H

AI API Gateway Performance Engineer

Hudson Larsson

Full-time · Job · Los_Angeles

About the role

Every AI model our clients deploy goes through our API gateway — 890 million requests per month across 200+ models. The gateway handles authentication, rate limiting, request validation, model routing, response caching, and usage metering. When it's slow, every model behind it is slow. When it's down, everything is down. We're looking for an engineer obsessed with latency. Our p50 is 12ms overhead; our p99 is 180ms. That p99 is unacceptable for real-time applications, and the tail latency comes from three sources: cold cache misses, garbage collection pauses, and slow model routing decisions. You'll profile, diagnose, and fix these bottlenecks while adding new gateway features that clients are asking for: request transformation, response streaming, and multi-model fan-out. The platform team is five engineers. You'll own gateway performance as a discipline — setting SLOs, building performance regression tests, and raising the alarm when a code change threatens latency guarantees.

Responsibilities

  • Profile and eliminate tail latency in the AI API gateway
  • Build performance regression testing into the CI/CD pipeline
  • Implement request transformation and response streaming features
  • Design caching strategies that reduce cold-miss latency spikes
  • Set and monitor latency SLOs across all gateway endpoints
  • Optimize garbage collection and memory management patterns

Requirements

  • 3+ years in backend performance engineering or systems programming
  • Experience profiling and optimizing high-throughput API services
  • Strong understanding of caching, connection pooling, and concurrency
  • Proficiency with Docker, Kubernetes, and cloud infrastructure
  • Comfort with performance measurement tools and methodology

Benefits

  • Fully remote with async-first culture
  • Health, dental, and vision coverage
  • $3,000 annual equipment budget
  • Stock options
  • Flexible PTO policy

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure Automation & AI Workflows

Skills

python docker kubernetes aws fastapi

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Principal Engineer

RemoteAI company

Full-time Principal

Staff Application Security Engineer

RemoteAI company

Full-time Staff

AI Resident UK/EMEA

RemoteAI company

Full-time

Devops Engineer, US

RemoteAI company

Full-time

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

Job Type

Full-time

Level

Mid-Level

Language

English

Salary Range

$100k – $130k / year

AI Expertise

MLOps & AI Infrastructure Automation & AI Workflows

Skills

python docker kubernetes aws fastapi mlflow

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile