Profile cover
O

AI Prompt Caching Engineer

Owen Goldberg

Full-time · Job · London

About the role

Our clients send 5 million LLM API calls per day, and 40% of them share identical system prompts. Without caching, every call reprocesses the same 2,000-token system prompt — wasting compute, increasing latency, and inflating costs. Our prompt caching layer eliminates this redundancy, reducing client costs by 35% and cutting P50 latency by 200ms. You'll own the caching infrastructure: KV-cache management for shared prompt prefixes, cache invalidation strategies when prompts are updated, the routing layer that directs requests to warm cache instances, and the analytics system that shows clients exactly how much caching saves them. Prompt caching has unique challenges compared to traditional web caching. Cache keys are semantic (similar prompts, not identical ones, should share cache). Cache values are enormous (KV states for 100K token contexts). Eviction policies must consider both recency and prompt similarity. You'll solve these problems at infrastructure scale.

Responsibilities

  • Design and optimize KV-cache management for shared prompt prefix reuse
  • Build cache invalidation strategies handling prompt updates without stale results
  • Develop routing layer directing requests to warm cache instances
  • Create analytics dashboards showing per-client caching savings
  • Scale caching infrastructure for 5M+ daily API calls with sub-200ms latency gains

Requirements

  • 5+ years in infrastructure engineering with caching system experience
  • Deep understanding of LLM inference and KV-cache mechanics
  • Kubernetes and Docker expertise for distributed cache management
  • Strong Python and FastAPI skills for routing and analytics
  • Experience with high-throughput, low-latency serving systems

Benefits

  • Fully remote — any time zone
  • Senior equity package
  • $3,500 annual conference budget
  • Solving a core LLM infrastructure efficiency problem

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure

Skills

python fastapi docker kubernetes aws

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Principal Engineer

RemoteAI company

Full-time Principal

Staff Application Security Engineer

RemoteAI company

Full-time Staff

AI Resident UK/EMEA

RemoteAI company

Full-time

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

Technical Writer

RemoteAI company

Part-time

Job Type

Full-time

Level

Senior

Language

English

Salary Range

$160k – $200k / year

AI Expertise

MLOps & AI Infrastructure

Skills

python fastapi docker kubernetes aws sql

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile