Profile cover
O

AI Prompt Caching Engineer

Emrex AI

Remote · $160k – $200k / year · Full-time · Posted 5 months ago

No longer accepting applications

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

Our clients send 5 million LLM API calls per day, and 40% of them share identical system prompts. Without caching, every call reprocesses the same 2,000-token system prompt — wasting compute, increasing latency, and inflating costs. Our prompt caching layer eliminates this redundancy, reducing client costs by 35% and cutting P50 latency by 200ms. You'll own the caching infrastructure: KV-cache management for shared prompt prefixes, cache invalidation strategies when prompts are updated, the routing layer that directs requests to warm cache instances, and the analytics system that shows clients exactly how much caching saves them. Prompt caching has unique challenges compared to traditional web caching. Cache keys are semantic (similar prompts, not identical ones, should share cache). Cache values are enormous (KV states for 100K token contexts). Eviction policies must consider both recency and prompt similarity. You'll solve these problems at infrastructure scale.

Fully remote.

Responsibilities

  • Design and optimize KV-cache management for shared prompt prefix reuse
  • Build cache invalidation strategies handling prompt updates without stale results
  • Develop routing layer directing requests to warm cache instances
  • Create analytics dashboards showing per-client caching savings
  • Scale caching infrastructure for 5M+ daily API calls with sub-200ms latency gains

Requirements

  • 5+ years in infrastructure engineering with caching system experience
  • Deep understanding of LLM inference and KV-cache mechanics
  • Kubernetes and Docker expertise for distributed cache management
  • Strong Python and FastAPI skills for routing and analytics
  • Experience with high-throughput, low-latency serving systems

Benefits

  • Fully remote — any time zone
  • Senior equity package
  • $3,500 annual conference budget
  • Solving a core LLM infrastructure efficiency problem

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure

Skills

python fastapi docker kubernetes aws

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Technical Support Engineer - APAC

Hyperbolic

Contract

Senior Engineer, Exposure & Vulnerability Management

Recursion

Full-time Senior

Software Engineer

Turing

Full-time

Software Engineer, GPU Infrastructure

Cohere

Full-time Staff

Senior Product Manager: Extensible Compute

Snowflake

Full-time Senior

Senior Solution Engineer, Partnerships - Brazil

Snowflake

Full-time Senior

Job Type

Full-time

Level

Senior

Language

English

Salary Range

$160k – $200k / year

AI Expertise

MLOps & AI Infrastructure

Skills

python fastapi docker kubernetes aws sql

Ready to apply for this role?

Create a free talent account.

  • Find AI roles in one place
  • Get AI-matched job recommendations
  • Reply to companies that message you
  • Build your public AI talent profile
Create free profile and apply Log in

This role is closed. Create your free profile to apply to similar open roles.

Don't miss the next one: get matched to new roles like this.

Create free profile