Profile cover
E

AI GPU Memory Defragmentation Engineer

Evan Ferrari

Full-time · Job · Los_Angeles

About the role

GPU memory fragmentation is the silent killer of training throughput. A cluster that should deliver 94% MFU sits at 71% because memory allocations from terminated jobs leave unusable gaps, forcing new jobs to spin up on fresh nodes rather than reusing partially available ones. At the scale we operate — 18,000 A100s across five data centers — every percentage point of MFU translates to millions of dollars per year. We need a staff engineer who understands GPU memory management at the allocator level: how PyTorch's caching allocator fragments across training runs, how CUDA driver memory interacts with framework-level allocation, and what defragmentation strategies work in practice without introducing latency spikes that hurt running jobs. This is a combination of systems programming, performance engineering, and distributed infrastructure. You'll own the memory management layer of our ML infrastructure: designing defragmentation policies, building monitoring that surfaces fragmentation before it impacts jobs, and implementing the scheduling changes that let us pack jobs more efficiently. The platform team is 12 engineers; this is a staff-level IC role with no management scope.

Responsibilities

  • Design GPU memory defragmentation policies for multi-tenant training clusters
  • Build fragmentation monitoring and alerting integrated with cluster scheduler
  • Implement memory-aware job scheduling to maximize cluster MFU
  • Analyze memory allocation patterns across PyTorch, JAX, and TF workloads
  • Profile and eliminate allocation hot spots in high-throughput training pipelines
  • Benchmark defragmentation impact on training throughput and job startup latency

Requirements

  • 7+ years in ML infrastructure or systems programming
  • Deep expertise in GPU memory management: CUDA allocators, driver memory, framework caches
  • Experience with cluster schedulers (SLURM, Kubernetes, or custom) at GPU scale
  • Strong C++/Python skills with performance profiling experience
  • Track record improving cluster efficiency by 10%+ through infrastructure changes

Benefits

  • Fully remote with complete schedule autonomy
  • Top-of-market total compensation
  • Significant equity package
  • Comprehensive health benefits
  • Annual $8,000 equipment and learning budget

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure

Skills

python kubernetes docker aws pytorch

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

AI Resident UK/EMEA

RemoteAI company

Full-time

Principal Engineer

RemoteAI company

Full-time Principal

Staff Application Security Engineer

RemoteAI company

Full-time Staff

Devops Engineer, US

RemoteAI company

Full-time

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

Job Type

Full-time

Level

Senior

Language

English

Salary Range

$200k+ / year

AI Expertise

MLOps & AI Infrastructure

Skills

python kubernetes docker aws pytorch fastapi

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile