AI Multi-Tenant GPU Scheduler Developer

Aevix Cloud

Remote · $160k – $200k / year · Full-time · Posted 4 months ago

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

Sharing GPU clusters across dozens of teams sounds simple until a rogue training job claims 80% of capacity and blocks everyone else's experiments for six hours. Multi-tenant GPU scheduling is a fairness problem, a packing problem, and a preemption problem simultaneously — and getting it wrong costs both money and team morale. We serve 200+ enterprise ML teams who share GPU infrastructure. Our scheduling layer assigns jobs to GPUs, enforces per-team quotas, preempts low-priority workloads when high-priority jobs arrive, and checkpoints long-running jobs so preemption doesn't mean losing days of training progress. The current scheduler uses a modified fair-share algorithm that has been patched so many times it is fragile. Your first project is a full redesign using a priority-weighted fair queuing approach with dynamic quota rebalancing based on actual utilization. Measurable target: reduce average queue wait time by 40% without increasing cost.

Fully remote.

Responsibilities

Redesign the multi-tenant GPU scheduler using priority-weighted fair queuing
Implement dynamic quota rebalancing based on real-time team utilization
Build preemption and checkpointing logic for long-running training jobs
Integrate scheduler with Kubernetes custom resource definitions
Create fairness monitoring dashboards visible to platform users
Benchmark scheduler performance against current system and academic baselines

Requirements

6+ years in platform or infrastructure engineering with scheduling experience
Deep knowledge of Kubernetes scheduling, custom schedulers, and resource management
Experience with distributed systems, fairness algorithms, or resource allocation
Strong Python skills and comfort with Go for Kubernetes operator work
Track record of designing systems that handle adversarial usage patterns