Profile cover
C

AI On-Call Fatigue Detection and Alert Routing Developer

Caleb Rodriguez

Full-time · Job · Chicago

About the role

On-call engineers are woken up at 2am to respond to alerts. If those alerts are noisy — firing on transient issues, firing on things that resolve themselves, firing on P3 issues that don't warrant a page — the on-call engineer is being trained to dismiss alerts and delay response. Alert fatigue is the precursor to the missed page that becomes the major incident. The solution isn't fewer alerts — it's smarter routing: alerts delivered to the right person, at the right urgency level, with enough context to respond rather than investigate. Our observability platform handles 12 million alert events per day across 45 enterprise clients. Alert routing intelligence is the layer we're building on top of our existing correlation and deduplication features. We need to go beyond deduplication to on-call load awareness: routing alerts based on current on-call engineer workload, time of day relative to their schedule, the historical response effectiveness of different engineers for different alert types, and the severity trajectory of the triggering condition. You'll build the intelligent alert routing system: computing on-call engineer load scores in real time, implementing routing policy logic that balances severity, expertise match, and fatigue signals, and building the routing effectiveness feedback loop that improves routing accuracy from outcome data.

Responsibilities

  • Build on-call engineer load scoring computing real-time fatigue state from recent alert volume and response burden
  • Implement expertise-based routing matching alert types to engineers with highest historical response effectiveness
  • Develop time-of-day sensitivity adjusting routing thresholds based on engineer local time and shift schedule
  • Create routing policy engine balancing severity urgency, fatigue risk, and expertise match
  • Build routing effectiveness feedback loop improving policies from time-to-acknowledge and resolution outcome data
  • Monitor mean time to acknowledge improvement as primary routing quality metric

Requirements

  • 3+ years in SRE, platform engineering, or observability tooling development
  • Strong Python and SQL skills with alerting systems, on-call tooling, or incident management platforms
  • Understanding of alert fatigue, on-call scheduling patterns, and incident response workflows
  • Experience building scoring systems that drive operational workflow decisions in real time
  • Ability to work with SRE teams to validate routing logic against real on-call experience

Benefits

  • Fully remote with flexible schedule
  • Health and dental insurance
  • $3,000 annual professional development budget
  • 401(k) with match
  • Equity options

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure Automation & AI Workflows

Skills

python sql fastapi docker statistics

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Technical Writer

RemoteAI company

Part-time

AI Resident UK/EMEA

RemoteAI company

Full-time

Software Engineer, Applications (Scientific Development Solutions) (High Seniority)

RemoteAI company

Full-time Senior

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

PhD Autonomy Engineer Intern - Computer Vision / Deep Learning Summer 2027

RemoteAI company

Full-time Intern

Job Type

Full-time

Level

Mid-Level

Language

English

Salary Range

$100k – $130k / year

AI Expertise

MLOps & AI Infrastructure Automation & AI Workflows

Skills

python sql fastapi docker statistics pandas

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile