Profile cover
C

AI Alert Correlation and Deduplication Engineer

Caleb Rodriguez

Full-time · Job · Chicago

About the role

Alert fatigue is an observability epidemic. A service experiencing a single network partition can generate 800 alerts across 40 different monitors in 3 minutes. The on-call engineer who needs to respond sees 800 pages, not 1. By the time they've processed the first dozen, the pattern is obvious — but the time lost to triaging the noise is time not spent on the fix. We serve 70 engineering organizations with AI-powered observability capabilities. Alert correlation and deduplication — automatically grouping related alerts into unified incidents, suppressing redundant notifications, and surfacing the root signal from the noise — is the highest-demand feature we've shipped. Our current implementation uses rule-based correlation. It works for known failure patterns. It fails for novel outage topologies. You'll redesign the correlation engine with ML: building unsupervised clustering that groups alerts by causal relationship rather than predefined rules, implementing temporal sequence modeling that understands typical alert propagation patterns, and developing the suppression logic that decides which alerts are informative given what's already been notified. The KPI is mean time to acknowledge for true incidents — if that doesn't drop, the feature isn't working.

Responsibilities

  • Redesign alert correlation engine using ML clustering grounded in causal relationship rather than fixed rules
  • Implement temporal sequence models capturing typical alert propagation patterns across service topologies
  • Build suppression logic deciding which alerts add information given prior notifications in an incident
  • Develop incident severity estimation from correlated alert clusters
  • Create correlation quality evaluation using on-call engineer feedback and incident postmortem data
  • Monitor MTTA improvement as the primary production success metric

Requirements

  • 6+ years in observability, SRE, or ML infrastructure engineering
  • Experience building alert correlation, incident management, or AIOps systems
  • Strong Python and ML skills with time series and sequence modeling experience
  • Deep understanding of distributed system failure modes and alert propagation
  • Track record improving on-call experience through data-driven automation

Benefits

  • Fully remote with flexible schedule
  • Comprehensive health benefits
  • $4,000 annual professional development budget
  • Equity participation
  • On-call compensation for system incidents

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure AI Product & Strategy

Skills

python pytorch docker kubernetes fastapi

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Principal Engineer

RemoteAI company

Full-time Principal

Technical Writer

RemoteAI company

Part-time

Senior AI R&D Engineer – Model Training (f/m/d)

RemoteAI company

Full-time Senior

Senior AI Researcher- Pre-training (f/m/d)

RemoteAI company

Full-time Senior

Staff Application Security Engineer

RemoteAI company

Full-time Staff

Job Type

Full-time

Level

Senior

Language

English

Salary Range

$130k – $160k / year

AI Expertise

MLOps & AI Infrastructure AI Product & Strategy

Skills

python pytorch docker kubernetes fastapi sql

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile