AI Distributed Tracing Intelligence Developer

Ulvex Systems

Remote · $100k – $130k / year · Full-time · Posted 5 months ago

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

A distributed trace with 3,000 spans is technically observable but practically unreadable. On-call engineers staring at a waterfall of spans during an incident don't need more data — they need the right data, highlighted, with the anomalous spans surfaced automatically. That's what we build. Our platform ingests distributed traces from 160 engineering teams and applies ML to make them intelligible: clustering similar error patterns, detecting latency anomalies in specific service interactions, and building the baseline models that distinguish a slow Tuesday from an actual incident. The signal-to-noise improvement is measurable — our clients resolve P1 incidents 40% faster after deploying our trace intelligence layer. You'll own the span anomaly detection and pattern clustering features. The core challenge is doing this at scale: 40 million spans per minute, with sub-second detection latency and high enough precision that on-call engineers trust the highlights instead of ignoring them.

Fully remote.

Responsibilities

Build span anomaly detection models that distinguish incidents from normal variance
Implement trace pattern clustering to group similar error signatures across services
Design per-service baseline models that adapt to traffic seasonality
Optimize detection pipeline for 40 million spans per minute with sub-second latency
Create precision/recall dashboards so on-call teams can tune alert thresholds
Collaborate with platform engineers on streaming pipeline architecture

Requirements

3+ years in ML engineering or data engineering with observability domain experience
Experience with streaming data processing at high throughput
Strong Python and SQL skills
Familiarity with distributed tracing concepts: spans, traces, service dependencies
Statistical reasoning for anomaly detection and threshold calibration