Profile cover
C

AI Distributed Tracing Intelligence Developer

Caleb Rodriguez

Full-time · Job · New_York

About the role

A distributed trace with 3,000 spans is technically observable but practically unreadable. On-call engineers staring at a waterfall of spans during an incident don't need more data — they need the right data, highlighted, with the anomalous spans surfaced automatically. That's what we build. Our platform ingests distributed traces from 160 engineering teams and applies ML to make them intelligible: clustering similar error patterns, detecting latency anomalies in specific service interactions, and building the baseline models that distinguish a slow Tuesday from an actual incident. The signal-to-noise improvement is measurable — our clients resolve P1 incidents 40% faster after deploying our trace intelligence layer. You'll own the span anomaly detection and pattern clustering features. The core challenge is doing this at scale: 40 million spans per minute, with sub-second detection latency and high enough precision that on-call engineers trust the highlights instead of ignoring them.

Responsibilities

  • Build span anomaly detection models that distinguish incidents from normal variance
  • Implement trace pattern clustering to group similar error signatures across services
  • Design per-service baseline models that adapt to traffic seasonality
  • Optimize detection pipeline for 40 million spans per minute with sub-second latency
  • Create precision/recall dashboards so on-call teams can tune alert thresholds
  • Collaborate with platform engineers on streaming pipeline architecture

Requirements

  • 3+ years in ML engineering or data engineering with observability domain experience
  • Experience with streaming data processing at high throughput
  • Strong Python and SQL skills
  • Familiarity with distributed tracing concepts: spans, traces, service dependencies
  • Statistical reasoning for anomaly detection and threshold calibration

Benefits

  • Fully remote
  • Health and dental
  • $3,000 professional development budget
  • Home office stipend
  • Quarterly bonus

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure Automation & AI Workflows

Skills

python fastapi sql docker aws

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

Technical Writer

RemoteAI company

Part-time

Staff Application Security Engineer

RemoteAI company

Full-time Staff

AI Resident UK/EMEA

RemoteAI company

Full-time

Devops Engineer, US

RemoteAI company

Full-time

Job Type

Full-time

Level

Mid-Level

Language

English

Salary Range

$100k – $130k / year

AI Expertise

MLOps & AI Infrastructure Automation & AI Workflows

Skills

python fastapi sql docker aws spark

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile