Profile cover
Z

AI Flaky Test Detection and Classification Developer

Zoe Levy

Full-time · Job · New_York

About the role

A test that sometimes passes and sometimes fails for no obvious reason is a flaky test — and it's the bane of every CI/CD pipeline. Engineering teams waste hours re-running builds hoping flaky tests will pass, lose trust in their test suite, and start ignoring test failures because "it's probably just flaky." Our detection system identifies flaky tests statistically and classifies their root cause. You'll build the detection and classification pipeline: analyzing test run histories to identify non-deterministic pass/fail patterns, classifying flakiness causes (timing dependencies, shared state, external service dependencies, random data), and prioritizing which flaky tests to fix first based on their impact on build times and developer trust. Each classified test gets a remediation recommendation. The quality engineering team is four engineers. You'll own the detection and classification module, working with the reliability team who fixes the highest-priority flaky tests and the platform team who manages the CI infrastructure.

Responsibilities

  • Build statistical detection of non-deterministic test pass/fail patterns
  • Classify flakiness root causes: timing, state, external services, randomness
  • Prioritize flaky tests by build impact and developer trust cost
  • Generate remediation recommendations for each flakiness class
  • Track flaky test remediation progress over time
  • Build dashboards showing flakiness trends across the test suite

Requirements

  • 3+ years in software quality engineering or CI/CD platform experience
  • Understanding of common flaky test root causes and debugging approaches
  • Strong Python and statistical analysis skills
  • Experience with test execution data and build analytics
  • Ability to communicate technical findings to engineering teams

Benefits

  • Fully remote with flexible schedule
  • Health, dental, and vision insurance
  • $3,000 annual learning budget
  • Stock options
  • Fix one of the most universally hated problems in software development

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure Automation & AI Workflows

Skills

python docker sql aws fastapi

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

Technical Writer

RemoteAI company

Part-time

AI Resident UK/EMEA

RemoteAI company

Full-time

Devops Engineer, US

RemoteAI company

Full-time

Software Engineer, Applications (Scientific Development Solutions) (High Seniority)

RemoteAI company

Full-time Senior

Job Type

Full-time

Level

Mid-Level

Language

English

Salary Range

$100k – $130k / year

AI Expertise

MLOps & AI Infrastructure Automation & AI Workflows

Skills

python docker sql aws fastapi statistics

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile