Profile cover
M

AI Feature Flag Impact Assessment Developer

Mia Levy

Full-time · Job · Los_Angeles

About the role

Feature flags let teams ship code gradually — exposing a new ML model or feature to 1% of traffic, then 10%, then 100%, with the ability to roll back instantly if something goes wrong. The question 'is something going wrong?' requires continuous impact assessment: monitoring model performance, user behavior, and system metrics across the flagged and unflagged populations to detect problems before they reach full rollout scale. Our deployment platform manages 2,800 active feature flags across 45 enterprise clients. Feature flag impact assessment is the analytics layer that makes gradual rollout trustworthy. We have basic metric comparison. We don't have the statistical rigor that lets engineering and product teams make confident rollout decisions: proper sequential testing that maintains error rates across continuous monitoring, segmentation analysis that catches problems affecting specific user cohorts, and automated anomaly detection that surfaces impact changes without requiring someone to watch dashboards. You'll build the impact assessment system: implementing sequential statistical testing for continuous monitoring, building segmentation analysis detecting cohort-specific impacts, and creating the rollout recommendation engine that synthesizes assessment signals into a clear proceed-pause-rollback decision.

Responsibilities

  • Implement sequential statistical testing maintaining type-I error control across continuous flag monitoring
  • Build segmentation analysis detecting impact differences across user cohort and traffic segment dimensions
  • Develop anomaly detection surfacing metric changes that don't reach statistical significance but warrant attention
  • Create rollout recommendation engine synthesizing assessment signals into proceed-pause-rollback decisions
  • Build impact assessment APIs consumed by deployment workflow gates and engineering dashboards
  • Monitor assessment accuracy comparing recommended rollout decisions to realized post-rollout outcomes

Requirements

  • 3+ years in data science, experimentation, or A/B testing platform engineering
  • Strong Python and SQL skills with sequential testing and experimentation statistics experience
  • Deep understanding of continuous monitoring challenges and multiple testing problems
  • Experience building experimentation or flag monitoring systems in production
  • Ability to design statistical systems that are understandable to engineers making rollout decisions

Benefits

  • Fully remote with flexible schedule
  • Health and dental insurance
  • $3,000 annual professional development budget
  • 401(k) with match
  • Equity options

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure data science & analytics

Skills

python sql statistics pandas fastapi

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Technical Writer

RemoteAI company

Part-time

AI Resident UK/EMEA

RemoteAI company

Full-time

Software Engineer, Applications (Scientific Development Solutions) (High Seniority)

RemoteAI company

Full-time Senior

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

PhD Autonomy Engineer Intern - Computer Vision / Deep Learning Summer 2027

RemoteAI company

Full-time Intern

Job Type

Full-time

Level

Mid-Level

Language

English

Salary Range

$100k – $130k / year

AI Expertise

MLOps & AI Infrastructure data science & analytics

Skills

python sql statistics pandas fastapi docker

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile