Profile cover
A

AI Feature Importance Drift Monitor

Aubrey Okafor

Full-time · Job · New_York

About the role

Data drift monitoring typically focuses on distribution shifts: has the statistical distribution of input feature X changed? That's necessary but not sufficient. A feature whose distribution is stable can still lose its predictive relationship with the target — which is the kind of model degradation that kills performance without showing up on standard drift dashboards. We monitor 320 production ML models across 65 enterprise clients. The gap in our monitoring stack is feature importance drift: detecting when the features that were predictive during training are losing their predictive relationship with the target in production, before that shows up in model performance metrics. Finding it early means retraining before users notice degradation; finding it late means an incident. You'll build the feature importance drift monitoring system: implementing online importance estimation using permutation-based and SHAP-based approaches on production inference batches, tracking importance changes over time, and building the alerting that triggers retraining pipelines when importance drift exceeds thresholds. The challenge is efficiency — computing SHAP values on every inference batch at production throughput isn't feasible, and you'll need smart sampling and approximation strategies.

Responsibilities

  • Build online feature importance estimation using efficient sampling and approximate SHAP computation
  • Implement importance drift tracking detecting changes in predictive feature relationships over time
  • Design alerting thresholds triggering retraining pipeline recommendations when importance shifts exceed limits
  • Create importance drift reports explaining which features changed and the magnitude of drift
  • Develop correlation between importance drift signals and downstream model performance degradation
  • Optimize importance computation to run at production throughput without adding significant latency

Requirements

  • 3+ years in ML engineering with model monitoring, MLOps, or feature engineering experience
  • Strong Python skills with SHAP, permutation importance, or similar explainability techniques
  • Understanding of production ML monitoring challenges and model degradation patterns
  • Experience with efficient sampling strategies for computationally expensive ML diagnostics
  • Familiarity with model retraining triggers and MLOps pipeline design

Benefits

  • Fully remote with flexible schedule
  • Health and dental insurance
  • $3,000 annual professional development budget
  • 401(k) with match
  • Equity options

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

data science & analytics MLOps & AI Infrastructure

Skills

python pytorch scikit-learn sql spark

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

AI Resident UK/EMEA

RemoteAI company

Full-time

Technical Writer

RemoteAI company

Part-time

Senior AI Researcher- Pre-training (f/m/d)

RemoteAI company

Full-time Senior

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Software Engineer, Applications (Scientific Development Solutions) (High Seniority)

RemoteAI company

Full-time Senior

PhD Autonomy Engineer Intern - Computer Vision / Deep Learning Summer 2027

RemoteAI company

Full-time Intern

Job Type

Full-time

Level

Mid-Level

Language

English

Salary Range

$100k – $130k / year

AI Expertise

data science & analytics MLOps & AI Infrastructure

Skills

python pytorch scikit-learn sql spark pandas

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile