AI Feature Importance Drift Monitor

Velmox Data

Remote · $100k – $130k / year · Full-time · Posted 4 months ago

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

Data drift monitoring typically focuses on distribution shifts: has the statistical distribution of input feature X changed? That's necessary but not sufficient. A feature whose distribution is stable can still lose its predictive relationship with the target — which is the kind of model degradation that kills performance without showing up on standard drift dashboards. We monitor 320 production ML models across 65 enterprise clients. The gap in our monitoring stack is feature importance drift: detecting when the features that were predictive during training are losing their predictive relationship with the target in production, before that shows up in model performance metrics. Finding it early means retraining before users notice degradation; finding it late means an incident. You'll build the feature importance drift monitoring system: implementing online importance estimation using permutation-based and SHAP-based approaches on production inference batches, tracking importance changes over time, and building the alerting that triggers retraining pipelines when importance drift exceeds thresholds. The challenge is efficiency — computing SHAP values on every inference batch at production throughput isn't feasible, and you'll need smart sampling and approximation strategies.

Fully remote.

Responsibilities

Build online feature importance estimation using efficient sampling and approximate SHAP computation
Implement importance drift tracking detecting changes in predictive feature relationships over time
Design alerting thresholds triggering retraining pipeline recommendations when importance shifts exceed limits
Create importance drift reports explaining which features changed and the magnitude of drift
Develop correlation between importance drift signals and downstream model performance degradation
Optimize importance computation to run at production throughput without adding significant latency

Requirements

3+ years in ML engineering with model monitoring, MLOps, or feature engineering experience
Strong Python skills with SHAP, permutation importance, or similar explainability techniques
Understanding of production ML monitoring challenges and model degradation patterns
Experience with efficient sampling strategies for computationally expensive ML diagnostics
Familiarity with model retraining triggers and MLOps pipeline design