Profile cover
S

AI Model Regression Testing Developer

Sofia Garcia

Full-time · Job · Chicago

About the role

Software has unit tests. ML models have... hopes and prayers? Not on our watch. Our model testing platform provides automated regression testing for ML models — checking that a new model version doesn't break things the old version got right, even if overall metrics look better. A model that improves average accuracy by 2% but starts failing on a critical demographic subset is a regression, not an improvement. You'll build the testing infrastructure that catches these hidden regressions. That means designing test suites that cover critical subsets, implementing statistical comparison methods that detect meaningful changes versus noise, and creating the reporting layer that tells ML engineers exactly what broke and where. Think of it as building the CI/CD quality gate specifically for ML models. The testing platform team is three engineers. As the junior developer, you'll start by implementing test runners and report generators, then progress to designing test strategies for different model types. The senior engineers handle the statistical methodology; you'll handle the infrastructure that makes it run reliably at scale.

Responsibilities

  • Build test runners that evaluate models across critical data subsets
  • Implement statistical comparison methods for detecting meaningful regressions
  • Create reporting dashboards showing regression test results and trends
  • Design test suite templates for common model types
  • Integrate regression tests into client CI/CD pipelines
  • Maintain documentation for test configuration and interpretation

Requirements

  • 1+ year of Python development with testing experience
  • Basic understanding of ML model evaluation metrics
  • Familiarity with Docker and CI/CD concepts
  • Interest in software quality and testing methodology
  • Ability to write clear, well-structured code

Benefits

  • Fully remote with flexible schedule
  • Health and dental insurance
  • $1,500 learning budget for courses and certifications
  • Structured mentorship from senior QA engineers
  • Impact across multiple client ML deployments

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure AI & Machine Learning Engineers

Skills

python docker mlflow aws sql

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

Technical Writer

RemoteAI company

Part-time

Staff Application Security Engineer

RemoteAI company

Full-time Staff

Senior AI R&D Engineer – Model Training (f/m/d)

RemoteAI company

Full-time Senior

Senior AI Researcher- Pre-training (f/m/d)

RemoteAI company

Full-time Senior

Job Type

Full-time

Level

Junior

Language

English

Salary Range

$60k – $80k / year

AI Expertise

MLOps & AI Infrastructure AI & Machine Learning Engineers

Skills

python docker mlflow aws sql fastapi

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile