Profile cover
R

AI Active Learning Sampling Engineer

Ruby Santos

Full-time · Job · Chicago

About the role

Labeling data is expensive. Our clients spend $0.04-$0.12 per labeled example and produce 50,000 new labels per week, which means annotation budgets are meaningful. The question of which examples to label next — from a pool of millions of unlabeled candidates — determines whether you need 100,000 labels or 40,000 to reach the same model accuracy. Active learning is the answer, and most of our clients aren't using it. You'll build the active learning sampling infrastructure that sits between our clients' unlabeled data pools and their annotation queues. The system needs to support multiple query strategies — uncertainty sampling, core-set selection, diversity-aware sampling — and integrate with our annotation workflow platform to deliver the right candidates to the right annotator pools at the right priority. The technical challenge is scale: evaluating 10 million candidate examples per day across multiple strategies requires efficient batching, approximate inference, and smart caching. The business challenge is proving ROI: you'll build the evaluation framework that shows clients exactly how many labels they saved versus random sampling.

Responsibilities

  • Build an active learning sampling system supporting multiple query strategies
  • Implement uncertainty, diversity, and core-set sampling at 10M candidate scale
  • Integrate sampling pipeline with annotation workflow APIs
  • Build approximate inference for efficient candidate scoring across large unlabeled pools
  • Create ROI evaluation framework comparing label efficiency vs. random sampling baseline
  • Monitor sample quality and adapt strategies as models improve during training

Requirements

  • 3+ years in ML engineering with experience in active learning or data-efficient training
  • Strong Python and PyTorch skills
  • Experience building ML inference at scale with batching and optimization
  • Understanding of active learning literature and query strategy trade-offs
  • Ability to design systems that integrate with existing annotation workflow platforms

Benefits

  • Fully remote with flexible schedule
  • Health, dental, and vision insurance
  • 401(k) with match
  • $3,000 professional development budget
  • Annual team retreat

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Job Type

Full-time

Level

Mid-Level

Language

English

Salary Range

$100k – $130k / year

AI Expertise

data science & analytics MLOps & AI Infrastructure

Skills

python pytorch scikit-learn sql aws pandas

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile

Interested in this role? Create your free profile to apply.

Apply — create free profile