Profile cover
A

AI Training Data Deduplication Engineer

Aubrey Okafor

Full-time · Job · Chicago

About the role

We process 80TB of training data per month for ML teams at 45 enterprise clients. Duplicated and near-duplicated training examples are one of the leading causes of model overfitting, benchmark contamination, and training instability — and most clients have no idea how much duplication is in their datasets until models start behaving strangely. You'll build the deduplication infrastructure that screens training data before it enters the pipeline. Exact deduplication is easy; near-duplicate detection at 80TB/month is a distributed systems problem. The challenge is defining and computing semantic similarity efficiently: comparing millions of text documents, image embeddings, or tabular records against a corpus without quadratic scaling. The data quality team is four engineers. You'll design the deduplication architecture from scratch, implement it on Spark, and integrate it into the client onboarding pipeline so that every new training batch is screened automatically. You'll also build the reporting that tells clients what percentage of their dataset was deduplicated and why.

Responsibilities

  • Design and implement near-duplicate detection for text, image, and tabular training data
  • Build distributed deduplication pipelines on Spark handling 80TB/month throughput
  • Implement MinHash LSH and embedding similarity approaches for different modalities
  • Create reporting dashboards showing deduplication rates per client and modality
  • Integrate deduplication into the client data onboarding pipeline
  • Benchmark deduplication quality vs. downstream model performance improvements

Requirements

  • 3+ years in data engineering with experience processing large-scale datasets
  • Strong Python and Spark skills for distributed data processing
  • Understanding of approximate nearest neighbor search and locality-sensitive hashing
  • Experience with deduplication, data cleaning, or training data quality
  • Ability to design scalable systems that don't degrade with dataset size

Benefits

  • Fully remote with flexible schedule
  • Health, dental, and vision insurance
  • 401(k) with 4% match
  • $3,000 annual conference and training budget
  • Home office setup stipend

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

data science & analytics MLOps & AI Infrastructure

Skills

python spark sql aws pytorch

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

AI Resident UK/EMEA

RemoteAI company

Full-time

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

Technical Writer

RemoteAI company

Part-time

Senior AI Researcher- Pre-training (f/m/d)

RemoteAI company

Full-time Senior

Staff Application Security Engineer

RemoteAI company

Full-time Staff

Job Type

Full-time

Level

Mid-Level

Language

English

Salary Range

$100k – $130k / year

AI Expertise

data science & analytics MLOps & AI Infrastructure

Skills

python spark sql aws pytorch pandas

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile