AI Training Data Deduplication Engineer

Velmox Data

Remote · $100k – $130k / year · Full-time · Posted 4 months ago

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

We process 80TB of training data per month for ML teams at 45 enterprise clients. Duplicated and near-duplicated training examples are one of the leading causes of model overfitting, benchmark contamination, and training instability — and most clients have no idea how much duplication is in their datasets until models start behaving strangely. You'll build the deduplication infrastructure that screens training data before it enters the pipeline. Exact deduplication is easy; near-duplicate detection at 80TB/month is a distributed systems problem. The challenge is defining and computing semantic similarity efficiently: comparing millions of text documents, image embeddings, or tabular records against a corpus without quadratic scaling. The data quality team is four engineers. You'll design the deduplication architecture from scratch, implement it on Spark, and integrate it into the client onboarding pipeline so that every new training batch is screened automatically. You'll also build the reporting that tells clients what percentage of their dataset was deduplicated and why.

Fully remote.

Responsibilities

Design and implement near-duplicate detection for text, image, and tabular training data
Build distributed deduplication pipelines on Spark handling 80TB/month throughput
Implement MinHash LSH and embedding similarity approaches for different modalities
Create reporting dashboards showing deduplication rates per client and modality
Integrate deduplication into the client data onboarding pipeline
Benchmark deduplication quality vs. downstream model performance improvements

Requirements

3+ years in data engineering with experience processing large-scale datasets
Strong Python and Spark skills for distributed data processing
Understanding of approximate nearest neighbor search and locality-sensitive hashing
Experience with deduplication, data cleaning, or training data quality
Ability to design scalable systems that don't degrade with dataset size