Profile cover
I

AI Entity Deduplication Engineer

Isaac Singh

Full-time · Job · Chicago

About the role

Spark, Hadoop, and a graph-based ontology engine are what our knowledge graph runs on, but duplicate entities still slip through and quietly degrade query quality for our clients. If you've worked on entity resolution at scale before, you know how easy it is to underestimate how messy real-world entity data actually is. You'll build the deduplication layer that catches near-duplicate entities before they pollute the graph, working with a team that reads relevant research and expects findings to make it into production, not just a slide deck.

Responsibilities

  • Build entity deduplication pipelines for large-scale knowledge graph data
  • Design matching rules and similarity scoring for near-duplicate entities
  • Work with Spark and Hadoop to process graph data at scale
  • Benchmark deduplication accuracy against manually labeled samples
  • Document methodology for internal research review

Requirements

  • 2+ years of experience with entity resolution or data deduplication
  • Experience with Spark or Hadoop for large-scale data processing
  • Strong SQL and ETL skills
  • Familiarity with graph databases or knowledge graphs a plus
  • Comfortable reading and applying academic research on entity matching

Benefits

  • Fully remote position
  • Health insurance stipend
  • Conference and publication budget
  • Flexible hours built around deep work
  • 25 days paid vacation

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

data science & analytics

Skills

python spark sql etl hadoop

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Backend Engineer, Platform

RemoteAI company

Full-time

PhD Autonomy Engineer Intern - Computer Vision / Deep Learning Summer 2027

RemoteAI company

Full-time Intern

Product Manager, Developer Experience

RemoteAI company

Full-time

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Business Operations Lead

RemoteAI company

Full-time Lead

Senior AI R&D Engineer – Model Training (f/m/d)

RemoteAI company

Full-time Senior

Job Type

Full-time

Level

Junior

Language

English

Salary Range

$80k – $100k / year

AI Expertise

data science & analytics

Skills

python spark sql etl hadoop

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile