Profile cover
R

AI Annotation Consensus Algorithm Developer

Ruby Santos

Full-time · Job · London

About the role

When three annotators label the same image, they disagree 35% of the time. The naive approach — majority vote — ignores the fact that some annotators are consistently more accurate, some tasks are inherently ambiguous, and some disagreements reveal genuine edge cases that the model should handle carefully rather than force into a single label. Our consensus system goes beyond majority vote. It models annotator reliability, task difficulty, and label ambiguity simultaneously using probabilistic graphical models. The output isn't just a consensus label — it's a confidence distribution that tells the model training pipeline how much to trust each example. High-confidence examples get full weight; ambiguous ones get soft labels or get routed for expert review. The data quality team is five engineers. You'll own the consensus algorithm and its integration with the annotation platform. Your models affect every downstream ML model our clients train — better consensus means better training data means better models, at scale across 200 annotators and 15,000 labels per day.

Responsibilities

  • Design probabilistic consensus models capturing annotator reliability and task difficulty
  • Produce confidence-weighted labels for model training pipelines
  • Route ambiguous cases to expert review based on disagreement patterns
  • Evaluate consensus quality against expert ground truth benchmarks
  • Optimize algorithm performance for real-time consensus on streaming annotations
  • Publish methodology documentation for client transparency

Requirements

  • 5+ years in ML with focus on data quality, annotation, or probabilistic modeling
  • Experience with graphical models, Bayesian methods, or inter-annotator agreement
  • Strong statistical background with Python implementation skills
  • Understanding of how label quality affects downstream model training
  • Track record of shipping systems that improve data quality at scale

Benefits

  • Fully remote across UK/EU time zones
  • 30 days PTO
  • Private health insurance
  • £4,000 annual conference budget
  • Impact across all downstream ML models

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

data science & analytics AI & Machine Learning Engineers

Skills

python pytorch statistics sql docker

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior AI Researcher- Pre-training (f/m/d)

RemoteAI company

Full-time Senior

Senior Product Security Engineer

RemoteAI company

Full-time Senior

AI Resident UK/EMEA

RemoteAI company

Full-time

Senior AI R&D Engineer – Model Training (f/m/d)

RemoteAI company

Full-time Senior

Senior Machine Learning Engineer (Clinical Team)

RemoteAI company

Full-time Senior

Conversational Modelling Research Engineer

RemoteAI company

Full-time

Job Type

Full-time

Level

Senior

Language

English

Salary Range

$130k – $160k / year

AI Expertise

data science & analytics AI & Machine Learning Engineers

Skills

python pytorch statistics sql docker fastapi

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile