Profile cover
M

AI Language Quality Evaluation Engineer

Marcus Sokolov

Full-time · Job · Berlin

About the role

Machine translation quality is easy to measure in controlled benchmarks and hard to measure in production. A BLEU score tells you nothing about whether the translated tagline is culturally appropriate for the target market, whether the legal disclaimer preserves its binding meaning, or whether the product name accidentally means something offensive in the target language. Our clients need quality signals that correlate with what actually matters to their users. We serve 180 enterprises using our AI translation platform, processing 280 million words per month across 47 language pairs. Your job is to build the quality evaluation framework that gives our clients confidence in the output: reference-free quality estimation models for languages without annotated test sets, automatic cultural appropriateness detection, and the real-time quality dashboards that flag segments needing human review. The NLP team is four engineers, working across 9 time zones. You'll own the evaluation and quality layer, interfacing with linguists who serve as domain experts for specific language pairs.

Responsibilities

  • Build reference-free quality estimation models for all 47 language pairs on the platform
  • Develop cultural appropriateness detection for marketing and consumer content
  • Create automated flagging pipelines routing low-confidence segments to human review
  • Design real-time quality dashboards for enterprise clients
  • Collaborate with linguists to build annotated evaluation sets for new language pairs
  • Benchmark quality models against human evaluator agreement rates

Requirements

  • 1+ year of NLP engineering or linguistics-adjacent software development experience
  • Familiarity with machine translation quality estimation methods
  • Proficiency with Hugging Face transformers and multilingual models
  • Strong statistical reasoning for evaluation framework design
  • Ability to collaborate across language and cultural backgrounds

Benefits

  • Fully remote — global team
  • 30 days PTO
  • Health insurance
  • Annual language learning stipend
  • Quarterly team meetups across EU cities

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

NLP & Prompt Engineering data science & analytics

Skills

python hugging face statistics pandas sql

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior AI Researcher- Pre-training (f/m/d)

RemoteAI company

Full-time Senior

AI Resident UK/EMEA

RemoteAI company

Full-time

Software Engineer, Full Stack - UK

RemoteAI company

Full-time

PhD Autonomy Engineer Intern - Computer Vision / Deep Learning Summer 2027

RemoteAI company

Full-time Intern

Backend Engineer, Platform

RemoteAI company

Full-time

Product Manager, Developer Experience

RemoteAI company

Full-time

Job Type

Full-time

Level

Junior

Language

English

Salary Range

$80k – $100k / year

AI Expertise

NLP & Prompt Engineering data science & analytics

Skills

python hugging face statistics pandas sql pytorch

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile