AI Language Quality Evaluation Engineer

Soltix AI

Remote · $80k – $100k / year · Full-time · Posted 4 months ago

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

Machine translation quality is easy to measure in controlled benchmarks and hard to measure in production. A BLEU score tells you nothing about whether the translated tagline is culturally appropriate for the target market, whether the legal disclaimer preserves its binding meaning, or whether the product name accidentally means something offensive in the target language. Our clients need quality signals that correlate with what actually matters to their users. We serve 180 enterprises using our AI translation platform, processing 280 million words per month across 47 language pairs. Your job is to build the quality evaluation framework that gives our clients confidence in the output: reference-free quality estimation models for languages without annotated test sets, automatic cultural appropriateness detection, and the real-time quality dashboards that flag segments needing human review. The NLP team is four engineers, working across 9 time zones. You'll own the evaluation and quality layer, interfacing with linguists who serve as domain experts for specific language pairs.

Fully remote.

Responsibilities

Build reference-free quality estimation models for all 47 language pairs on the platform
Develop cultural appropriateness detection for marketing and consumer content
Create automated flagging pipelines routing low-confidence segments to human review
Design real-time quality dashboards for enterprise clients
Collaborate with linguists to build annotated evaluation sets for new language pairs
Benchmark quality models against human evaluator agreement rates

Requirements

1+ year of NLP engineering or linguistics-adjacent software development experience
Familiarity with machine translation quality estimation methods
Proficiency with Hugging Face transformers and multilingual models
Strong statistical reasoning for evaluation framework design
Ability to collaborate across language and cultural backgrounds