LLM Evaluation Engineer - Compliance Reviews
AutoTrace AI
Remote · $116K - $138K · Full-time · Posted 5 months ago
This role is no longer accepting applications
Don't miss the next role like this
Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.
Create free profile, 2 minutesSimilar open roles you can apply to now:
- Senior Machine Learning Engineer, LLM Inference Optimization Nebius View →
- Scientist II Machine Learning Engineer (LLM & GenAI Specialist) Owkin View →
- Intermediate/ Senior Software Engineer - Cortex LLM Training Platform Snowflake View →
This role is no longer accepting applications
Don't miss the next role like this
Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.
Create free profile, 2 minutesSimilar open roles you can apply to now:
- Senior Machine Learning Engineer, LLM Inference Optimization Nebius View →
- Scientist II Machine Learning Engineer (LLM & GenAI Specialist) Owkin View →
- Intermediate/ Senior Software Engineer - Cortex LLM Training Platform Snowflake View →
About the role
Fully remote.
Trust & safety
Report this job
Help us review suspicious or misleading job posts. Your report goes directly to RemoteAI.
Report submitted
Thanks for flagging this. RemoteAI can review suspicious job posts and take action when needed.
Responsibilities
- Build repeatable evaluation workflows for compliance review assistants
- Create benchmark sets from real historical cases and reviewer feedback
- Analyse regressions across prompts, model versions, and retrieval changes
- Partner with product and compliance teams on acceptance criteria
- Raise the overall discipline of how LLM quality is discussed internally
Requirements
- 3+ years in AI engineering, evaluation, or NLP tooling
- Python for evaluation harnesses and internal tools
- OpenAI API experience in production settings
- Prompt Engineering for structured and auditable outputs
- RAG understanding for evidence-grounded responses
- SQL for storing and comparing evaluation outcomes
Benefits
- $116,000 - $138,000 base salary
- Remote-first in Europe
- Annual learning budget
- Private health insurance
- A narrow, important problem space with strong quality standards
Keep exploring related roles
Use these topic pages to discover more roles in the same hiring cluster.
Categories
Skills
Related remote AI jobs
Nearby roles that share similar skills, categories, or hiring intent.
Sr Technical Program Manager, Data
Hyperscience
Senior Site Reliability Engineer - (US-Remote)
ClickHouse
Data Annotation Specialist, Safety
Cohere
Principal Engineer Team Lead, Behavior Safety & Metrics
Motional
Software engineer, agents (UK)
Writer
Senior Implementation Success Partner
Qventus
Job Type
Full-time
Level
Mid-level
Language
English
Salary Range
$116,000 - $138,000
AI Expertise
Skills
Ready to apply for this role?
Create a free talent account.
- Find AI roles in one place
- Get AI-matched job recommendations
- Reply to companies that message you
- Build your public AI talent profile
About AutoTrace AI
View companyThis role is closed. Create your free profile to apply to similar open roles.
Don't miss the next one: get matched to new roles like this.
Create free profile