LLM Evaluation Engineer

Chiara Lombardi

Full-time · Mid-level · Vancouver

This role is no longer accepting applications

Similar open roles you can apply to now:

About the role

Most teams working with LLMs spend more time talking about output quality than measuring it properly. We would like to stop doing that. This role is for someone who enjoys building the boring, valuable machinery around evaluation: benchmarks, rubrics, review workflows, failure taxonomies, and reporting that actually influences launch decisions. You will be the person in the room who can say whether a feature improved, not just whether it looked good in a demo.

Responsibilities

Design benchmark suites and rubrics for LLM-powered enterprise workflows
Build automated evaluation pipelines and human review loops for new model or prompt changes
Work with product and engineering to define task-specific quality thresholds before launch
Investigate failure patterns and classify them by retrieval, reasoning, formatting, or instruction-following issues
Create reporting that makes model quality legible to technical and non-technical stakeholders

Requirements

3+ years of software, ML, or quality engineering with hands-on work around LLM features
Python for evaluation tooling and data pipelines
OpenAI API familiarity including structured outputs and comparison workflows
Prompt Engineering experience sufficient to understand how prompt changes affect evaluation results
RAG familiarity so retrieval-specific failure can be separated from generation failure
SQL for test-set analysis, failure segmentation, and dashboarding support