Profile cover
S

AI Evaluation Engineer

Sofia Garcia

Contract · Job · New_York

About the role

How do you know if an AI system is actually good? That's the question we help companies answer. We build evaluation frameworks for LLM-powered products — testing for accuracy, hallucination rates, bias, latency, and cost. Our clients include mid-size SaaS companies who've shipped AI features and now need to know if they're working. This is an entry-level role for someone who's analytical, detail-oriented, and curious about AI quality. You'll start by running evaluation pipelines, analyzing results, and writing reports. Over time, you'll design new evaluation metrics and build automated testing harnesses. We're a team of 5, and this is a contract-to-hire position. If you're someone who finds bugs that others miss and enjoys the satisfaction of quantifying quality, you'll do well here.

Responsibilities

  • Run LLM evaluation pipelines and analyze results for client reports
  • Track and report on accuracy, hallucination rates, and bias metrics
  • Build automated test harnesses for AI feature validation
  • Maintain evaluation datasets and golden test sets
  • Present findings to clients with clear visualizations

Requirements

  • Bachelor's degree in CS, math, statistics, or related field
  • Python proficiency with pandas for data analysis
  • Basic understanding of LLMs and prompt engineering
  • Analytical mindset with strong attention to detail
  • Good written communication for client-facing reports

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

AI & Machine Learning Engineers Automation & AI Workflows

Skills

python openai api langchain sql pandas

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Technical Writer

RemoteAI company

Part-time

Senior AI R&D Engineer – Model Training (f/m/d)

RemoteAI company

Full-time Senior

Senior AI Researcher- Pre-training (f/m/d)

RemoteAI company

Full-time Senior

Multimodal AI Model Optimization Research Engineer

RemoteAI company

Full-time

Software Engineer, Applications (Scientific Development Solutions) (High Seniority)

RemoteAI company

Full-time Senior

Senior Autonomy Engineer, Data Curation

RemoteAI company

Full-time Senior

Job Type

Contract

Level

Junior

Language

English

Salary Range

$60k – $80k / year

AI Expertise

AI & Machine Learning Engineers Automation & AI Workflows

Skills

python openai api langchain sql pandas statistics

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile