Profile cover
L

AI Quality Assurance Engineer

Lillian Nguyen

Full-time · Job · New_York

About the role

How do you QA something that gives a different answer every time you ask? That's the fundamental challenge of testing AI products, and we've built our entire business around solving it. We provide AI testing services for companies that ship LLM-powered features but don't have internal evaluation expertise. You'll design and execute test plans for AI features: accuracy benchmarks, regression tests, edge case discovery, hallucination detection, and latency profiling. Each client engagement is a puzzle — different models, different use cases, different failure modes. This is an on-site role at our office in Boston. We work on-site because our testing methodology involves a lot of collaborative analysis — two people looking at the same model output and debating whether it's correct. That's hard to do async.

Responsibilities

  • Design test plans and evaluation frameworks for client AI products
  • Build automated testing pipelines for accuracy and regression detection
  • Identify edge cases and failure modes through systematic exploration
  • Analyze test results and deliver detailed quality reports to clients
  • Develop reusable testing tools and benchmarks across engagements

Requirements

  • 3+ years in QA or software testing
  • Python proficiency for test automation
  • Experience with LLMs or AI products (even as a user/tester)
  • Strong analytical skills and attention to detail
  • Good written communication for client-facing reports

Benefits

  • On-site in Boston
  • Health and dental insurance
  • Professional development budget
  • Exposure to diverse AI products and industries

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

Automation & AI Workflows AI & Machine Learning Engineers

Skills

python sql openai api langchain pandas

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Technical Writer

RemoteAI company

Part-time

Senior AI R&D Engineer – Model Training (f/m/d)

RemoteAI company

Full-time Senior

Senior AI Researcher- Pre-training (f/m/d)

RemoteAI company

Full-time Senior

Multimodal AI Model Optimization Research Engineer

RemoteAI company

Full-time

Software Engineer, Applications (Scientific Development Solutions) (High Seniority)

RemoteAI company

Full-time Senior

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

Job Type

Full-time

Level

Mid-Level

Language

English

Salary Range

$80k – $100k / year

AI Expertise

Automation & AI Workflows AI & Machine Learning Engineers

Skills

python sql openai api langchain pandas statistics

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile