Profile cover
L

AI Quality Assurance Engineer

Ravio Tech

Remote · $80k – $100k / year · Full-time · Posted 5 months ago

No longer accepting applications

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

How do you QA something that gives a different answer every time you ask? That's the fundamental challenge of testing AI products, and we've built our entire business around solving it. We provide AI testing services for companies that ship LLM-powered features but don't have internal evaluation expertise. You'll design and execute test plans for AI features: accuracy benchmarks, regression tests, edge case discovery, hallucination detection, and latency profiling. Each client engagement is a puzzle — different models, different use cases, different failure modes. This is an on-site role at our office in Boston. We work on-site because our testing methodology involves a lot of collaborative analysis — two people looking at the same model output and debating whether it's correct. That's hard to do async.

Fully remote.

Responsibilities

  • Design test plans and evaluation frameworks for client AI products
  • Build automated testing pipelines for accuracy and regression detection
  • Identify edge cases and failure modes through systematic exploration
  • Analyze test results and deliver detailed quality reports to clients
  • Develop reusable testing tools and benchmarks across engagements

Requirements

  • 3+ years in QA or software testing
  • Python proficiency for test automation
  • Experience with LLMs or AI products (even as a user/tester)
  • Strong analytical skills and attention to detail
  • Good written communication for client-facing reports

Benefits

  • On-site in Boston
  • Health and dental insurance
  • Professional development budget
  • Exposure to diverse AI products and industries

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

Automation & AI Workflows AI & Machine Learning Engineers

Skills

python sql openai api langchain pandas

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Sr Technical Program Manager, Data

Hyperscience

Full-time Senior

Principal Engineer Team Lead, Behavior Safety & Metrics

Motional

Full-time Principal

Materials Lead

Lambda

Full-time Lead

Lead/ Senior Machine Learning Scientist, Integrative Phenotyping

insitro

Full-time Lead

AWS Alliances Lead

LangChain

Full-time Lead

AWS Alliances Lead

LangChain

Full-time Lead

Job Type

Full-time

Level

Mid-Level

Language

English

Salary Range

$80k – $100k / year

AI Expertise

Automation & AI Workflows AI & Machine Learning Engineers

Skills

python sql openai api langchain pandas statistics

Ready to apply for this role?

Create a free talent account.

  • Find AI roles in one place
  • Get AI-matched job recommendations
  • Reply to companies that message you
  • Build your public AI talent profile
Create free account & apply Log in

This role is closed. Create your free profile to apply to similar open roles.

Don't miss the next one: get matched to new roles like this.

Create free profile