Profile cover
Z

AI Testing Automation Engineer

Zoe Levy

Full-time · Job · New_York

About the role

Every AI feature we ship needs to be tested, but traditional QA doesn't work when the output is non-deterministic. You can't write a unit test that says 'assert response == expected' when the response is different every time. That's the problem we solve — and the problem you'll work on every day. We've built a testing framework for AI products: statistical evaluation suites, regression detection for model quality, and automated red-teaming pipelines. You'll extend this framework, apply it to client engagements, and help define what 'quality' means for AI features across different domains. The work is part engineering, part applied statistics. You'll write Python code to build test harnesses, but you'll also design evaluation metrics and analyze distributions of model outputs. If you find the intersection of testing and statistics genuinely interesting, you'll love this role.

Responsibilities

  • Build and maintain automated testing frameworks for AI product evaluation
  • Design statistical evaluation suites for non-deterministic AI outputs
  • Implement regression detection pipelines for model quality monitoring
  • Run automated red-teaming pipelines to identify AI failure modes
  • Analyze test results and deliver actionable quality reports

Requirements

  • 3+ years in software testing or quality engineering
  • Strong Python skills for test automation
  • Understanding of statistical methods for evaluating non-deterministic systems
  • Experience with LLMs or AI products
  • Familiarity with Docker and CI/CD pipelines

Benefits

  • Fully remote — US East Coast hours
  • Health and dental insurance
  • Annual bonus tied to client satisfaction
  • $2,500 learning budget

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

Automation & AI Workflows MLOps & AI Infrastructure

Skills

python openai api langchain docker sql

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Technical Writer

RemoteAI company

Part-time

AI Resident UK/EMEA

RemoteAI company

Full-time

Software Engineer, Applications (Scientific Development Solutions) (High Seniority)

RemoteAI company

Full-time Senior

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

PhD Autonomy Engineer Intern - Computer Vision / Deep Learning Summer 2027

RemoteAI company

Full-time Intern

Job Type

Full-time

Level

Mid-Level

Language

English

Salary Range

$100k – $130k / year

AI Expertise

Automation & AI Workflows MLOps & AI Infrastructure

Skills

python openai api langchain docker sql pandas statistics

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile