Profile cover
Z

AI Testing Automation Engineer

Joltura Labs

Remote · $100k – $130k / year · Full-time · Posted 5 months ago

No longer accepting applications

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

Every AI feature we ship needs to be tested, but traditional QA doesn't work when the output is non-deterministic. You can't write a unit test that says 'assert response == expected' when the response is different every time. That's the problem we solve — and the problem you'll work on every day. We've built a testing framework for AI products: statistical evaluation suites, regression detection for model quality, and automated red-teaming pipelines. You'll extend this framework, apply it to client engagements, and help define what 'quality' means for AI features across different domains. The work is part engineering, part applied statistics. You'll write Python code to build test harnesses, but you'll also design evaluation metrics and analyze distributions of model outputs. If you find the intersection of testing and statistics genuinely interesting, you'll love this role.

Fully remote.

Responsibilities

  • Build and maintain automated testing frameworks for AI product evaluation
  • Design statistical evaluation suites for non-deterministic AI outputs
  • Implement regression detection pipelines for model quality monitoring
  • Run automated red-teaming pipelines to identify AI failure modes
  • Analyze test results and deliver actionable quality reports

Requirements

  • 3+ years in software testing or quality engineering
  • Strong Python skills for test automation
  • Understanding of statistical methods for evaluating non-deterministic systems
  • Experience with LLMs or AI products
  • Familiarity with Docker and CI/CD pipelines

Benefits

  • Fully remote — US East Coast hours
  • Health and dental insurance
  • Annual bonus tied to client satisfaction
  • $2,500 learning budget

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

Automation & AI Workflows MLOps & AI Infrastructure

Skills

python openai api langchain docker sql

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Principal Engineer Team Lead, Behavior Safety & Metrics

Motional

Full-time Principal

Materials Lead

Lambda

Full-time Lead

Sr. Solutions Architect - Travel and Hospitality

Databricks

Full-time Senior

Software Engineering Manager, AI Migrations

Snowflake

Full-time

Senior Software Engineer - Full Stack - Portal Platform

Abnormal Security

Full-time Senior

AWS Alliances Lead

LangChain

Full-time Lead

Job Type

Full-time

Level

Mid-Level

Language

English

Salary Range

$100k – $130k / year

AI Expertise

Automation & AI Workflows MLOps & AI Infrastructure

Skills

python openai api langchain docker sql pandas statistics

Ready to apply for this role?

Create a free talent account.

  • Find AI roles in one place
  • Get AI-matched job recommendations
  • Reply to companies that message you
  • Build your public AI talent profile
Create free account & apply Log in

This role is closed. Create your free profile to apply to similar open roles.

Don't miss the next one: get matched to new roles like this.

Create free profile