AI Synthetic Dataset Evaluation Benchmarker

About the role

Responsibilities

• Design downstream utility evaluation benchmarks measuring synthetic dataset quality via actual model training outcomes
• Build comparison pipelines running synthetic vs. real data through controlled ML training experiments
• Implement statistical fidelity metrics beyond standard similarity measures, grounded in task-specific utility
• Create benchmark suite covering tabular, text, and sequential synthetic data modalities
• Develop evaluation reporting clients use to validate synthetic data before production use
• Monitor benchmark results across generation model updates to track fidelity trends

Requirements

• 3+ years in ML engineering or data science with synthetic data, evaluation, or benchmarking experience
• Strong Python and statistical computing skills
• Understanding of synthetic data generation techniques and their known failure modes
• Experience designing controlled experiments for ML system comparison
• Ability to communicate evaluation methodology clearly to non-technical clients

Benefits

• Fully remote with flexible schedule
• Health, dental, and vision insurance
• $3,000 annual learning budget
• 401(k) with match
• Equity options

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.