Profile cover
B

AI Data Lakehouse Query Optimizer

Benjamin Leclerc

Full-time · Job · Chicago

About the role

A query that takes 4 minutes to run costs 10x more than one that takes 24 seconds. At petabyte scale across a shared data lakehouse serving 800 data scientists and engineers, poorly optimized queries do not just waste money — they create resource contention that slows everyone down. Query optimization is normally a manual process requiring database expertise that most users do not have. We build the intelligence layer for enterprise data lakehouses. Our query optimization platform analyzes submitted queries, identifies optimization opportunities — suboptimal join order, missing partition filters, inefficient aggregation patterns — and rewrites them automatically before execution. The system improves query cost by an average of 47% across our 35 enterprise clients. You will lead the query rewrite engine: extending coverage to new anti-pattern categories, improving the cost estimation models that predict which rewrites will help, and building the feedback loop from actual execution costs back into the optimization models.

Responsibilities

  • Extend query rewrite engine coverage to new optimization anti-pattern categories
  • Improve cost estimation models for query rewrite candidate ranking
  • Build execution feedback loops connecting actual costs to model improvement
  • Design optimization strategies for Spark, Trino, and Databricks SQL dialects
  • Create evaluation benchmarks from production query workloads
  • Collaborate with data platform engineers on execution environment instrumentation

Requirements

  • 6+ years in data engineering or query optimization with distributed systems expertise
  • Deep knowledge of Spark SQL, query planning, and cost-based optimization
  • Experience with Databricks, Trino, or similar lakehouse query engines
  • Strong Python skills and statistical reasoning for cost modeling
  • Understanding of data lake storage formats: Delta, Parquet, Iceberg

Benefits

  • Fully remote
  • Top-of-market compensation
  • Health, dental, and vision
  • $5,000 annual learning budget
  • Equity with 4-year vesting

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure data science & analytics

Skills

python spark sql aws databricks

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Technical Writer

RemoteAI company

Part-time

AI Resident UK/EMEA

RemoteAI company

Full-time

Devops Engineer, US

RemoteAI company

Full-time

Software Engineer, Applications (Scientific Development Solutions) (High Seniority)

RemoteAI company

Full-time Senior

Job Type

Full-time

Level

Senior

Language

English

Salary Range

$160k – $200k / year

AI Expertise

MLOps & AI Infrastructure data science & analytics

Skills

python spark sql aws databricks statistics

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile