AI Data Lakehouse Query Optimizer

Infera Cloud

Remote · $160k – $200k / year · Full-time · Posted 5 months ago

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

A query that takes 4 minutes to run costs 10x more than one that takes 24 seconds. At petabyte scale across a shared data lakehouse serving 800 data scientists and engineers, poorly optimized queries do not just waste money — they create resource contention that slows everyone down. Query optimization is normally a manual process requiring database expertise that most users do not have. We build the intelligence layer for enterprise data lakehouses. Our query optimization platform analyzes submitted queries, identifies optimization opportunities — suboptimal join order, missing partition filters, inefficient aggregation patterns — and rewrites them automatically before execution. The system improves query cost by an average of 47% across our 35 enterprise clients. You will lead the query rewrite engine: extending coverage to new anti-pattern categories, improving the cost estimation models that predict which rewrites will help, and building the feedback loop from actual execution costs back into the optimization models.

Fully remote.

Responsibilities

Extend query rewrite engine coverage to new optimization anti-pattern categories
Improve cost estimation models for query rewrite candidate ranking
Build execution feedback loops connecting actual costs to model improvement
Design optimization strategies for Spark, Trino, and Databricks SQL dialects
Create evaluation benchmarks from production query workloads
Collaborate with data platform engineers on execution environment instrumentation

Requirements

6+ years in data engineering or query optimization with distributed systems expertise
Deep knowledge of Spark SQL, query planning, and cost-based optimization
Experience with Databricks, Trino, or similar lakehouse query engines
Strong Python skills and statistical reasoning for cost modeling
Understanding of data lake storage formats: Delta, Parquet, Iceberg