Profile cover
B

Junior AI Data Pipeline Backfill Orchestration Developer

Benjamin Leclerc

Contract · Job · Los_Angeles

About the role

When a data pipeline fails, the data for that time window doesn't just disappear — it needs to be recovered and processed to fill the gap. When a schema changes and historical data needs to be reprocessed with the new schema, all previous partitions need to be backfilled. When a new feature is computed that needs historical values for model training, the pipeline needs to run against years of historical data. Backfill orchestration — managing these large-scale historical reprocessing jobs safely and efficiently — is unglamorous infrastructure work that breaks badly when done wrong. This is a 7-month contract to build the backfill orchestration system for our data platform. You'll build the tooling that pipelines engineers use to define, schedule, and monitor backfill jobs: partition tracking to ensure no gaps or double-processing, progress monitoring with ETA estimation, cost controls that pause expensive backfills when they exceed budget, and failure recovery that resumes from the last successful partition rather than restarting. Senior data engineers define the infrastructure architecture. Your job is to implement the orchestration tooling that makes backfills reliable, observable, and safe to run in parallel with production pipelines.

Responsibilities

  • Build backfill job definition tooling allowing engineers to specify date ranges, partition sizes, and parallelism
  • Implement partition tracking ensuring complete coverage without duplicate processing
  • Create progress monitoring with ETA estimation and cost tracking for running backfills
  • Develop cost controls pausing backfills that exceed defined compute budget thresholds
  • Build failure recovery enabling resume-from-checkpoint rather than full restart
  • Create backfill safety checks preventing interference with production pipeline partitions

Requirements

  • 1+ year of Python and SQL development experience
  • Basic familiarity with Airflow, data pipelines, or workflow orchestration
  • Interest in data engineering, platform engineering, or distributed systems
  • Attention to correctness in partition tracking and completeness verification
  • Reliable async communication for a contract engagement

Benefits

  • Fully remote contract — 7-month engagement
  • Flexible schedule
  • Competitive contract rate
  • Mentorship from senior data engineers
  • Potential for full-time conversion

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure Automation & AI Workflows

Skills

python sql airflow docker aws

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

AI Resident UK/EMEA

RemoteAI company

Full-time

Devops Engineer, US

RemoteAI company

Full-time

Infrastructure Engineer

RemoteAI company

Full-time

PhD Autonomy Engineer Intern - Computer Vision / Deep Learning Summer 2027

RemoteAI company

Full-time Intern

Backend Engineer, Platform

RemoteAI company

Full-time

Job Type

Contract

Level

Junior

Language

English

Salary Range

$60k – $80k / year

AI Expertise

MLOps & AI Infrastructure Automation & AI Workflows

Skills

python sql airflow docker aws pandas

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile