Profile cover
B

AI Real-time CDC Stream Processor Developer

Benjamin Leclerc

Contract · Job · New_York

About the role

Change Data Capture pipelines move database changes into data warehouses and feature stores in real time — but only if you can handle the operational complexity: schema changes that break consumers, out-of-order events from database replicas, deduplication of retried messages, and late-arriving updates to records that downstream models have already processed. Our CDC platform processes 400 million database change events per day across 85 client databases, maintaining sub-second latency from source commit to warehouse availability. You'll own the stream processing layer: the Kafka consumer logic that handles event ordering and deduplication, the transformation engine that applies schema migrations without downtime, and the exactly-once delivery semantics that prevent duplicate updates from corrupting downstream models. The streaming infrastructure team is six engineers. You'll be the senior engineer who defines the processing architecture, setting the standard for how CDC events are handled reliably at scale. This is deep distributed systems work — you'll need strong intuitions about ordering, consistency, and failure modes.

Responsibilities

  • Build stream processing for 400M CDC events per day with sub-second latency
  • Implement exactly-once delivery semantics preventing duplicate downstream updates
  • Design out-of-order event handling for database replica lag
  • Create zero-downtime schema migration for consumer pipelines
  • Build deduplication logic for retried messages
  • Set architectural standards for reliable CDC processing

Requirements

  • 6+ years in data engineering with streaming systems expertise
  • Deep expertise in Kafka, Spark Streaming, or Flink at scale
  • Strong understanding of distributed systems consistency and ordering
  • Experience with CDC tools: Debezium, Maxwell, or similar
  • Track record of running streaming systems reliably in production

Benefits

  • Fully remote with async communication
  • Top-of-market compensation plus equity
  • Comprehensive health coverage
  • $5,000 annual conference and learning budget
  • Work on mission-critical data infrastructure

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

MLOps & AI Infrastructure data science & analytics

Skills

python kafka spark sql docker

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

Backend Engineer, Platform

RemoteAI company

Full-time

Technical Writer

RemoteAI company

Part-time

Principal Engineer

RemoteAI company

Full-time Principal

Staff Application Security Engineer

RemoteAI company

Full-time Staff

Job Type

Contract

Level

Senior

Language

English

Salary Range

$160k – $200k / year

AI Expertise

MLOps & AI Infrastructure data science & analytics

Skills

python kafka spark sql docker aws

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile