Freelance Data Engineer — Streaming Events Migration

Sarah Bennett

Freelance · Mid-level · London

This role is no longer accepting applications

Similar open roles you can apply to now:

About the role

Short version: our event pipeline grew faster than its design. Now we need to migrate key fintech product streams onto something cleaner and easier to support. This is a 14-week freelance engagement with specific deliverables around ingestion redesign, duplicate handling, data quality, and documentation. We are not looking for strategy decks. We are looking for someone who can get into the pipeline, make sensible decisions, and leave the team with something maintainable.

Key Deliverables

List the expected deliverables for this project.

Migrate key event pipelines to a cleaner streaming architecture built around Kafka and Spark
Reduce duplicate-event and late-arrival handling complexity in the current processing layer
Implement data quality checks and operational documentation for the new flow
Work with internal engineers on handover, testing, and deployment sequencing
Produce a concise architecture note explaining trade-offs and remaining risks

Requirements

Technical stack needed for this mission.

5+ years of data engineering with hands-on streaming pipeline ownership
Python for pipeline logic and tooling
Kafka at production depth including topics, consumer groups, retry patterns, and schema evolution concerns
Spark for streaming or batch transformation over event data
GCP or equivalent cloud data infrastructure experience
SQL for validation, data modelling, and operational investigation