Freelance Data Engineer — Streaming Events Migration

TerraVision AI

Remote · $24K / 14 weeks · Freelance · Posted 5 months ago

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

Short version: our event pipeline grew faster than its design. Now we need to migrate key fintech product streams onto something cleaner and easier to support. This is a 14-week freelance engagement with specific deliverables around ingestion redesign, duplicate handling, data quality, and documentation. We are not looking for strategy decks. We are looking for someone who can get into the pipeline, make sensible decisions, and leave the team with something maintainable.

Fully remote.

Key Deliverables

List the expected deliverables for this project.

Migrate key event pipelines to a cleaner streaming architecture built around Kafka and Spark
Reduce duplicate-event and late-arrival handling complexity in the current processing layer
Implement data quality checks and operational documentation for the new flow
Work with internal engineers on handover, testing, and deployment sequencing
Produce a concise architecture note explaining trade-offs and remaining risks

Requirements

Technical stack needed for this mission.

5+ years of data engineering with hands-on streaming pipeline ownership
Python for pipeline logic and tooling
Kafka at production depth including topics, consumer groups, retry patterns, and schema evolution concerns
Spark for streaming or batch transformation over event data
GCP or equivalent cloud data infrastructure experience
SQL for validation, data modelling, and operational investigation