Profile cover
S

AI Open Data Pipeline Developer

Samuel Pham

Freelance · Job · Chicago

About the role

Government agencies publish vast amounts of data — court records, permit filings, inspection reports, budget documents — but each one in a different format, on a different schedule, with a different API (or no API at all). Our platform ingests data from 400+ government sources, normalizes it into consistent schemas, and makes it searchable for journalists, researchers, and civic tech developers. We need a freelance pipeline developer to build new source connectors and fix the ones that keep breaking. Government websites change without notice, APIs get deprecated, file formats shift, and rate limits tighten. Each source requires a custom ingestion strategy: some offer bulk downloads, some have paginated APIs, some require scraping PDFs. You'll build robust, self-healing pipelines that handle all of it. This is a 6-month engagement, 25-30 hours per week. We have a backlog of 50 source requests prioritized by user demand. You'll ship 2-3 new connectors per week once you're ramped up, plus maintain existing ones that need attention.

Responsibilities

  • Build data ingestion connectors for government open data sources
  • Normalize diverse data formats into consistent target schemas
  • Implement self-healing logic for when source websites change
  • Handle rate limiting, pagination, and authentication for APIs
  • Monitor pipeline health and fix failures promptly
  • Document source-specific ingestion patterns and maintenance notes

Requirements

  • 3+ years building data pipelines with Python
  • Experience with web scraping, API integration, and PDF extraction
  • Familiarity with workflow orchestration tools like Airflow or Prefect
  • Comfort with messy, inconsistent real-world data sources
  • Ability to ship 2-3 connectors per week after ramp-up

Benefits

  • Fully remote freelance role
  • Competitive hourly rate
  • Civic impact — your work helps journalists and researchers
  • Flexible schedule
  • Potential for ongoing engagement

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

data science & analytics Automation & AI Workflows

Skills

python sql airflow docker aws

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

Technical Writer

RemoteAI company

Part-time

Staff Application Security Engineer

RemoteAI company

Full-time Staff

Software Engineer, Applications (Scientific Development Solutions) (High Seniority)

RemoteAI company

Full-time Senior

PhD Autonomy Engineer Intern - Computer Vision / Deep Learning Summer 2027

RemoteAI company

Full-time Intern

Job Type

Freelance

Level

Mid-Level

Language

English

Salary Range

$80k – $100k / year

AI Expertise

data science & analytics Automation & AI Workflows

Skills

python sql airflow docker aws pandas

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile