Profile cover
H

Junior AI Docstring Quality Reviewer

Hunter Carvalho

Contract · Job · New_York

About the role

We generate docstrings for 300,000 Python functions per month for software companies that want their codebases documented automatically. The generation is good. The quality control isn't — we currently do a statistical sample of 2% and rely on client feedback for the rest. When a client's codebase has 50,000 functions and 3% of docstrings are wrong, that's 1,500 documentation errors in production. This is a 5-month contract to build out the automated quality review layer. You'll write evaluation logic that checks generated docstrings against a set of quality criteria: factual consistency with the function signature, completeness of parameter documentation, presence of examples for complex functions, and style compliance with client conventions. The output is a quality score per docstring and a prioritized list of docstrings that need human review. You'll work with our lead documentation engineer and two ML engineers who built the generation system. It's a well-scoped project with clear success criteria: reduce the error rate that clients report from 3% to under 0.5%.

Responsibilities

  • Build automated quality scoring for generated docstrings across quality dimensions
  • Implement factual consistency checks between docstrings and function signatures
  • Develop parameter documentation completeness validation
  • Create style compliance checking against client coding convention configurations
  • Build a prioritized review queue routing low-confidence docstrings for human review
  • Measure quality score correlation with human reviewer accuracy

Requirements

  • 1+ year of Python development experience
  • Familiarity with docstring conventions (NumPy, Google, reStructuredText)
  • Basic understanding of LLM output evaluation techniques
  • Attention to detail in code documentation quality
  • Comfortable reading and understanding Python code in unfamiliar codebases

Benefits

  • Fully remote contract position
  • Competitive hourly rate
  • Flexible hours within US/EU overlap
  • Mentorship from lead documentation engineer
  • Potential full-time offer after contract period

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

Automation & AI Workflows NLP & Prompt Engineering

Skills

python sql openai api fastapi

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Technical Writer

RemoteAI company

Part-time

Software Engineer, Full Stack - UK

RemoteAI company

Full-time

Software Engineer, Applications (Scientific Development Solutions) (High Seniority)

RemoteAI company

Full-time Senior

Drone Platform Integration & Network Support Engineer

RemoteAI company

Full-time

PhD Autonomy Engineer Intern - Computer Vision / Deep Learning Summer 2027

RemoteAI company

Full-time Intern

Backend Engineer, Platform

RemoteAI company

Full-time

Job Type

Contract

Level

Junior

Language

English

Salary Range

Under $60k / year

AI Expertise

Automation & AI Workflows NLP & Prompt Engineering

Skills

python sql openai api fastapi

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile