AI Knowledge Base Chunking Strategy Developer

Emrex AI

Remote · $100k – $130k / year · Full-time · Posted 4 months ago

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

This role is no longer accepting applications

Don't miss the next role like this

Create a free profile with your skills. We match you to new AI roles as they open, and companies hiring can contact you directly.

Create free profile, 2 minutes

Similar open roles you can apply to now:

See all open AI jobs

About the role

RAG performance lives or dies on chunking strategy. Fixed-size character chunks miss sentence boundaries and break coherent thought units. Sentence-level chunks lose context that spans paragraphs. Semantic chunking is more principled but computationally expensive and requires a model that understands the content domain well enough to identify coherent segments. The right chunking strategy varies by content type, query distribution, and the embedding model used for retrieval. We build enterprise RAG infrastructure for 40 clients with knowledge bases ranging from 100,000 to 50 million documents. Chunking strategy is the configuration decision with the largest empirical impact on retrieval quality, and we've been using fixed strategies per deployment instead of optimizing per client. The opportunity is to build adaptive chunking: automatically selecting and tuning the chunking approach based on the content type and observed retrieval quality. You'll build the chunking strategy optimization system: implementing multiple chunking approaches (fixed, sentence, semantic, hierarchical), building the retrieval quality evaluation that compares them empirically per deployment, and designing the adaptive selection logic that chooses the best strategy per content type and tunes its parameters based on production retrieval feedback.

Fully remote.

Responsibilities

Implement diverse chunking approaches: fixed-size, sentence-boundary, semantic, and hierarchical
Build retrieval quality evaluation comparing chunking strategies on representative query-answer pairs
Design adaptive chunking selection choosing the best approach per content type and query distribution
Implement chunk parameter tuning optimizing overlap and size for retrieval performance
Create chunking evaluation benchmarks clients use to validate RAG performance improvements
Monitor production retrieval quality and trigger chunking reoptimization when quality drifts

Requirements

3+ years in NLP engineering with RAG, information retrieval, or LLM application development
Strong Python and HuggingFace skills with embedding model and vector database experience
Deep understanding of chunking strategy trade-offs and their impact on retrieval quality
Experience designing A/B evaluation frameworks for information retrieval systems
Ability to build optimization systems that operate without requiring manual tuning per deployment