Profile cover
K

AI Semantic Search Quality Engineer

Kyle Park

Full-time · Job · New_York

About the role

Search quality is the difference between a product people love and a product people abandon. Our enterprise search platform serves 60 clients who've indexed a combined 2.3 billion documents. When a user searches for "Q3 board presentation about APAC expansion," we need to return the right deck from among 14,000 board presentations — not the one about EMEA, not the Q2 version, not the draft. Semantic search gets you 80% of the way. The last 20% requires understanding document context, recency signals, organizational hierarchy, and the implicit filters that users don't type but definitely mean. You'll own search quality: defining what "good" looks like, building the evaluation infrastructure that measures it, and implementing the ranking improvements that move the needle. The search team is five engineers. You'll work most closely with the retrieval engineer who manages the vector index and the product manager who triages user complaints into actionable quality issues. Every ranking change ships behind a feature flag and gets measured against a holdout before full rollout.

Responsibilities

  • Define and maintain search quality metrics across enterprise client deployments
  • Build evaluation datasets and automated quality measurement pipelines
  • Implement ranking improvements based on relevance feedback and query analysis
  • Design hybrid retrieval strategies combining semantic and structured signals
  • Analyze search failure cases and prioritize fixes by client impact
  • Run A/B tests for ranking changes with controlled rollout

Requirements

  • 5+ years in search, information retrieval, or NLP
  • Experience with embedding models, vector search, and hybrid retrieval
  • Strong evaluation methodology: nDCG, MRR, and human relevance judgment
  • Proficiency in Python and SQL
  • Understanding of enterprise search challenges: access control, freshness, domain specificity

Benefits

  • Fully remote with flexible hours
  • Health, dental, and vision insurance
  • $3,000 annual conference and learning budget
  • Stock options with standard vesting
  • Quarterly team offsites

Keep exploring related roles

Use these topic pages to discover more roles in the same hiring cluster.

Categories

NLP & Prompt Engineering data science & analytics

Skills

python hugging face rag sql fastapi

Related remote AI jobs

Nearby roles that share similar skills, categories, or hiring intent.

Browse all jobs

Senior Product Security Engineer

RemoteAI company

Full-time Senior

Software Engineer, Full Stack - UK

RemoteAI company

Full-time

Principal Engineer

RemoteAI company

Full-time Principal

Technical Writer

RemoteAI company

Part-time

Software Engineer, Applications (Scientific Development Solutions) (High Seniority)

RemoteAI company

Full-time Senior

Senior Autonomy Engineer, Data Curation

RemoteAI company

Full-time Senior

Job Type

Full-time

Level

Senior

Language

English

Salary Range

$100k – $130k / year

AI Expertise

NLP & Prompt Engineering data science & analytics

Skills

python hugging face rag sql fastapi docker

Ready to apply for this role?

Create a free talent account in under 2 minutes.

  • Apply to verified AI companies
  • Get AI-matched job recommendations
  • Message hiring managers directly
  • Build your public AI talent profile
Create free account & apply Log in

Interested in this role? Create your free profile to apply.

Apply — create free profile