SlipstreamJobs tracks this role from the company's public career site. Apply directly on the employer's site.
Perplexity AI is building an AI answer engine and search product. This role sits at the intersection of data analysis and engineering, focusing on the crawling and storage system that processes trillions of URLs across multiple stages: URL discovery, crawling, parsing, and indexing.
You will diagnose and fix quality issues in the crawling pipeline, working on complex problems with many opportunities for improvement. Key responsibilities include:
- Finding and diagnosing quality issues in the crawling pipeline
- Training small ML models to optimize specific aspects of the pipeline (e.g., parsing quality)
- Building datasets for model training, including LLM-as-a-judge labeling pipelines
- Improving page selection algorithms for indexing
- Designing and analyzing experiments to validate improvements
This is a hands-on technical role requiring both analytical rigor and engineering execution. You'll own the full lifecycle from metric design through model deployment and validation.
REQUIREMENTS:
- 4+ years of experience as a data analyst, ML engineer, or related role
- Strong coding skills: production-grade code at mid-level backend engineer level
- Experience designing metrics from scratch
- Experience training ML models that shipped to production with measurable metric improvements
- Nice to have: direct experience with web crawling or indexing pipelines
About Perplexity AI
AI / Data / Infrastructure — AI answer engine and search product for consumers and enterprises.
