Member of Technical Staff (Search Crawler Analyst)

The internet contains trillions of URLs. Perplexity's crawling and storage system is complex, with multiple stages including URL discovery, crawling, parsing, and indexing. You will work at the intersection of data analysis and engineering, designing metrics, building data pipelines, and improving the quality of search and answer systems.

Responsibilities:

• Identify and diagnose quality issues in the crawling pipeline
• Train small models to optimise specific aspects of the pipeline, such as parsing quality
• Develop datasets for model training, including LLM-as-a-judge labelling pipelines
• Enhance page selection algorithms for indexing
• Design and analyse experiments to validate improvements

Qualifications:

• 4+ years of experience as a data analyst, ML engineer, or in a related role
• Strong coding skills, with the ability to write production-grade code at the level of a mid-level backend engineer
• Experience designing metrics from scratch
• Experience training ML models that have shipped to production with measurable metric improvements

Nice to have:

• Direct experience working on web crawling or indexing pipelines