Description
About Mistral AI
At Mistral AI, we believe in the power of AI to simplify tasks, save time, and enhance learning and creativity. Our technology is designed to integrate seamlessly into daily working life.
We are looking for a skilled and motivated Web Crawling Engineer to join our dynamic engineering team. The ideal candidate should have a solid background in distributed web crawling, scraping and data extraction, with experience using advanced tools and technologies to collect and process large-scale data from diverse web sources at large scale.
Responsibilities
As a Web crawling engineer, you will be responsible for:
- Developing and maintaining web crawlers using Go to extract data from target websites.
- Utilizing headless browsing techniques, such as Chrome DevTools, to automate and optimize data collection processes.
- Collaborating with cross-functional teams to identify, scrape, and integrate data from APIs and web pages to support business objectives.
- Creating and implementing efficient parsing patterns using tokenizers, regular expressions, XPaths, and CSS selectors to ensure accurate data extraction.
- Designing and managing distributed job queues using technologies such as Redis, Aerospike and Kubernetes to handle large-scale distributed crawling and processing tasks.
- Developing strategies to monitor and ensure data quality, accuracy, and integrity throughout the crawling and indexing process.
- Continuously improving and optimizing existing web crawling infrastructure to maximize efficiency and adapt to new challenges.
About You
Core programming and web technologies
- Proficiency in Go (Golang)/Rust/Zig for building scalable and efficient web crawlers.
- Deep understanding of TCP, UDP, TLS and HTTP/1.1,2,3 protocols and web communication.
- Knowledge of HTML, CSS, and JavaScript for parsing and navigating web content.
- Familiarity with cloud platforms (AWS, GCP), orchestration (Kubernetes, Nomad), and containerization (Docker) for deployment.
Data Structures & Algorithms
- Mastery of queues, stacks, hash maps, and other data structures for efficient data handling.
- Ability to design and optimize algorithms for large-scale web crawling.
Web Scraping & Data Acquisition
- Hands-on experience with networking and web scraping libraries.
- Understanding of how search engines work and best practices for web crawling optimization.
Databases & Data Storage
- Experience with SQL and/or NoSQL databases (knowing Aerospike is a bonus) for storing and managing crawled data.
- Familiarity with data warehousing and scalable storage solutions.
Distributed Systems & Big Data
- Knowledge of distributed systems (e.g., Hadoop, Spark) for processing large datasets.
Bonus Skills (Nice-to-Have)
- Experience with web archiving projects & tooling, open-source archiving is a big plus!
- Experience applying Machine Learning to improve crawling efficiency or accuracy.
- Experience with low-level networking programming and/or userspace TCP/IP stacks.
Hiring Process
Here is what you should expect:
- Introduction call - 35 min
- Hiring Manager Interview - 30 min
- Live-coding Interview - 45 min
- System Design Interview - 45 min
- Deep dive interview (optional) - 60min
- Culture-fit discussion - 30 min
- Reference checks
Additional Information
Location & Remote
This role is primarily based in one of our European offices , Paris, France and London, UK. We will prioritize candidates who either reside there or are open to relocating. We strongly believe in the value of in-person collaboration to foster strong relationships and seamless communication within our team. In certain specific situations, we will also consider remote candidates based in one of the countries listed in this job posting , currently France, UK, Germany, Belgium, Netherlands, Spain and Italy. In any case, we ask all new hires to visit our Paris HQ office:
- for the first week of their onboarding (accommodation and travelling covered)
- then at least 2 days per month
What we offer
💰 Competitive salary and equity
🧑⚕️ Health insurance
🚴 Transportation allowance
🥎 Sport allowance
🥕 Meal vouchers
💰 Private pension plan
🍼 Parental : Generous parental leave policy
🌎 Visa sponsorship