Data Quality Specialist

We're seeking highly motivated Data Quality Specialists with strong analytical skills and a keen eye for detail to join our Human Data Annotation team within the Science organisation.

This is a hybrid quality reviewing and tooling role: you'll spend the majority of your time reviewing and auditing code annotations against rubrics to ensure data used for training and evaluating AI models meets a high bar, and the remainder building, maintaining, and troubleshooting the internal tooling that annotators rely on day-to-day.

You'll collaborate closely with the annotators, technical program manager, and engineer stakeholders, and contribute to refining the guidelines and processes that shape how our data is produced.

Key Responsibilities

• Generate and validate high-quality data annotations, based on guidelines and continuous feedback, for the development and evaluation of AI models
• Surface systemic issues, edge cases, and gaps in guidelines back to annotation operations and technical stakeholders
• Produce annotations yourself when needed, modeling the quality bar expected of the team
• Build and maintain internal tools and automation that streamline annotator workflows such as visualization dashboards, batch configuration scripts, output management utilities, and similar
• Troubleshoot environment, tooling, and CLI/git issues for annotators on their local machines, liaising with IT and engineering as needed

About You

• A degree in computer science, engineering, or a related field. Alternatively, 2 to 5 years of professional experience in software engineering, technical support, or developing tools
• Hands-on experience using code agents (e.g. Mistral’s vibe) in your own development workflow, and genuine interest in how they're evolving
• Proficient in at least one programming language (e.g. Python, JavaScript, or similar), with enough breadth to read and reason about code across a few core languages
• Able to apply consistent judgment against a rubric and surface edge cases, ambiguities, or gaps in guidelines
• Sustained focus and accuracy on detail-oriented, high-volume review work
• Comfortable working in a Unix-like terminal: shell basics, package managers, environment setup, and git workflows (branches, merges, resolving conflicts)
• Able to troubleshoot local development environment issues (dependencies, virtual environments, paths, permissions) across common operating systems
• Professional proficiency in English, with strong writing and comprehension skills

Nice to Have

• Prior experience in data annotation for AI/ML, especially LLM training (SFT, RLHF, preference data), evals/benchmarks, or agentic data
• Experience building an annotation team through interviews and training
• Experience supporting technical users or troubleshooting developer environments (internal tools support, DevRel, teaching assistant for coding courses, etc.)
• Fluency across multiple programming languages, or domain depth in one of: frontend, backend, DevOps, MLOps, data engineering
• Familiarity with rubric-based evaluation concepts, inter-annotator agreement, or quality measurement for human-labeled data
• Experience developing, deploying, and managing internal tooling or automation scripts