Description
The Government Technology Agency (GovTech) is building an in-house capability to evaluate, select and adapt AI models for government use. As a Data Scientist on this team, you will work across the full model engineering lifecycle, from evaluation and diagnosis through data preparation, training and validation.
Responsibilities:
- Design and build evaluation suites to compare models on government-relevant tasks, behaviours and costs, and establish baselines across prompting, retrieval and tool use
- Diagnose model weaknesses through observational and controlled evaluations, and determine which gaps persist after prompting, retrieval and tool-use improvements
- Set up and operate model hosting and evaluation infrastructure, and measure inference performance and cost under realistic workloads
- Prepare authorised training data: curate demonstrations and preference data, generate and filter synthetic examples, implement quality checks, and maintain strict separation between training and evaluation data
- Run post-training experiments such as supervised fine-tuning, parameter-efficient tuning, preference optimisation and distillation, varying data composition and hyperparameters in a controlled way
- Evaluate trained models against untrained baselines, quantify improvements, regressions and costs, and document training recipes so that results can be independently reproduced
- Work with government agencies to assess data readiness, identify usable data sources, and assemble training examples and evaluation cases
- Engage external partners, including research institutes and local companies, for technical exchange and knowledge transfer
- Write clear technical documentation: evaluation reports, training specifications, cost analyses, and records of both successful and unsuccessful approaches
Requirements:
- Bachelor's degree or higher in Computer Science, AI/Machine Learning, Engineering, Mathematics, Statistics or a related field, with at least 2-3 years of relevant professional experience
- Strong programming proficiency in Python and hands-on experience with the modern deep learning stack (e.g. PyTorch, Hugging Face Transformers)
- Practical experience fine-tuning or post-training language models (e.g. supervised fine-tuning, LoRA/QLoRA, DPO or similar preference optimisation methods), with the ability to debug training runs and reason about failures
- Strong grounding in evaluation methodology and experiment design: constructing baselines and held-out test sets, controlling variables, and drawing honest conclusions from results, including negative ones
- Experience building and cleaning datasets for model training and evaluation, including data pipelines, deduplication, filtering and data quality assessment, and the patience to do this work well
- Working knowledge of GPU infrastructure and inference serving (e.g. vLLM, SGLang or similar), including quantisation and the measurement of throughput and cost, in cloud or on-premises environments
- Effective and critical use of LLMs and AI coding tools: able to leverage them to move fast, while exercising judgment on output quality
- Strong communication skills, with the ability to explain technical trade-offs and evaluation findings clearly to both technical and non-technical stakeholders
- Comfort with ambiguity and a disposition towards evidence: willing to report what the results show, including when an approach does not work
Preferred Requirements:
- Experience fine-tuning models that span multiple GPUs (e.g. with FSDP or DeepSpeed)
- Experience with reinforcement learning based post-training (e.g. RLHF, PPO, GRPO) or model distillation
- Experience with continued pre-training or large-scale pre-training of language models, including building and curating large training corpora
- Experience building evaluation harnesses or benchmarks for language models, including LLM-as-judge pipelines
- Experience with synthetic data generation for model training
- Experience handling sensitive data or working in regulated environments, including data governance and data handling requirements
- Experience engaging stakeholders across organisations, such as sourcing data from partner agencies or coordinating with external research partners