# AI Applications Test Engineer

**Company**: Freshfields US LLP
**Experience**: senior
**Job type**: full-time
**Salary**: $100,000 to $200,000
**Category**: Engineering
**Industry**: Consulting

**Apply**: https://freshfields.wd3.myworkdayjobs.com/en-US/FBD_101/job/New-York/Evaluation-Engineer_R-08721?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply
**Canonical**: https://yubhub.co/jobs/job_31de1853-bd0

## Description

We are seeking an AI Applications Test Engineer to build and run tests that assess the quality and reliability of our AI applications.

You will help maintain evaluation datasets, investigate failures and provide clear evidence of how AI systems perform on legal and business tasks.

**Key Responsibilities:**

- Implement automated evaluation cases and supporting scripts using established frameworks and guidance from senior engineers.

- Prepare and maintain test datasets, source documents and reference examples with legal specialists, following requirements for confidentiality, permitted data use and version control.

- Run evaluations of answer accuracy, completeness, citations, retrieval relevance, instruction following and appropriate escalation, applying agreed scoring criteria and flagging cases requiring expert judgement.

- Execute functional and integration tests for AI workflows, including document ingestion, tool calls, structured outputs and error handling, in coordination with application QA.

- Review outputs and execution traces to identify failures, reproduce issues, document supporting evidence and help distinguish data, retrieval, model and application defects.

- Compare results across model, prompt and application versions, reporting changes accurately, including uncertainty and limitations.

- Support the calibration of automated graders by comparing their scores with human reviews and investigating inconsistencies.

- Maintain regression suites, contribute to CI/CD integration and turn approved production failure examples into new test cases.

**Required Skills and Experience:**

- Minimum five years’ experience developing software demonstrated through employment, placements or substantial projects using programming languages such as Python, Java or C#.

- Foundational conceptual understanding of large language models, including their capabilities, limitations and common failure modes.

- Ability to use coding agents effectively to support development, testing and analysis.

- Working knowledge of APIs, Git, automated testing and software development lifecycle practices and CI/CD tools such as Jira and Jenkins.

- Demonstrated curiosity and ability to learn in a rapidly evolving AI landscape.

- Foundational understanding of statistics and measurement, including sampling, precision and recall.

**Compensation:**

For individuals assigned and/or hired to work in New York and California, a reasonable estimate of the compensation range for this role is $100,000 to $200,000.

## Skills

### Required
- Python
- Java
- C#
- APIs
- Git
- Jira
- Jenkins
- large language models
- statistics
- measurement

---

Source: [Apply at freshfields.wd3.myworkdayjobs.com](https://freshfields.wd3.myworkdayjobs.com/en-US/FBD_101/job/New-York/Evaluation-Engineer_R-08721?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply)