Description
About the Team
Customer Education helps customers build practical skills and confidence to use AI and OpenAI products safely and effectively. The Certification team helps individuals demonstrate competence for in-demand AI skills and helps customers and partners validate that their workforce has the skills they need to meet their AI-enabled business goals.
About the Role
As a Certification Assessment Specialist, you will design, build, test, launch, and maintain hands-on assessments behind an OpenAI technical role-based certification. Your work will help candidates demonstrate what they can do in realistic scenarios and give reviewers reliable evidence to score their work.
This is a hands-on individual contributor role reporting to the Technical Certification Lead. You will turn competency models, assessment blueprints, and scoring rubrics into working tasks, assessment environments, and workflows that capture candidate evidence and produce results reviewers can verify. You will use scripting and automation and partner with engineering on software development and delivery integrations.
You will work with the certification, measurement, engineering, and operations teams, as well as technical subject matter experts and external delivery providers. The Technical Certification Lead will coordinate alignment with learning content while keeping confidential assessment details separate from learning materials.
You will take assessments from prototype through pilot, launch, and ongoing maintenance across live and virtual delivery. As products and tools change, you will update tasks, environments, and scoring together so candidates are assessed against the intended skills.
Success means candidates have the opportunity to demonstrate the competencies required for the certification, and reviewers can consistently score that work using evidence they can trace back to the submission.
Responsibilities
- Translate assessment blueprints into realistic scenarios, candidate instructions, and hands-on tasks. Work with subject matter experts to check that tasks reflect the job and elicit appropriate evidence for each rubric.
- Define assessment environment requirements and build or configure environments candidates will use. Establish repeatable provisioning and reset workflows for accounts, tools, data, and permissions so candidates start from consistent conditions.
- Build and maintain pipelines that capture candidate submissions and other scoring evidence, apply scoring rules, and route results for review. Make scores traceable to underlying evidence and protect sensitive data under approved controls.
- Build and test model-assisted scoring with qualified human reviewers using established validation criteria. Compare model scores with calibrated expert judgments, investigate disagreements, implement agreed rules for routing cases to human review, and rerun validation tests after material changes to the model, task, or rubric.
- Test assessments end-to-end, from environment setup to evidence capture and scoring. Use dry runs and pilots to check instructions, timing, accessibility, and failure recovery; fix defects and verify fixes before release. Test scoring against multiple valid approaches, partial and incorrect submissions, and failures in the assessment environment, so a platform issue isn’t mistaken for a candidate error.
- Translate assessment needs into requirements for delivery integrations. Partner with engineering and delivery providers to test candidate access, submission, and results handoffs and troubleshoot issues.
- Prepare assessments for launch with platform and operations partners. Document setup, version dependencies, and troubleshooting so delivery teams can run assessments consistently and respond when something breaks.
- Maintain versioned tasks, environments, scoring configurations, and test fixtures. Use candidate feedback and delivery and scoring data to identify issues, update or retire tasks, and check that changes preserve intended skills and difficulty.
- Investigate technical issues raised during delivery or candidate review. Reproduce defects where possible, document what was affected, and test fixes with relevant delivery partners.
Requirements
- Personally designed and built performance-based, hands-on assessments, labs, or simulations for technical roles, such as software developers or solution architects, and taken them through testing, launch, and maintenance.
- Used scripting and automation to build and troubleshoot technical assessment environments.
- Experience working at a technology company in a fast-paced environment, setting priorities, and delivering work amid changing requirements and evolving business needs.
- Can use a blueprint and rubric to design realistic tasks and specify what evidence distinguishes competent performance from task completion.
- Built or maintained workflows that capture submissions, process evidence, and produce traceable scores or reviewer outputs.
- Implemented or validated model-assisted scoring or evaluation against expert human judgments, and can explain when a score should go to human review.
- Worked directly with external technical experts or delivery providers and can translate requirements into integrations you can test end-to-end.
- Used pilot results, candidate feedback, and scoring or delivery data to improve assessment quality.
- Understand how to build for accessibility and protect candidate evidence and confidential assessment content.
- Communicate technical tradeoffs clearly to subject matter experts, measurement specialists, platform engineers, and operations teams.
Nice to Have
- Experience building hands-on assessments for AI application development, agentic workflows, or customer deployments.
- Working with psychometricians or measurement specialists on rubric calibration, scoring consistency, or validation of performance assessments.
- Launching performance-based certification assessments across both live and remote delivery.
Salary
$207K - $230K
For more fresh new jobs, check out yubhub.co.