Prompt Engineer, Agent Prompts & Evals

About the Role

We're looking for prompt and context engineers to join our product engineering team to help build AI-first products, features, and evaluations. Your mission will be to bridge the gap between model capabilities and real product experience, working with product teams to build consistent, safe, and beneficial user experiences across all product surfaces.

Key Responsibilities

• Design, test, and optimize system prompts and feature-specific prompts that shape Claude's behavior across consumer and API products.
• Build and maintain comprehensive evaluation suites that ensure model quality and consistency across product launches and updates.
• Partner closely with product teams, research teams, and safeguards to ensure new features meet quality and safety standards.
• Play a critical role in model releases, ensuring smooth rollouts and catching regressions before they impact users.
• Help build and improve the frameworks and tools that allow teams to develop and test prompts and features with confidence.
• Mentor product engineers on prompt engineering best practices and help teams build their first evaluations.
• Work in a fast-paced environment where model capabilities advance daily, requiring quick adaptation and creative problem-solving.

What We're Looking For

• 5+ years of software engineering experience with Python or similar languages.
• Demonstrated experience with LLMs and prompt engineering (through work, research, or significant personal projects).
• Strong understanding of evaluation methodologies and metrics for AI systems.
• Excellent written and verbal communication skills – you'll need to explain complex model behaviors to diverse stakeholders.
• Ability to manage multiple concurrent projects and prioritize effectively.
• Experience with version control, CI/CD, and modern software development practices.

You Might Thrive in This Role If You…

• Get excited about the nuances of how language models behave and love finding creative ways to improve their outputs.
• Enjoy being at the intersection of research and product, translating cutting-edge capabilities into user value.
• Are comfortable with ambiguity and can define success metrics for novel AI features.
• Have a strong sense of ownership and drive projects from conception to production.
• Are passionate about building AI systems that are helpful, harmless, and honest.
• Thrive in collaborative environments and enjoy teaching others.