Product Designer, Evals & Prompts

Anthropic's mission is to create reliable, interpretable, and steerable AI systems. We want AI to be safe and beneficial for our users and for society as a whole.

You will be a foundational member of the Product Prompt and Eval Design team, focusing on building evals that check prompts, the harness that runs them, and tools for designers. Your work ensures the model and product align with user expectations, product strategy, and safety requirements.

Key responsibilities:

• Write and revise prompts for Claude's tools, features, and behaviors; test surfaces, turn findings into prompt fixes, and confirm user prompts.
• Build graders that prove prompt fixes and automate evals; read transcripts for what evals miss.
• Develop visual, low-code eval tools for designers; assemble comparison sets, turn rubrics into graders, and compare prompt variants.
• Watch designers use these tools and simplify them.
• Support model releases by testing surfaces, writing prompt fixes, and creating prompts for new features.
• Stand up and scale the eval harness; build the test environment and keep evals green across models.
• Package what prompting can't fix for training, with attached evals.

Minimum qualifications:

• Production-quality Python.
• Experience with evaluation pipelines for LLM products.
• Experience building internal tools with interfaces for non-coders.
• Experience with test harnesses and making runs comparable.
• Experience shipping prompts or working with prompt engineers.
• Ability to read transcripts, not just scores.

Preferred qualifications:

• Experience within a model-launch cycle.
• A/B testing experience and connecting offline evals to online outcomes.
• Front-end or notebook-to-app experience.
• Turning product rubrics into training signals.
• Focus on Claude's behavior for users, not just metrics.

The annual compensation range for this role is $305,000-$385,000 USD.