Performance Engineer, Inference Engine

Anthropic's mission is to create reliable, interpretable, and steerable AI systems. We want AI to be safe and beneficial for our users and for society as a whole.

You will work on building and optimizing Anthropic's inference engine, which manages the entire token path between accelerator kernels and the routing layer. This role involves improving throughput, cost, reliability, and latency across all accelerator and cloud platforms.

Responsibilities:

• Improve device utilization and keep accelerators from waiting due to overheads.
• Optimize reuse of model state instead of recomputing it.
• Measure, model, and change the system for continuous improvement.
• Ensure model quality and safety across platforms.
• Work closely with safeguards and safety teams.

Minimum Qualifications:

• Understanding of LLM inference and its impact on accelerators, compute, memory, and interconnect.
• Proven ability to quickly learn and adapt to new systems.
• Strong systems programming skills in languages like Rust or C++.
• Analytical approach to performance optimization.
• Team player with a low ego and willingness to ask questions.
• Enjoy pair programming and care about societal impacts.

Preferred Qualifications:

• Experience with LLM serving engines.
• GPU/accelerator programming knowledge.
• Familiarity with OS internals and language modeling with transformers.
• Experience building allocators, caches, schedulers, or high-bandwidth transport.
• Fluency in Rust and experience with systems reproducibility.

The annual compensation range for this role is $350,000-$850,000 USD.

Logistics:

• Minimum education: Bachelor’s degree or equivalent.
• Required field of study: Relevant to the role.
• Minimum years of experience: Correlates with internal job level requirements.
• Location-based hybrid policy: 25% office time expected.
• Visa sponsorship: Available with efforts to secure visas.