# Principal Software Engineer, Applied AI

**Company**: Okta
**Location**: San Francisco, California
**Experience**: senior
**Job type**: full-time
**Salary**: $224,000-$308,000 USD
**Category**: Engineering
**Industry**: Technology

**Apply**: https://job-boards.greenhouse.io/okta/jobs/8282012?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply
**Canonical**: https://yubhub.co/jobs/job_1d3ef2f5-d88

## Description

We are seeking a Principal Software Engineer to join the Digital Workers team at Okta, which focuses on developing autonomous agents for identity operations and security. As a Principal Software Engineer, you will play a crucial role in shaping the product and building the harnesses and evaluations that make these workers reliable.

Responsibilities:

- Build the agent harness for planning, tool use, memory, context compaction, interruption, and recovery during long-running work.

- Integrate OpenAI and Anthropic model APIs, selecting models based on measured quality, latency, and cost, with clear failure handling.

- Evaluate existing harnesses and build only the missing pieces, keeping the harness connected to AWS workflow execution and approval state.

- Ensure authorisation and sensitive writes are behind application policy checks, requiring evidence of the resulting state before marking work complete.

- Develop evaluation datasets and replay tests for valid findings, access decisions, app setup, and AI agent onboarding.

- Measure performance metrics such as false positives, missed risks, unsafe actions, completion rate, and cost.

- Improve prompts, retrieval, tools, and model routing, evaluating fine-tuning or other post-training methods when justified.

Requirements:

- Strong Python engineering skills and experience shipping an LLM application that uses tools.

- Hands-on experience building agent workflows or evaluation systems for tool-using LLM applications.

- Experience integrating model APIs, structured outputs, and asynchronous services, particularly with OpenAI or Anthropic.

- Depth in agent execution or evaluation systems, with working code and failure analysis.

- Ability to distinguish a plausible model response from a supported finding or verified action.

Nice to Have:

- Open-source contributions, agent observability, security evaluations, or synthetic evaluation data.

- Experience with fine-tuning, preference data, post-training evaluation, or model distillation using approved datasets.

- Knowledge of identity and security workflows.

Our tech stack includes Python FastAPI, AWS Step Functions, Amazon Bedrock AgentCore, PostgreSQL on Amazon RDS, and pgvector, with plans to add ClickHouse or BigQuery for analytics.

In terms of compensation, the annual base salary range for this position in the San Francisco Bay Area is between $224,000-$308,000 USD. Okta also offers equity, bonus, and benefits, including health, dental, and vision insurance, 401(k), flexible spending account, and paid leave.

## Skills

### Required
- Python
- LLM application development
- agent workflows
- evaluation systems
- OpenAI
- Anthropic

### Nice to have
- open-source contributions
- agent observability
- security evaluations
- synthetic evaluation data
- fine-tuning
- preference data
- post-training evaluation
- model distillation

---

Source: [Apply at job-boards.greenhouse.io](https://job-boards.greenhouse.io/okta/jobs/8282012?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply)