# Staff Software Engineer, Frontier Security Team **Company**: Snowflake **Location**: Menlo Park, CA **Experience**: staff **Job type**: Full time **Salary**: $236K – $295K **Category**: Engineering **Industry**: Technology **Apply**: https://jobs.ashbyhq.com/snowflake/aeb3fed3-b66c-475f-809c-06a456be6fb7?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply **Canonical**: https://yubhub.co/jobs/job_1898afa8-b01 ## Description At Snowflake, we are powering the era of the agentic enterprise. To usher in this new era, we seek AI-native thinkers across every function who are energized by the opportunity to reinvent how they work. We are hiring a Staff Software Engineer for our Frontier Security AI team. Snowflake's Frontier Security AI teams develop production-grade LLM applications, intelligent agents, AI infrastructure, and evaluation systems for enterprise customers , products that must meet a high bar for quality, security, reliability, and efficiency while operating over sensitive data at large scale. ## Responsibilities As a Staff Software Engineer at Snowflake, you will: - Architect and build the Agentic Harness that executes complex, multi-step AI workflows across models, tools, data, and services. - Design stable interfaces for tool execution, context construction, state management, memory, permissions, retries, fallbacks, and human review. - Own agent quality end to end by building evaluation harnesses, representative datasets, automated graders, experiment pipelines, and release gates. - Convert ambiguous reports such as "the agent feels worse" into measurable failure modes, reproducible tests, and durable fixes. - Analyze production agent trajectories to identify failures in reasoning, retrieval, tool use, context, orchestration, and application code. - Close the loop between production incidents, root-cause analysis, evaluation coverage, and regression prevention. - Develop offline and online measurements for task completion, correctness, groundedness, safety, latency, reliability, and cost. - Build simulation and replay infrastructure for golden-set tests, adversarial scenarios, model comparisons, and large-scale experiments. - Improve agent efficiency through model routing, prompt and semantic caching, context compaction, tool-result management, and token optimization. - Productionize new model capabilities as secure, observable, multi-tenant services with clear operational controls. - Establish standards for evaluation design, including sampling, ground-truth quality, grader calibration, leakage prevention, and statistical significance. - Define technical direction across multiple teams and lead projects whose scope extends beyond a single service. - Mentor engineers, raise the quality of architecture reviews, and remain directly involved in implementation and debugging. ## Requirements Our ideal Staff Software Engineer will have: - 9+ years of software engineering experience, including technical leadership of complex production systems. - Direct experience shipping and operating LLM applications, AI agents, or model-backed workflows in production. - Strong background in distributed systems, service architecture, high-throughput APIs, concurrency, and failure handling. - Experience building an agent runtime, workflow engine, developer platform, evaluation system, or similar infrastructure. - Demonstrated ability to evaluate nondeterministic systems without relying on a single aggregate score. - Fluency in Python and strong proficiency in at least one systems or application language such as Java, Go, Rust, or TypeScript. - Hands-on knowledge of tool calling, structured generation, retrieval, context engineering, prompt management, and model APIs. - Experience with production observability, including structured traces, replay, metrics, logs, and incident diagnosis. - Ability to balance agent quality with latency, reliability, security, and inference cost. - Track record of setting technical direction and delivering results across organizational boundaries. - Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience. - Clear written and verbal communication with engineering, product, and leadership audiences. ## Nice to Have Bonus points for the following: - Building evaluation or observability infrastructure for agentic coding, data engineering, or analytics systems. - Designing human-evaluation programs, scoring rubrics, annotation workflows, or grader-calibration methods. - Working with multi-agent orchestration, long-running agents, asynchronous workflows, or durable execution. - Developing synthetic tasks, simulations, adversarial tests, red-team exercises, or safety guardrails. - Building retrieval systems that use vector search, hybrid search, semantic indexing, ranking, or caching. - Operating multi-tenant systems that process sensitive enterprise data. - Working with model training, fine-tuning, reinforcement learning, or feedback-driven optimization. - Evaluating and onboarding frontier models based on measured product outcomes. - Experience with databases, SQL engines, data platforms, Kubernetes, or cloud-native infrastructure. ## What We're Looking For You may be a particularly good fit if you: - Treat evaluation as part of product engineering rather than a final validation step. - Can move between agent behavior, distributed infrastructure, data analysis, and production debugging. - Question metrics that do not reconcile and design tests that can expose misleading results. - Take ownership from early architecture through deployment, operations, and measurable customer outcomes. - Prefer evidence from representative tasks and production behavior over isolated benchmark results. - Work effectively in fast-moving environments where requirements develop through experimentation. ## Skills ### Required - Python - Java - Go - Rust - TypeScript - distributed systems - service architecture - high-throughput APIs - concurrency - failure handling - LLM applications - AI agents - model-backed workflows ### Nice to have - evaluation or observability infrastructure - human-evaluation programs - multi-agent orchestration - synthetic tasks - simulations - adversarial tests - retrieval systems - model training - fine-tuning - reinforcement learning --- Source: [Apply at jobs.ashbyhq.com](https://jobs.ashbyhq.com/snowflake/aeb3fed3-b66c-475f-809c-06a456be6fb7?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply)