# Principal Engineer, Distributed Systems **Company**: CoreWeave **Location**: New York, NY **Work arrangement**: onsite **Experience**: senior **Job type**: full-time **Salary**: $227,000 to $303,000 **Category**: Engineering **Industry**: Technology **Apply**: https://job-boards.greenhouse.io/coreweave/jobs/4712126006?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply **Canonical**: https://yubhub.co/jobs/job_08b95ca5-94b ## Description ### Job Summary We are looking for a Principal Engineer to provide technical leadership across Security Products. This is a senior individual-contributor role for an engineer who can define architecture, guide execution across multiple teams, and solve complex distributed-systems problems in security-critical infrastructure. ### Responsibilities - Establish architectural approaches for distributed systems - multi-region operation, including service placement, failover, replication, traffic management, disaster recovery, and regional independence. - Drive decisions around consistency models, caching, invalidation, propagation, revocation, idempotency, concurrency, and ordering where correctness and security are critical. - Design for data residency, tenant isolation, trust boundaries, blast-radius reduction, and controlled handling of sensitive security data. - Define fault-tolerance strategies for dependencies, networks, regions, storage systems, and control-plane components, including graceful degradation and safe recovery. - Design systems that meet four-nines availability goals while maintaining predictable low latency and high throughput under normal operation, traffic spikes, and partial failures. - Improve the reliability and operability of critical services through SLOs, error budgets, metrics, logs, traces, audit events, alerting, incident response, and post-incident learning. - Guide teams through architecture reviews, design reviews, implementation tradeoffs, capacity planning, load testing, performance analysis, and production readiness assessments. - Mentor senior and staff engineers, develop technical talent, and raise the quality of engineering practice across the organization. - Communicate architecture, tradeoffs, risks, and recommendations clearly to technical and executive audiences. - Lead the design of high-scale authorization systems that support policy authoring, policy evaluation, access-control enforcement, auditability, and integration across many services and tenants. - Provide technical direction for a Security Token Service, including token issuance, validation, lifecycle management, trust relationships, key rotation, revocation, and secure service-to-service access. - Guide the design and implementation of an API Authentication Gateway that provides consistent, secure, observable authentication and authorization for CoreWeave APIs. - Set technical standards for API design, service contracts, threat modeling, cryptographic key management, secrets handling, observability, and operational readiness. - Partner with service teams to make authorization and authentication capabilities easy to adopt through clear interfaces, SDKs, reference implementations, documentation, and reliable integration patterns. ### Requirements - 12+ years of experience designing and building production software, including substantial experience with distributed systems and platform infrastructure. - A track record of serving as a principal, staff-plus, distinguished, or equivalent technical leader across multiple teams or a broad technical domain. - Deep expertise in distributed-systems design, including scalability, availability, latency, consistency, concurrency, partition tolerance, caching, replication, and failure recovery. - Experience designing and operating security-sensitive or reliability-critical services in production. - Strong software engineering experience with one or more systems languages or backend ecosystems, such as Go, Java, Rust, C++, Python, or equivalent. - Demonstrated ability to move from ambiguous requirements to clear architecture, sequenced execution plans, and durable engineering outcomes. - Experience making architecture decisions that balance security, correctness, performance, operational simplicity, and time to value. - Strong written and verbal communication skills, including the ability to influence without direct authority and build alignment across organizational boundaries. - A track record of improving engineering standards, mentoring senior engineers, and helping teams deliver complex systems safely. ### Preferred Qualifications - Experience designing and implementing authorization systems at high scale, including policy engines, permission models, policy distribution, decision services, enforcement points, or authorization observability. - Experience designing or operating a Security Token Service, identity platform, credential service, or comparable security-critical control plane. - Experience designing or operating an API Authentication Gateway, service-mesh authorization layer, or centralized authentication and authorization platform. - Experience operating systems with four-nines availability requirements, high request volume, strict latency objectives, and demanding reliability expectations. - Experience with multi-region architecture, regional failover, active-active or active-passive operation, replication, disaster recovery, and regional isolation. - Experience reasoning about strong, eventual, and bounded-staleness consistency models, especially for security policy, credentials, tokens, permissions, and revocation. - Experience designing for data residency, regional data controls, tenant isolation, trust-domain separation, and compliance-sensitive deployment models. - Experience building systems that remain secure and useful during dependency failures, network partitions, regional outages, stale data, degraded capacity, or partial compromise. - Experience with authentication and identity standards such as OAuth 2.0, OpenID Connect, SAML, SCIM, JWT, mTLS, SPIFFE/SPIRE, or related protocols. - Experience with cryptography, key management, HSMs, secrets management, certificate authorities, signing keys, token validation, or secure credential lifecycle management. - Experience with Kubernetes, cloud infrastructure, service networking, distributed storage, messaging systems, and multi-cluster operations. - Experience building security and platform services with strong observability, including metrics, logs, traces, audit records, security events, and actionable alerting. - Experience working with enterprise customers, regulated workloads, or hybrid and multi-cloud environments. ### What We Offer - Medical, dental, and vision insurance - 100% paid for by CoreWeave - Company-paid Life Insurance - Voluntary supplemental life insurance - Short and long-term disability insurance - Flexible Spending Account - Health Savings Account - Tuition Reimbursement - Ability to Participate in Employee Stock Purchase Program (ESPP) - Mental Wellness Benefits through Spring Health - Family-Forming support provided by Carrot - Paid Parental Leave - Flexible, full-service childcare support with Kinside - 401(k) with a generous employer match - Flexible PTO - Catered lunch each day in our office and data center locations - A casual work environment - A work culture focused on innovative disruption ## Skills ### Required - distributed systems - security-critical infrastructure - architecture - technical leadership - software engineering - Go - Java - Rust - C++ - Python ### Nice to have - authorization systems - Security Token Service - API Authentication Gateway - multi-region architecture - cryptography - key management - Kubernetes - cloud infrastructure --- Source: [Apply at job-boards.greenhouse.io](https://job-boards.greenhouse.io/coreweave/jobs/4712126006?utm_source=yubhub.co&utm_medium=jobs_feed&utm_campaign=apply)