Principal Engineer, Distributed Systems

Job Summary

We are looking for a Principal Engineer to provide technical leadership across Security Products. This is a senior individual-contributor role for an engineer who can define architecture, guide execution across multiple teams, and solve complex distributed-systems problems in security-critical infrastructure.

Responsibilities

• Establish architectural approaches for distributed systems - multi-region operation, including service placement, failover, replication, traffic management, disaster recovery, and regional independence.
• Drive decisions around consistency models, caching, invalidation, propagation, revocation, idempotency, concurrency, and ordering where correctness and security are critical.
• Design for data residency, tenant isolation, trust boundaries, blast-radius reduction, and controlled handling of sensitive security data.
• Define fault-tolerance strategies for dependencies, networks, regions, storage systems, and control-plane components, including graceful degradation and safe recovery.
• Design systems that meet four-nines availability goals while maintaining predictable low latency and high throughput under normal operation, traffic spikes, and partial failures.
• Improve the reliability and operability of critical services through SLOs, error budgets, metrics, logs, traces, audit events, alerting, incident response, and post-incident learning.
• Guide teams through architecture reviews, design reviews, implementation tradeoffs, capacity planning, load testing, performance analysis, and production readiness assessments.
• Mentor senior and staff engineers, develop technical talent, and raise the quality of engineering practice across the organization.
• Communicate architecture, tradeoffs, risks, and recommendations clearly to technical and executive audiences.
• Lead the design of high-scale authorization systems that support policy authoring, policy evaluation, access-control enforcement, auditability, and integration across many services and tenants.
• Provide technical direction for a Security Token Service, including token issuance, validation, lifecycle management, trust relationships, key rotation, revocation, and secure service-to-service access.
• Guide the design and implementation of an API Authentication Gateway that provides consistent, secure, observable authentication and authorization for CoreWeave APIs.
• Set technical standards for API design, service contracts, threat modeling, cryptographic key management, secrets handling, observability, and operational readiness.
• Partner with service teams to make authorization and authentication capabilities easy to adopt through clear interfaces, SDKs, reference implementations, documentation, and reliable integration patterns.

Requirements

• 12+ years of experience designing and building production software, including substantial experience with distributed systems and platform infrastructure.
• A track record of serving as a principal, staff-plus, distinguished, or equivalent technical leader across multiple teams or a broad technical domain.
• Deep expertise in distributed-systems design, including scalability, availability, latency, consistency, concurrency, partition tolerance, caching, replication, and failure recovery.
• Experience designing and operating security-sensitive or reliability-critical services in production.
• Strong software engineering experience with one or more systems languages or backend ecosystems, such as Go, Java, Rust, C++, Python, or equivalent.
• Demonstrated ability to move from ambiguous requirements to clear architecture, sequenced execution plans, and durable engineering outcomes.
• Experience making architecture decisions that balance security, correctness, performance, operational simplicity, and time to value.
• Strong written and verbal communication skills, including the ability to influence without direct authority and build alignment across organizational boundaries.
• A track record of improving engineering standards, mentoring senior engineers, and helping teams deliver complex systems safely.

Preferred Qualifications

• Experience designing and implementing authorization systems at high scale, including policy engines, permission models, policy distribution, decision services, enforcement points, or authorization observability.
• Experience designing or operating a Security Token Service, identity platform, credential service, or comparable security-critical control plane.
• Experience designing or operating an API Authentication Gateway, service-mesh authorization layer, or centralized authentication and authorization platform.
• Experience operating systems with four-nines availability requirements, high request volume, strict latency objectives, and demanding reliability expectations.
• Experience with multi-region architecture, regional failover, active-active or active-passive operation, replication, disaster recovery, and regional isolation.
• Experience reasoning about strong, eventual, and bounded-staleness consistency models, especially for security policy, credentials, tokens, permissions, and revocation.
• Experience designing for data residency, regional data controls, tenant isolation, trust-domain separation, and compliance-sensitive deployment models.
• Experience building systems that remain secure and useful during dependency failures, network partitions, regional outages, stale data, degraded capacity, or partial compromise.
• Experience with authentication and identity standards such as OAuth 2.0, OpenID Connect, SAML, SCIM, JWT, mTLS, SPIFFE/SPIRE, or related protocols.
• Experience with cryptography, key management, HSMs, secrets management, certificate authorities, signing keys, token validation, or secure credential lifecycle management.
• Experience with Kubernetes, cloud infrastructure, service networking, distributed storage, messaging systems, and multi-cluster operations.
• Experience building security and platform services with strong observability, including metrics, logs, traces, audit records, security events, and actionable alerting.
• Experience working with enterprise customers, regulated workloads, or hybrid and multi-cloud environments.

What We Offer

• Medical, dental, and vision insurance - 100% paid for by CoreWeave
• Company-paid Life Insurance
• Voluntary supplemental life insurance
• Short and long-term disability insurance
• Flexible Spending Account
• Health Savings Account
• Tuition Reimbursement
• Ability to Participate in Employee Stock Purchase Program (ESPP)
• Mental Wellness Benefits through Spring Health
• Family-Forming support provided by Carrot
• Paid Parental Leave
• Flexible, full-service childcare support with Kinside
• 401(k) with a generous employer match
• Flexible PTO
• Catered lunch each day in our office and data center locations
• A casual work environment
• A work culture focused on innovative disruption