Description
Adobe’s Security Data Platform team builds and operates a petabyte-scale security data lakehouse that turns enterprise telemetry into trusted data for threat detection, investigations, compliance, and security analytics.
As a Staff Security Data Engineer, you will own full-system designs and lead critical engineering initiatives across ingestion, storage, governance, orchestration, and data serving. This is a hands-on individual contributor role: you will write and review production code, build infrastructure, define test strategy, fix complex failures, and work with other engineers to deliver the architecture you help shape.
Core Responsibilities
Platform Architecture and Engineering
- Own end-to-end designs that shape the Security Data Platform’s fundamental architecture on Databricks and AWS, using Spark, Delta Lake, and Unity Catalog.
- Build and evolve distributed batch and streaming pipelines for EDR telemetry, network logs, identity events, cloud audit logs, and new security data sources.
- Write maintainable Python and SQL, develop complex platform features, and refactor unreliable code.
- Build and maintain tested, reusable Terraform modules for AWS and Databricks infrastructure.
Data Modeling Governance and Operations
- Design layered lakehouse data models and reusable normalization patterns.
- Establish data contracts, quality checks, cataloging, classification, lineage, access controls, and retention policies with data owners.
- Engineer and operate Airflow and MWAA orchestration.
- Lead diagnosis and resolution of complex production issues across Spark, storage, orchestration, and cloud infrastructure.
Performance and Technical Leadership
- Profile workloads and lead measurable cost and performance improvements through query and Spark tuning, storage lifecycle policies, partitioning, and compute right-sizing.
- Lead cross-team technical discussions, reconcile competing requirements, and build agreement on architecture and delivery priorities.
- Partner with detection and security analytics engineers to deliver the data interfaces, governance, and pipeline capabilities needed for SIEM modernization and automated detection deployment.
- Mentor engineers through pairing, technical reviews, documentation, and knowledge sharing.
Required Qualifications
- Substantial experience building and operating production data platforms, typically 10 or more years in data or software engineering.
- A record of personally delivering complex distributed systems and leading full-system design and implementation across team boundaries.
- Deep production expertise in Spark's traditional and real-time data workloads, Databricks, Delta Lake, and lakehouse architecture.
- Strong AWS experience across storage, compute, networking, and IAM, plus production Terraform experience.
- Practical understanding of distributed systems, including partitioning, replication, consistency, back pressure, scalability, and fault tolerance.
Preferred Qualifications
- Experience with security telemetry, OCSF or comparable security schemas, and data requirements for SIEM, SOAR, threat detection, or investigations.
- Experience with Unity Catalog governance, data classification and retention, regional data controls, or multi-region platform design.
- Experience with Cribl Stream, Vector, or similar telemetry routing and enrichment tools, and platform integrations supporting automated threat detection rollout.
Expected Pay Range: The U.S. pay range for this position is $159,200 - $301,600 annually. Pay within this range varies by work location and may also depend on job-related knowledge, skills, and experience.