Description
This job delivers complete solutions spanning all phases of the Software Development Lifecycle (SDLC). As a Senior Engineer on the Business Continuity team, you will build and operate the systems that ensure Venmo's infrastructure can withstand and recover from any failure scenario.
Responsibilities:
- Act as a hands-on contributor, while leading by example
- Mentor junior engineers
- Enjoy a high degree of independence in decision-making while being accountable for results of yourself and your team
- Be part of a team dedicated to driving the scalability and reliability of Venmo's AWS cloud infrastructure
- Contribute to initiatives: internal teams rarely have dedicated project managers
- Troubleshoot incidents, identify root causes, fix and document problems, and implement preventive measures
- Lead by example, making meaningful contributions to the improvement of engineering teams' production operations
- Develop and improve tools and automation to manage infrastructure and application configuration as code
- Enhance the quality, reliability, and stability of our infrastructure and operations
- Design, implement, and operate chaos engineering experiments to proactively identify and remediate system weaknesses
- Build and maintain disaster recovery automation, runbooks, and infrastructure across Venmo's cloud environment
- Plan and execute chaos and incident game day exercises to validate system resilience and team preparedness
- Develop and maintain backup and restore tooling and processes for critical systems and data
- Define and track resilience metrics, SLOs, and recovery time objectives for owned systems
Requirements:
- Bachelor's in computer science or related field of study
- 5+ years' experience in software development or a related field
- 3+ years' experience operating distributed applications 24x7x365, as part of a Cloud Engineering, DevOps, and/or SRE team
- Extensive hands-on experience with designing, implementing, and supporting infrastructure (AWS experience preferred) to support global-scale services
- Deep hands-on experience with IaaS and PaaS solutions from AWS (or similar cloud provider)
- Hands-on programming and scripting (Python, Java, Bash, Go)
- Hands-on experience with containers and container orchestration: Docker, Kubernetes
- Strong communication skills with the ability to understand and explain technical issues to a non-technical audience
- Experience with chaos engineering tools and frameworks (e.g., AWS Fault Injection Simulator, Gremlin, Chaos Monkey, Litmus)
- Hands-on experience designing and implementing disaster recovery solutions for distributed systems
- Experience developing and maintaining backup and restore tooling and strategies
- Experience planning and executing game day exercises or incident simulations
- Understanding of RTO/RPO requirements and how to design systems to meet them