Senior Site Reliability Engineer, Scalability: Practices at GitLab

Job Description

GitLab.com is a unique site and it brings unique challenges it’s the biggest GitLab instance in existence. In fact, it’s one of the largest single-tenancy SaaS sites on the internet developed and run with transparency in mind. GitLab.com runs using the same tools we provide to GitLab customers running self-managed installations. The experience of our team feeds back into other engineering groups within the company, as well as self-managed customers.

SRE’s with Scalability specialization focus on how to scale both the application and the infrastructure supporting the application. This is the main difference between a Scalability SRE and other SRE’s at GitLab, is that your day to day will be spent looking at how to enable other engineers at GitLab to incorporate availability, reliability and performance considerations into their daily work. We do this by providing access to information and systems in a safe and sustainable way. We do this for our application and aim to extend this offering to our infrastructure and services. .

Some examples of projects you could work:

Working from the Scalability team’s issue tracker, driving changes required to scale GitLab at GitLab.com size.
Architectural Blueprints used to define how the component fits into a bigger picture
Developer Documentation and Guidelines used to ensure consistency between multiple components
Self-service deployment and management tools, used to ensure simpler maintenance of the component
Service Maturity Model, ensuring that the component has consistent interfaces necessary to operate a robust and reliable componentObservability and Forecasting Tools
Logging and Alerting Guidelines
Operational Runbooks
What you’ll do

Be on a PagerDuty rotation to respond to GitLab.com availability incidents and provide support for service engineers with customer incidents.
Analyze existing, create and maintain new GitLab.com Service Level Objectives.
Troubleshoot, evaluate and resolve operational challenges contributing to defined SLO’s.
Define, improve, and engage in adapting architectural application bottlenecks as observed on GitLab.com.
Work with other engineering stakeholders on resolving larger architectural bottlenecks and participate by offering GitLab.com point of view.
Work in close collaboration with software development teams to shape the future roadmap and establish strong operational readiness across teams.
Scale systems through automation, improving change velocity and reliability.
Leverage technical skills to partner with team members and be comfortable diving into a problem as needed.
Work with counterparts in other teams of the Infrastructure department to improve infrastructure running with Chef, Terraform and Kubernetes.
Make monitoring and alerting alert on symptoms and not on outages.
Document every action so your findings turn into repeatable actions and then into automation.
Debug production issues across services and levels of the stack.
What you’ll bring

Strong programming skills – Preferably with Ruby and/or Go.
Production experience with the Kubernetes ecosystem
SRE experience in running and operating distributed systems is a bonus
Are able to reason about large systems – how they work on large scale, edge cases, failure modes, behaviors.
Know your way around Linux and the Unix Shell.
Have experience in collaborating and communicating asynchronously.
Have an urge to document all the things so you don’t need to learn the same thing twice.
Have an enthusiastic, go-for-it attitude. When you see something broken, you can’t help but fix it.
Have a strong sense for action and know how to iterate through a problem quickly.
Share our values, and work in accordance with those values.
Have experience with Nginx, HAProxy, Docker, Terraform, or similar technologies.
A solid understanding of, and experience with, implementing and working with SLI/SLO
Are able to leverage GitLab as your day-to-day go-to tool.
About the team

The Practices team focuses on tools and frameworks that enable the stage groups to support their features on our production systems. These challenges exist in high-load, critical services without dedicated owners, shared architectures, and complex operational configurations. The expertise within the Practices team helps overcome these challenges by responding to technical needs, promoting uniform processes, and increasing engineering efficiency by eliminating toil.

How GitLab will support you

Benefits to support your health, finances, and well-being
All remote, asynchronous work environment
Flexible Paid Time Off
Team Member Resource Groups
Equity Compensation & Employee Stock Purchase Plan
Growth and development budget
Parental leave
Home office support
Please note that we welcome interest from candidates with varying levels of experience; many successful candidates do not meet every single requirement. Additionally, studies have shown that people from underrepresented groups are less likely to apply to a job unless they meet every single qualification. If you’re excited about this role, please apply and allow our recruiters to assess your application.