Principal Site Reliability Operations Engineer

We're looking for a Principal Site Reliability Operations Engineer with a passion for problem-solving to join our Reliability Response team. The ideal candidate will have demonstrated expertise in handling incidents and thrive in a dynamic, complex, and ever-evolving distributed environment. Your ability to identify and address root causes will be crucial for driving sustainable, long-term solutions and achieving success in this role.

As a Principal Site Reliability Operations Engineer on the Reliability Team, you will handle production incidents and improve Roblox’s incident processes. You will maintain reliability service-level objectives, lead incident resolution with determination, and collaborate with service teams to identify and implement actionable improvements during the incident postmortem process. If you are passionate about maintaining uptime in a sophisticated distributed environment full of continuous change, you'll be right at home with our Reliability team.

This role will report to the Senior Manager, Reliability and requires 3 in-office days per week.

You Will:

• Lead and manage production incidents.
• Collaborate cross-functionally to troubleshoot and resolve sophisticated technical challenges.
• Guide the implementation of incident management processes and procedures, ensuring fast and effective responses to minimize impact.
• Continually monitor system health, performance and capacity, proactively addressing potential issues.
• Conduct comprehensive post-mortem analysis to ascertain the root cause of incidents and formulate corrective measures.
• Contribute substantially to the design and improvement of system architecture to boost reliability and performance.
• Leverage coding skills to automate daily routine tasks and improve system efficiency.
• Serve in the Incident Manager On-Call rotation.
• Mentor junior team members.

You Have:

• At least 8+ years of experience in a comparable role within a Site Reliability Team.
• Advanced knowledge of systems and network infrastructure protocols.
• Demonstrated ability in managing, troubleshooting, and resolving incidents in distributed environments.
• Experience solving problems.
• An ability to distill complex technical issues into clear and concise language.
• Familiarity with at least one scripting or programming language to automate routine tasks (Python, Golang, or similar languages preferred).
• You have a Bachelor’s degree, or equivalent experience, in Computer Science, Computer Engineering, or a similar technical field.

You Are:

• A great communicator; you are able to explain complex systems clearly to stakeholders and fellow engineers.
• Able to operate in potentially ambiguous circumstances during a production incident.
• Familiar with the interactions of services in a distributed system.
• Tenacious towards driving challenging production incidents to resolution.