Links

What is You Build It You Run It

An ever-increasing number of organisations have an IT department that includes on-call product teams responsible for building and running digital services. Each team builds and runs long-lived products, and provides 24/7 support for them. This is known as You Build It You Run It, a phrase first coined by Werner Vogels in 2006:
Giving developers operational responsibilities has greatly enhanced the quality of the services, both from a customer and a technology point of view. The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it. This brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer. This customer feedback loop is essential for improving the quality of the service.
Werner Vogels, Chief Technology Officer at Amazon
As explained by Jez Humble et al in Lean Enterprise, it's a myth that IT governance standards require segregation of duties. You Build It You Run It is compatible with IT Service Management (ITSM), IT Infrastructure Library (ITIL) v3 Service Transition, ITIL v3 Service Operations, PCI-DSS for credit card payments, and SOX for USA federal securities.
There's no divide between Delivery and Operations functions in You Build It You Run It. There are no short-lived delivery teams. There are long-lived on-call product teams, who own their digital services from inception to live traffic and potentially through to decommission. They are incentivised to move fast, and provide reliability.
Operations teams are likely to exist alongside You Build It You Run It. They will manage self-hosted foundational systems, such as COTS applications and custom integrations. They also remain responsible for lower-level operational capabilities, such as on-premise Domain Name System (DNS) and firewall management.
Every organisation is different, and every You Build It You Run It implementation is different. See our operating model examples from our customers.
Moving to You Build It You Run It on O2 Priority
I joined O2 in 2016, one of the UK's major telecommunication companies, at a time when it wanted to revamp its digital services. O2 Priority was a customer loyalty service, which was vital to its customer retention strategy. At the time, O2 Priority was developed by a delivery team, and supported by an operations team. There were plenty of problems:
  • The service was plagued with many production incidents
  • Deployments were fragile, and often had to be rolled back
  • A painful deployment experience limited the delivery team's ability to deliver new features.
  • Every change request slowly went through detailed scrutiny
  • Deployments were done out of hours by the Ops team
  • The Ops team had other responsibilities, and could not fully monitor the service or automate the deployment pipelines
A cross functional team was put together to re-architect and rebuild the components of O2 Priority, its infrastructure and pipelines. The team implemented You Build It You Run It, and owned delivery and operations of the service. Fully automated pipelines were put into all environments, to ensure Continuous Delivery of both microservices and infrastructure. Zero-downtime deployments gave O2 sufficient confidence that creating change requests and attending CAB meetings were no longer required unless downtime was unavoidable. Enhanced monitoring capabilities meant the team could be proactive in detecting and fixing operational issues, faster than ever before. Measures such as circuit-breakers were also put in place, to gracefully manage third party failures downstream which were capable of affecting the service. The outcome of this change in operating model was much shorter lead times, a stable O2 Priority service with no major customer impacting incidents, and happier customers.
Ogonna Iwunze
Ogonna Iwunze Operability Engineer EE UK