Embracing Radical DevOps
Over the past twelve years, RightScale has continually worked to embrace modern development practices including agile, continuous delivery and DevOps. When we implemented each of these practices, we did so with the goal of delivering features to customers more quickly and more reliably. Recently we’ve taken a significant new step that has transformed the way RightScale is delivering software. We reorganized our engineering organization into fully self-contained teams that are empowered to make all the decisions needed to build software, from working with customers to define the features to be delivered all the way to running and supporting services in production. This change has brought many significant benefits — some expected and some not. We’ll share our path to success and the challenges we had to overcome along the way.
In October 2017, we needed to tackle a strategic project with an aggressive deadline. We decided to try something radically different. Up until then, discrete development, QA, operations, and product management teams worked together to deliver new features and releases of our products — a fairly typical setup for a twelve-year-old software company. The product managers would filter, prioritize, and group feature requests coming from various sources and present the list to the development teams who would then work on features in the order of the priorities that had been set. While this separation of duties helped keep the development teams focused on execution, we saw that there was an opportunity to engage the creativity of our developers much earlier in the process. By involving them directly in gathering customer input, they could approach development process with a direct understanding of the problems they needed to solve.
Another growing challenge we were facing was the increasingly blurry line between the roles of development and operations. We adopted Docker more than two years ago and are now in the process of implementing Kubernetes. With the use of these tools, we needed to take our DevOps practices to the next level. Configuring and running Kubernetes clusters efficiently requires an intimate knowledge of the services, their interdependencies, and their performance profiles. The ability to share that knowledge among multiple teams became harder over time as the fleet of services kept growing.
To embark on this strategic project, we created a new team that was empowered to make all the decisions needed to deliver. The members of the team came from our existing development, QA, operations, and product management groups covering all the expertise needed to deliver the project to market. The team was set up so that the members had an equal voice in making all the decisions. The product manager was responsible for bringing all of the customer input and research data to the entire team. For example, every customer call to gather requirements and feedback was open to the entire team, and the calls were recorded for those who could not attend. All feedback notes and design prototypes were also shared. Using this input, the team as a whole decided on the execution plan including which features to deliver, how those features would work, and what order to work on them. As you can imagine, the project generated a lot of excitement, and the team was highly motivated. Being able to ship features to production quickly and get feedback on them in real time from customers was very rewarding. The team was also able to fully embrace Kubernetes and came up with new ways to stand up infrastructure that helped accelerate the overall development cycle. The project ended up being a huge success: Not only was it completed on time, but everyone on the team enjoyed the whole process.
As a result of our initial success, we decided to adopt this new approach more broadly across the rest of the organization. At that point there was little doubt that this was the right thing to do. The transformation went something like this:
- First, we identified the need for several different types of teams. Some teams would be focused on creating new features, while other teams would maintain and enhance our core platform or focus on cross-team tools to support the overall development effort.
- Next we identified the specific makeup needed for each of the unified teams, spreading necessary expertise (development, QA, operations) appropriately. For example, rather than having a separate ops team, we now embedded our operations experts into each of the unified teams.
- We then split maintenance of the existing product among the teams while taking into account their specific charters. We also had to adjust the production support process accordingly.
- Finally, we updated and reassigned the compliance controls because many were still owned by the now decentralized ops team.
As with any reorganization of this magnitude, there was a bit of emotional stress and chaos initially, but overall the benefits were understood by everyone. It just made sense, especially given the success of the initial team. Interestingly, while it was clear from the onset that the technical skills needed by team members would have to become broader over time, that ended up being almost a non-issue. The developers were very quick to pick up the skill set needed to deploy and run systems in production, and our operations folks were very quick at ramping up their programming skills.
A bit more challenging was learning how to think broadly about development: Without the product team doing all the work to analyze customer needs, define precise requirements, and perform product validation, the team members now needed to think at a higher level about the customer problems beyond just the technical solutions. The flip side should be obvious: The role of a software engineer is now much more rewarding as it isn’t just about coding anymore — it’s also about understanding the problem, coming up with creative solutions, and pushing the implementation all the way to production.
In the end, the new setup has had many positive benefits. One perfect example is the adoption of Kubernetes — a tool that is hard to leverage properly without a DevOps setup but that complements it perfectly. At a broader level, we also worked to create greater transparency among the various groups in the organization. Sales has more visibility into what Engineering is producing and can better communicate expectations to customers. The Customer Success team is also able to better anticipate new features and can help shape requirements more directly. Engineering has the full context when working on features: The team members understand exactly the problem that they are trying to solve and can validate directly with customers as they develop solutions.
So was the transformation challenging? At times it certainly was. Figuring out a new development process is never simple. Was it worth it? That’s a resounding yes. There’s nothing like working with a group of highly motivated people enjoying what they are doing while delivering a ton of value to our customers. We look forward to seeing what we can accomplish with our new approach.
If working as part of an empowered team helping to shape product requirements, implementation designs, systems architecture, and running a fleet of microservices using exciting new technology sounds interesting to you, check out these opportunities to join the RightScale team!