DevOps at Scale

DevOps at Scale

Change is the key to growth, DevOps is the agent of change

Most business can now be considered technology businesses in some form or other. Technology is an enabler in most industries and business has increasingly come to rely on it. Software is at the heart of technology. Forbes even went so far as to say that “now every company is a software company”.

Stability in software services protects revenue. Having your products available and secure protects the revenue they provide to your business. To grow your revenue you need to either introduce new services or enhance the ones you have. Growth in revenue will inevitably be realized by introducing new ideas; to introduce change into your technology landscape. DevOps at its core is a reflection of LEAN manufacturing efficiencies geared towards software development. It is about delivering change as efficiently as possible from inception to release.

The processes for implementing DevOps within single teams and small organizations are now well established. There are multiple examples of how it has been achieved successfully in “Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation” by David Farley and Jez Humble. You look at the journey of your product from ideas to release. This is known as it’s software development ‘value stream’. You can then document them with value stream maps and then start to automate, beginning with the parts of the process which are either easiest or will have the most impact, removing bottlenecks along the way. It’s easy to plan (if not always easy to implement), provided you have a homogeneous organization based around a single product.

A typical software development value stream map

What if your product has multiple value streams? You may develop your e-commerce front end in a very different way to your application back-end. What if your business is made of multiple products such as e-commerce, logistics, finance, CRM, all with multiple value streams? What if you are a systems integrator and you are doing all of these for multiple customers? They likely have very different cultures and are at very different points in their own digital transformation journey. They may also have different partners delivering different parts of their own ecosystems that you will have to work with.

Product landscapes can be complex

Planning your transformation

When implementing DevOps at this kind of scale, it’s unlikely you will have the resources or freedom to implement DevOps across your entire organization and all its products at once. There are many obstacles to implementing business transformation. People generally fear change unless they can see concrete benefits. Many of us will remember consultants coming into organizations we have worked in, promising benefits from sweeping changes that in retrospect, had no chance of being realized. The approach should be to start small, perhaps with a pilot, and use the success to build a platform with which to continue.

Identify your first transformation

The fundamental idea is to understand where change is coming from within your business. Some of your products will have a greater flow of change through them than others. Typically, the closer a system is to the consumer, the quicker the rate of change. For example, customer facing e-commerce sites need to be constantly updated to maintain consumer interest whereas internal financial and HR systems typically value stability over change. DevOps is about increasing the efficiency of change. The greatest value from DevOps will come from prioritizing the transformation of products that see the most change being introduced over the products that are more stable.

Incremental change

A common method is to start small, focus on one area and show incremental improvement. Done well, people see others happy in their work, often with more autonomy, having less failures (and less late nights!). By doing this, you can create change champions for DevOps transformation within the business who can sell the improvements that they see to colleagues within other areas of the business. The ripple effect of positive promotion is very powerful. It provides a bottom-up demand for your transformation, provided you continue to demonstrate the continuous incremental improvement of your endeavors. There is one temptation you do need to resist here. Stick to your plan and don’t try to satisfy all of the demand immediately. If you spread too thin you will end up with pools of people without the necessary skills to keep the progress of improvement at the pace that people will have come to expect. Focus quickly drifts and old habits will return the minute a bad release happens.

When you have identified the value stream you want to work with, and you have mapped it out to understand the flow of idea to release, you can understand the parts of the process that eat the most time, such as a procurement cycle, and the bottlenecks, such as a senior ops team performing multiple deployments for multiple products. These are the key activities that would benefit from improvements such as automation and process re-engineering. The automation of the technical activities from committing code, testing and release are very important. These allow for greater experimentation and remove ‘release anxiety’ by making release routine. However, these are not the only areas that will need focus. For example, if you need more compute resources to stand up a new environment, you may have to pay for it. To pay for it you will need to predict your spend, write a business case, wait for the business case review meeting, speak to procurement, wait for the PO to be raised, speak to a supplier, wait for delivery and then someone to rack it and prepare it before calling you to say it’s ready. The value stream mapping needs to include the end-to-end process.

Long lead times in a value streams are good candidates for transformation

For this specific sequence of problems, there is a relatively simple solution, use a shared platform that has adequate capacity for running your tests (and make sure it has an API driven front-end to automate your provisioning). This could take the form of an approved budget for a public cloud provider or an in-house XaaS platform. This won’t be the only problem you might encounter. Every organization is different as are their processes. You will need to be creative in your solutions, but the rule of thumb is the same, do as much of the slow stuff up-front of the value stream. If manual processes are slow and unavoidable, think about how you can batch them up to gain time by not having to use the process for every change.

Scaling up and out

You will at some point have to scale out your transformation to other value streams. This is a difficult decision and shouldn’t be taken lightly, weighing the cost of change with the benefits you expect. You may be faced with the law of diminishing returns whereby the increased amounts of effort you need to put in are yielding smaller and smaller benefits. You may also have another value stream that has significant flows of change that requires your attention. If the returns on your efforts within your existing transformation are getting smaller than the easiest transformations on your next, then it’s worth considering expanding your program.

It will also depend on the number of people you have with the experience you need. To switch to another value stream or to focus on both you will likely have to move one or some of the most experienced people from you main stream to your new one. You need to make sure this doesn’t dilute the experience and allow old habits to return. The flip-side to this is that it is an opportunity to bring new people into your experienced team to start to train them up in the arts of DevOps. This is something that you will inevitably have to do, just be careful not to do it too soon.

Use experienced engineers to scale out

Now you have started to make progress, you need to maintain it or increase it slightly. If you do too much too soon you will likely meet with the resistance to change. Too slow or you slow down and you will find that behavior starts to regress and the amount of fire-fighting of problems starts to creep back up. It is also important to be able to clearly demonstrate the progress. You need to pick the performance indicators which are important in your reporting structure. The aim of your transformation is to show that change is more efficient to implement than it was before so key things to demonstrate are:

• The time it takes to get a new idea into release,
• The number of releases you make in any period (try aiming for one a day),
• Showing that the number of failed releases are going down over time,
• Mean-time-to-resolution (MTTR).

You should try to get the same information for the value stream before you transform it and for the other value streams in your portfolio so that you can show the improvement against the baseline of your organization’s products. This is important to show how far you’ve come over time and to show people in other value streams the benefits of what you are doing. This helps create the top-down market demand for your transformation.

As you scale your transformation up and out, keep focused on the areas of highest rates of change first. This needs to take into account the current rate of change as well as the future potential rate of change. This needs to be a technical and business measure. Developers will have a good handle on the backlog their teams face but not necessarily on what is in the pipeline. For example, you may have a relatively stable product in your portfolio but a deal in your sales pipeline that is about to change all that. You need to make sure as you gather information about your value streams that you do it at all levels of your organization. To be able to do this you need executive sponsorship to provide you with the authority you require.

Summary

Driving meaningful transformation in large organizations is difficult, but not impossible. You need to understand your organization, it’s product portfolio, their road maps and the people that live with them day to day. You can use value stream mapping to understand the flow of change from idea to release to understand where the problems are. With this information you can direct your attention to where it will provide the greatest benefit. Start small and increment frequently and know how you will demonstrate this progress. Be prepared for setbacks and resistance and know that you can counter this with what you have learned and the improvements you have made. Don’t forget to apply the principle of DevOps to your DevOps transformation itself: encourage and act on fast feedback. Not every change you make will have the impact you desire. It may take reversion or revision to see the change you want. There is also a marketing exercise to be done. Have the beneficiaries of your improvements do the work for you. Create communities of your change champions and like-minded people and let the word get out so the demand is there before your get to them.

Good luck!