Why DevOps can’t do without automated testing
According to the Testing Trends in 2017: A Survey of Software Professionals published by Saucelabs in January 2017, today almost every company wants to embrace DevOps. Of the surveyed companies, 10% said they have already done so, while another 73% are in some stage of that journey, and 15% more are thinking about it.
But there is one thing that is preventing some businesses (or stopping them at some stage like those 73%) to set out for DevOps. They have not yet fully accepted automated testing. Without automated testing, the dream of DevOps will stay just that – a dream.
In this article, we’ll look at three ways of how automated testing is necessary for the introduction of DevOps, as well as where manual testing is in all of this (hint: it’s not gone).
A little about DevOps (What it is, what it isn’t)
Before we can start discussing how automated testing enables the flow of software through the DevOps pipeline, it’s important to have a clear picture of what DevOps is and what it is not.
DevOps is all about bringing practices from different domains together and adopting them into an IT organization. These are just some philosophies you will find integrated into DevOps:
- Lean
- Toyota Production System
- Agile
- Human Factors (for example, human error risks and cost of manpower can be reduced with the help of the automated testing company A1QA).
- Theory of Constraints
- Learning Organizations
- Safety Culture
- Organizational Change Movement
- High-trust Management Cultures
- Resilience Engineering
That’s what DevOps is, but let’s look at what it is NOT:
- A new tool
Tools do not make DevOps. They make DevOps possible, and many of the things DevOps recommends cannot be done without tools. But they are not DevOps.
- A new team
DevOps is about connecting the development (Dev) with the operational (Ops) part of IT. It is not something that should stand between Dev and Ops, but bring these two together.
- A new role
A company might be looking for a “DevOps Engineer”. This just shows their lack of knowledge about DevOps itself. Since DevOps encompasses everything that goes on in an IT organization, a DevOps Engineer would be your typical guy for everything, yet without a clear role. Instead, we should talk about Automation Engineers.
- A new body of knowledge
You won’t find a single DevOps standard or a body of knowledge. There’s no single right or wrong way to do DevOps. This varies from one company to another.
Does automated testing completely eliminate manual testing?
No, and saying that it does is wrong. Not everything can be automated, and manual testing will remain necessary.
Yes, the bulk of testing can and should be automated, but someone still needs to evaluate those results. However, we should automate what we can.
Test automation is no longer optional
If a company wants to achieve the three promises of DevOps (innovation, speed, and operational stability) it needs to automate its testing (or at least most of it).
Gene Kim explains why automated testing is necessary for each of these in his “Three Ways”:
- The First Way: Flow
The First Way talks about the left-to-right flow or the flow of software from Dev to Ops.
Jez Humble and David Farley visualized this in their book “Continuous Delivery” as a deployment pipeline, starting from the commit stage, leading to the automated acceptance testing, followed by automated capacity testing and manual testing, and finishing with the release.
Every time a developer makes a change, the pipeline starts again. According to Humble and Farley, the first commit stage, shouldn’t take longer than 10 minutes (ideally, it should take five minutes). The next two stages, automated acceptance testing and automated capacity testing, should take about an hour or so.
All of these phases are automated, and may lead to manual testing. Thanks to the previous stages, the manual stage itself shouldn’t take too long. However, not every company sees this stage in its deployment pipeline.
The deployment pipeline has two important features:
- Automation, which is necessary to move release candidates quickly through the pipeline, and
- One test is followed by another, then another… all the way to the release.
- The Second Way: Feedback
Another view of the deployment pipeline is shown in this sequence diagram. A release candidate can only pass to the next phase of the pipeline if it passed all tests in the previous phase. Failing a test at any phase means disqualification, and the new journey with a new candidate must start.
But even if a pipeline fails, it’s still not wasted, as it provides feedback to the Dev team. Thanks to automated testing, this feedback comes quickly.
- The Third Way: Continual Learning
The Third Way is all about learning from mistakes, not just getting feedback and starting all over again.
Defects can be seen as a way to learn more about testing. If a defect is found in one stage, we can ask “Why did we miss it in the previous stage?” and “What can we do to improve that test to pick up this defect next time?”
A defect can also become a way for the development team to learn. The Dev team can thus ask: “What mistake(s) we made have led to this defect?” or “What can we do to prevent making the same mistake in the future?” Thanks to the Third Way, the Dev team can learn from its mistakes, reduce the number of defects that slip through testing and increase the number of release candidates that do pass the tests.
Conclusion
“Automate everything you can” is no longer just an empty phrase people can throw around. It is a reality now. If you can automate something, automate it. Without automated testing, you cannot embark on a DevOps journey.
Of course, you can’t truly see the benefits of DevOps until you have adopted it. For instance, for 57% of respondents to the ESC Digital’s 7 DevOps Adoption Statistics for 2017, the top benefit (out of possible 7) is better collaboration. For the 43% that haven’t yet adopted DevOps, it ranks the fifth.
