The quality game changer: benefits of Continuous Delivery for QA teams
Introducing Continuous Delivery is arguably one of the most impactful changes a technical team can make.
Continuous Delivery is the process of pushing software updates to your product as soon as they are developed and tested. This practice typically results in updating your product several times a day, with the goal of giving customers a regular stream of updates.
As a result, introducing Continuous Delivery leads to several benefits, including an accelerated feedback cycle and more efficient releases, among many others. The best way to fully understand the impact of introducing Continuous Delivery is to look at the stark differences in the before and after for quality assurance (QA) teams.
The average world for QA teams without Continuous Delivery
Without Continuous Delivery, QA teams typically operate independently from development teams in a Waterfall environment.
In this world, companies require independent manual testing, meaning someone’s main responsibility is to test the software before it goes out to customers. Updates also get batched into large, infrequent releases that occur monthly or even quarterly. As a result, QA teams must conduct a full regression test of all the features of the software.
Overall, this situation creates numerous challenges:
Fully independent operations create confusion: Because the QA team operates independently from the development team, they are typically not involved in the planning process and, as a result, usually don’t have the same context on new releases as the development team does. This can cause miscommunication and confusion about what exactly new features are supposed to do. In turn, it puts the onus on QA analysts to know the app well enough to test all kinds of potential variations and outcomes based on different ways that users engage with the product.
High levels of uncertainty lead to more testing: The large amount of changes in a single release and the lack of shared context with developers also creates a lot of uncertainty for QA analysts. This uncertainty makes it difficult to write focused testing plans. So rather than select a subset of test cases that can adequately test the changes slated for release, analysts will end up doing full regression tests to cover the entire application. The same uncertainty applies to re-testing any fixes to a release before it goes to production. Altogether, this results in over-testing a release, which takes substantially more time.
Waterfall approach increases uncertainty for planning: The fact that any feedback on quality gets held until the very end of development introduces even more uncertainty. Specifically, this Waterfall approach means that QA analysts won’t know how many issues to expect until they can begin working, which has a cascading effect. Specifically, it makes it difficult to plan for how long testing will take and when the new features will be ready for other departments such as sales, marketing and customer support.
Large size of releases creates risk: The large size of releases also creates a lot of risk in both the testing and release process. From the testing side, the amount of changes occurring all at once typically affects various dependencies within the software and leads to a lot of bugs as a result. QA analysts must catch and fix these bugs before regression testing can even begin. This risk carries over into the release, making it a stressful process. Additionally, if bugs do come up during or after the release, teams must typically wait until the next release to fix them properly.
Inconsistent QA work leads to under-investments in quality: Finally, these types of large batch releases create inconsistent work for QA teams. Companies will often compensate by re-assigning QA analysts to other roles, such as product support, between releases. However, this approach can be cost inefficient, produce lower quality testing and lead to a lack of accountability for QA teams.
The new world for QA teams with Continuous Delivery
Introducing Continuous Delivery as a practice changes everything for QA teams. It does so by introducing a more agile approach that releases new software in smaller pieces, creates a steadier stream of testing work and makes quality everyone’s responsibility.
In this world, QA is involved throughout the entire development lifecycle, from planning and reviewing designs to discussing potential risks. QA analysts also conduct some form of testing every single day. This approach leads to more automated and consistent testing of smaller-sized releases, which lowers risk.
This approach not only resolves the challenges noted above, but it also leads to several benefits:
Better collaboration leads to more clarity: One of the first benefits of Continuous Delivery for QA teams is an opportunity for better collaboration with developers. For example, it’s a best practice for the developer that worked on the changes and a QA analyst to collaborate on a test plan for that set of changes. The developer can alert the analyst to areas of the code that need additional testing and/or let them know that a certain set of test cases is not necessary. Along the way, the QA and development teams should work together to change the definition of “done” for stories to be when they are through QA, merged and deployed all the way to production.
Increased automation leads to improved coverage: The frequency of testing in a Continuous Delivery environment combined with the fact that QA analysts are tasked with testing smaller changes rather than large batches of changes all at once allows for more automation of unit tests. In turn, this automation allows for improved coverage of basic use cases and frees analysts' time to focus on more advanced testing. For instance, it allows analysts to think through important dependencies within the software and explore edge cases to find more bugs before shipping new code to production. Ultimately, this improved coverage leads to higher levels of confidence when shipping the updates.
Smaller changes lead to more efficient and lower cost testing: When big batches of changes occur all at once, it becomes difficult for QA analysts to determine what caused a particular issue. But with a small code change, it’s easy for an analyst to identify the cause of the issue and show developers the difference between the test environment and the production environment. Overall, this makes for more efficient testing that brings new features to production faster since it’s easier for both QA and development teams to pinpoint the source of an error and revert any issues. This focus and efficiency also reduces the amount of time it takes from identifying a bug to resolving it. It may seem counterintuitive that testing more often makes testing more time and cost efficient, but that’s a common fallacy. Quite the opposite, testing smaller amounts of changes and doing so every day makes testing faster and more predictable and reduces costs due to improved coverage and confidence.
More regular and predictable testing improves QA investments: From a management perspective, Continuous Delivery helps improve the quality and speed of testing due to QA analysts being integrated into the development process and being able to lead more focused testing. In turn, this type of regular, predictable and more measurable testing allows managers to invest in building full-time QA teams that can develop a high level of expertise and continue to produce higher quality results due to this deeper focus.
Faster releases accelerate market feedback: Finally, Continuous Delivery offers benefits that extend well beyond the QA team. Because a Continuous Delivery environment leads to smaller, more frequent changes, it creates more opportunities to show progress to internal stakeholders like product managers and company leadership. The smaller sets of more frequent changes also make it easier to train support teams on new releases since they can learn a little bit each day rather than having to understand big changes all at once. Most importantly, it accelerates time to market for new products so that customers can begin using them right away and the team can quickly and easily iterate on the product based on user feedback, rather than spending an enormous amount of time to get a big release to market and then having to wait months to take any changes live.
How to adopt Continuous Delivery within your QA team
Given the benefits that Continuous Delivery can bring to QA teams, how can you get started introducing Continuous Delivery in your organization? The change won’t happen overnight, but there are several steps you can take to get there sooner rather than later. To start, consider these six steps:
- Embed QA into planning meetings for your development team so that analysts have the same context as any designers, product managers and developers.
- Start considering testing plans when user stories first get written. Then have QA analysts write down their test plans so the developer who worked on the set of changes can review them for any additions or deletions.
- Have the QA team track which feature flags are in active use and in what permutations so that the code paths available to customers all get adequately tested.
- Have QA analysts write test cases in advance so they can be selected a la carte for each smaller release.
- Track how much time testing takes to determine if the company needs to invest in more QA staff. Also look for opportunities to invest in professional development for QA analysts and talk to them about tooling changes that can expedite and improve the quality of testing.
- Establish a rule that any pre-existing issues in production should not be blockers for release. Instead, track and ship those issues separately.
Are you ready to get started with Continuous Delivery?
Introducing Continuous Delivery can prove to be a game changer for QA teams as well as the entire organization. Critically, the time to get started in introducing this approach is sooner rather than later, as the longer you wait, the more difficult it can become to catch up.
Interested in learning more about getting started with Continuous Delivery? Contact us today to find out how Spaceship can help.
Special thanks to Radhika Solanki for sharing her thoughts on continuous delivery as a QA leader, many of which are incoporated into this article.