How to improve confidence in your Continuous Delivery pipeline through software testing
How confident are you in the viability of the software your team just released? It can be a scary question to ask, and there’s really only one right answer.
Confidence is essential for any kind of software delivery, but it’s particularly important in the world of Continuous Delivery, where your team pushes updates several times a day. That’s because if your team isn’t entirely confident in every single update, you’re never going to fully automate your Continuous Delivery pipeline.
Four types of testing to improve confidence
Quite simply, if you don’t believe you’re sending out good software every single time, Continuous Delivery will never work. The best way to achieve this confidence is through a best practice testing program built around four types of automated tests.
Static tests review the shape and structure of your code to catch errors and typos. Much like a grammar check, static tests confirm that your code is valid and error-free. They do not confirm anything about what the code actually does.
Unit tests evaluate the smallest units of software, such as a function call, by themselves in complete isolation. For example, if you have a function that adds two numbers, a unit test will confirm that it adds them correctly for a variety of inputs and that it throws an error when someone inputs non-numeric text.
Unit tests should be very direct and small, making them easy to write. However, you shouldn’t use too many unit tests because the fact that they only test things in isolation limits their value. Consider the case of a sliding door with a deadbolt lock. You can use a unit test to confirm that the door slides open and closed and you can use another unit test to confirm that the deadbolt goes in and out. Individually, they each work as intended, but the unit test doesn’t evaluate how they work together, so it would miss the fact that locking the deadbolt doesn’t actually work to lock the door, since someone can still slide the door right open.
Integration tests determine how multiple units within a single system work together by going through a simulated version of what the system should do. For instance, in the case of the sliding door and deadbolt, it would test how the door opens and closes when the lock is in use to make sure that they work together as intended.
Integration tests are generally the most important type of tests to run because they are large enough in scope that they provide a good picture of how things fit together and where things don’t work without being too enormous that they become difficult to manage.
Additionally, because integration tests are good at identifying how different units work together, they’re particularly valuable when you have different teams of people working on the same system. Specifically, integration tests can bring together these teams’ work to pinpoint any issues once the different sets of changes begin interacting in the same system.
End to end (E2E) tests
E2E tests look at the entire application as a whole, including all services and code repositories. These tests use a simulated environment to go from one end of the user experience to the other, including any interactions between multiple systems or subsystems.
For instance, if you have a search functionality on the front end and a separate service that delivers those results on the backend, E2E tests are the only way to ensure those two systems interact well together (since integration tests are limited to a single system).
Because E2E tests cover the complete experience, they have an enormous scope and are not used as extensively as integration tests.
How to structure testing to improve confidence
Having each of the four types of testing – static, unit, integration and E2E tests – in place are essential to improving confidence in your Continuous Delivery pipeline. But having each of these tests is only a start; it’s also important to structure your testing program properly.
Having the proper mix of testing ensures that you have a well-rounded base for confidence in the quality of your releases. It’s always possible to have false confidence, but if you have a wide variety of tests, then the chances of this become highly unlikely.
Specifically, the ideal mix of testing looks like this:
While you don’t need this exact mix to get started with Continuous Delivery, it should be something your team works toward long term. For example, if you don’t currently run E2E tests*, that’s understandable because these tests are a fair amount of work to get going – but they are something you will need to do at some point as your program matures.
Looking to kickstart E2E testing? If you’re just getting started and need to build E2E tests from scratch, check out Cypress. If you already have manual processes in place but need help automating them, check out testRigor.
Advanced testing to further cement confidence
Finally, once your team achieves an ideal mix of these four core tests, you can begin to further build out your testing program to truly cement confidence as you ramp up Continuous Delivery. These advanced tests include:
- Security: Scanning for vulnerability and known bad coding practices that might cause insecurities to occur.
- Updates: Checking for updates to dependencies or the code itself to proactively warn your team about anything that might be out of date and cause issues as a result.
- Code Coverage: Tracking which pieces of code have been tested to report on how much gets covered by your test suite to ensure confidence before releases. This type of testing should be automated and you should use it to develop a threshold for minimum levels of coverage.
- Automated Load Tests: Simulating different loads to monitor how they impact performance and ensure the software can handle spikes in demand as expected.
- Visual Regression Tests: Taking screenshots of the outputs of your tests and comparing them visually to existing visuals to easily identify any issues with design changes and ensure designs render correctly, since how they appear on the screen is different than how they are represented in code.
Confidence is a must for Continuous Delivery success, and the time to get started is now
Continuous Delivery is the future of software delivery. It helps teams deliver new releases faster to create an accelerated feedback cycle and increase efficiency in production. But to succeed with Continuous Delivery, your team must be confident in the quality of the software you release.
This confidence imperative sounds simple enough, but it’s easier said than done when your team begins releasing updates multiple times a day. And the only way to ensure this confidence is through a best practice testing program that covers various levels of the software as described here.
Interested in learning more and making Continuous Delivery work for your team? Contact Spaceship today to get started.