Boost confidence around new releases with canary deploys

Tim Dorr
CTO and Founder

Confidence is everything. It can impact what you choose to do (or not do), how you choose to do it and even how others perceive you. And this is no different in the world of software development.

Having confidence in every new piece of software your team releases as well as the way in which you deliver those releases is critical. But when it comes to getting new features in front of customers, there’s a lot on the line. That can make achieving the necessary level of confidence difficult.

Fortunately, there are several ways your team can build confidence to ensure more successful releases and, equally as important, avoid falling into a trap of over-confidence. One of the best ways to do so is through canary deploys.

What is a canary deploy?

A canary deploy is a deployment you run for a small subset of users as a litmus test to make sure the new release runs properly before you deploy it to your entire user base. The name comes from the old practice of sending a canary into a coal mine to warn of any danger.

This approach provides added confidence in the efficacy of a new release on top of testing, since it will reveal any kind of latent bug that’s hard to see or not caught by testing. While it still carries some risk, since your deployment will go in front of real customers and can cause real problems for them if a serious issue exists, it lessens the risk since the issue only hits a small group of customers – usually 20% or less.

Why should you run a canary deploy?

Canary deploys give your team confidence that what you’ve built is correct and runs as intended in the real world before you release it to all of your customers. Deploying to only a small subset of your users minimizes any potential damage.

Beyond the increased confidence, canary deploys can also help better understand performance implications that are hard to gauge in a realistic way in a test environment. For example, it’s very difficult to properly load test in a realistic way. While you can direct a lot of traffic at a service to see if it breaks or slows down, in the real world multiple services are working together in concert and one failing (even a small one that might not be noticeable during testing) can have a cascading effect across the entire product. Next thing you know, your entire product might be slowed down or broken. Running a canary deploy is a good way to push those limits and see those implications ahead of time before they become a major outage or issue for all of your customers.

How can you get started with canary deploys?

How to initiate a canary deploy

If you use a more traditional deployment system, you typically need someone to manually intervene and designate a certain set of servers to receive a canary deploy. In Kubernetes, you can automate this process. Either way, your team will essentially run two deployments, the first of which is the canary deploy that goes to a small subset of your users.

This approach still works well in a Continuous Delivery environment because even though you’re constantly deploying code, it’s not all instantaneous. Rather, the deployment process takes time to roll out completely, so there’s always a period of time when you’re running both old and new code simultaneously. Depending on your tolerance, you can always adjust the length of this period so that it’s longer to accommodate a full canary deploy.

How to determine your canary deploy subset

Canary deploys will go to a small percentage of your customers at random, usually less than 20%. There’s no right or wrong number for this, and it depends on both the size of your customer base and your appetite for risk. For instance, the larger your customer base, the smaller percentage you can do and still hit a sizable amount of servers. Think of it this way: Facebook can roll out a canary deploy to 1% of its customer base and that’s still millions of people, but for a smaller player with a few thousand customers, 1% is only one person and that’s not enough to see any kind of impact.

Overall, it’s a balancing act between confidence and customer experience because canary deploys do expose a subset of your users to risk. That said, 5-10% is usually a safe number. You can also introduce additional steps within your canary deploy to gradually increase the number as you gain more and more confidence. Github is one provider that enables this, usually starting with 5% and then jumping to 20% before deploying to the full user base. The best approach is to start with a low number to ensure as few people as possible are impacted by the initial change and to keep it quick.

How to monitor the impact of a canary deploy

One of the most important pieces of running a canary deploy is very closely monitoring the impact after the initial rollout so that you can identify any potential issues or determine if the deployment is safe to continue.

This monitoring should cover error reporting and performance metrics. Error reporting should watch for new bug reports and, if the system allows, mark any new bugs based on whether they come from the canary deploy or production. From there, your team can determine whether any issues coming from the canary deploy are acceptable to deal with later (and then proceed with the full deployment) or are deal breakers that require the full deployment to be cancelled.

Performance metrics should look at speed and functionality, but keep in mind that the impact will be scaled down. So if performance monitoring reveals a slow down of 100 milliseconds in a canary deploy, that could extrapolate out to two full seconds if the deployment goes to your entire user base. As a result, not only monitoring performance, but also scaling out the impact of any issues (no matter how small they may seem) is essential for determining if the full deployment can proceed or needs to be cancelled.

Give your team’s confidence a boost

If you’re ready to give your team’s confidence a boost when it comes to releasing new software, canary deploys are a good place to start. For even more best practices on boosting confidence and improving your software deployments, contact Spaceship today.

Want to be first in line to get early access to Spaceship?

Be the first to know when we launch! Sign up for our waiting list below: