Zero-Downtime Deployments: Feature Flags and Canary Releases

Deploy is not release

Most deployment risk is not in the mechanics of shipping code but in exposing new behaviour to users. Separating the two lets you deploy frequently, verify the artefact runs correctly, and then decide when and to whom the behaviour becomes visible.

Feature flags are the mechanism. Code is deployed with the new path disabled, verified in production, and then enabled for a controlled population. If the flag evaluation is failing, the safe default is the old behaviour rather than an error.

The discipline this requires is that a flag must have a defined lifecycle. Deciding at creation whether it is a release flag, an experiment flag, an operational kill switch or a permission flag determines how it should be governed and when it should be removed.

• Deploy disabled and enable deliberately.
• Default flags to the safe behaviour on evaluation failure.
• Classify flags at creation and document their lifecycle.
• Keep permission flags separate from release flags.

Progressive rollout and metric gates

A canary release routes a small share of traffic to the new version and compares it against the baseline. The comparison must be on business-relevant metrics such as error rate, latency and conversion, not only on host health, because a version can be healthy and still wrong.

Progress in defined stages: internal users, then one percent, then ten, then the remainder, holding at each stage long enough to observe the metric. Traffic-based canaries work for stateless services; where sessions are sticky, define how a cohort is selected so the comparison remains valid.

Automate the decision. A rollout that waits for an engineer to notice an error-rate spike in a dashboard is a manual process with automated marketing. Define thresholds in advance and let the system halt or revert.

Automated rollback and kill switches

Rollback must be fast and boring. With feature flags, disabling the flag is the fastest possible mitigation and should be the first action in the runbook for most behavioural incidents. For artefact-level problems, blue-green or quick redeploy of the previous version applies.

Define the trigger signal explicitly: error rate above a threshold for a defined period, latency percentile regression, or a business metric falling below expectation. Ownership for that signal must be clear, and the automated response documented so it is not surprising during an incident.

Keep a kill switch for anything that can cause external side effects, such as outbound notifications, payments or third-party calls. Being able to stop a class of activity immediately is worth the small amount of code it costs.

• Make flag disabling the first mitigation in the runbook.
• Define rollback triggers with thresholds and durations.
• Keep kill switches for externally visible side effects.
• Rehearse rollback so it is routine rather than novel.

Flag hygiene

Flags that are never removed become permanent conditional branches that accumulate and are rarely tested in both states. Every release flag should have an owner and an expected removal date, and the platform should report stale flags.

Limit the number of evaluations per request and cache flag values, because a flag evaluation that calls a remote service on every request is both a latency and an availability risk. Default to a cached, safe value when the flag service is unavailable.

Test both states of critical flags in continuous integration. A flag whose disabled path is never exercised is a latent outage waiting for an unrelated incident to expose it.

• Give every release flag an owner and a removal date.
• Report and prune stale flags on a schedule.
• Cache flag values and degrade to a safe default.
• Test both flag states for critical paths in CI.