Production Kubernetes succeeds or fails on three things: right-sized resource requests and limits, deliberate cluster topology, and disciplined upgrade practice. Autoscaling only helps once resource accounting is accurate, because the scheduler cannot place pods correctly without it.
Resource management is the foundation
The scheduler places pods using requests. If requests are absent or arbitrary, the cluster oversubscribes, nodes become unstable and workloads are evicted unpredictably. Accurate requests, derived from observed usage rather than from guesswork, are the single highest-value configuration in a production cluster.
Limits control burst. A CPU limit that is too low throttles a workload that has spare node capacity; memory limits that are too low cause termination under load. Set memory limits close to requests for predictability, and be cautious with CPU limits on latency-sensitive services.
Use namespaces with quota and limit ranges to prevent one team's runaway deployment from consuming the cluster. Use priority classes so that critical workloads are scheduled and less critical ones are preempted under pressure, rather than everything degrading equally.
- Derive requests from observed usage, not from guesses.
- Set memory limits near requests; be cautious with CPU limits on latency-sensitive services.
- Apply namespace quotas and limit ranges per team.
- Define priority classes so preemption protects critical workloads.
Cluster topology and tenancy
The topology decision trades operational overhead against isolation. A single large cluster is simpler to operate and gives better utilisation; many smaller clusters provide stronger isolation and a smaller blast radius for control-plane incidents and noisy neighbours.
A pragmatic default is one cluster per environment per region, with namespaces for team separation, and separate clusters reserved for hard isolation requirements such as regulatory boundaries, untrusted workloads or tenants with dedicated compliance needs.
Multi-cluster management then needs a consistent control plane layer: one place to define policy, deploy workloads and observe state across clusters. Without it, drift between clusters becomes the dominant operational cost.
| Mechanism | Scales | Best for |
|---|---|---|
| Horizontal Pod Autoscaler | Pod replicas | Stateless services with CPU or custom metrics |
| Vertical Pod Autoscaler | Pod requests | Right-sizing advice, batch workloads |
| KEDA | Pod replicas from event sources | Queue and stream consumers |
| Cluster Autoscaler | Node count | General capacity following pod demand |
| Karpenter | Node provisioning | Fast, varied instance selection |
| Scheduled scaling | Replicas or nodes | Predictable daily or weekly patterns |
Hardening and supply chain
A default Kubernetes install is not a security boundary. Enforce network policies so pods can only reach what they must, run containers as non-root with read-only root filesystems and dropped capabilities, and forbid privileged containers and host path mounts.
Admission control is where policy becomes enforceable rather than advisory. Use a policy engine to reject workloads that violate the rules, and require signed images from a trusted registry so that only artefacts built by your pipeline can run.
Secrets belong in a managed secret store accessed by workload identity, not in manifest files or environment variables committed to a repository. Rotate them, and audit access to them.
- Enforce default-deny network policies between namespaces.
- Run as non-root with read-only root filesystem and dropped capabilities.
- Reject non-compliant workloads at admission rather than reviewing later.
- Require signed images and pull secrets from a managed store.
Upgrades and progressive delivery
Kubernetes releases roughly three minor versions a year, and each minor version is supported for a limited window. Falling behind compounds: skipping versions is unsupported, deprecations accumulate and the eventual upgrade becomes a project rather than a routine.
Rehearse upgrades in a staging cluster with production-shaped workloads first, then roll clusters in sequence with workloads drained gracefully. Never upgrade all clusters simultaneously, and always confirm that the workloads' manifests are free of deprecated APIs before starting.
Ship with progressive delivery. Rolling updates handle the common case; canary and blue-green approaches handle riskier changes. Whatever the mechanism, define the success metric and the automatic rollback trigger in advance.
- Stay within supported versions; never skip more than one minor release.
- Rehearse upgrades on production-shaped staging first.
- Upgrade clusters sequentially with graceful draining.
- Define automatic rollback triggers before deploying.
Frequently asked questions
How many Kubernetes clusters should we run?
A practical default is one per environment per region with namespaces for team separation, plus dedicated clusters only where hard isolation is required by regulation, tenancy or untrusted workloads.
Why do pods get evicted in production?
Almost always because resource requests are inaccurate or absent, so the scheduler overcommits nodes. Deriving requests from observed usage and setting memory limits close to requests removes most eviction incidents.
When should we use KEDA instead of CPU-based autoscaling?
When demand is better expressed by an external signal, such as queue depth, stream lag or a business metric. CPU is a lagging proxy for queue consumers, so scaling on the queue itself is faster and more accurate.
References
Engineer your next strategic platform
Estimate your project investment and delivery timeline instantly, or schedule an architectural discovery call with our engineering leadership.