Hardening a 20M-visit platform for traffic spikes and calm deploysMaking a 20 million visit platform boring again

The problem

The platform served more than 20 million visits a month from managed Kubernetes on a major cloud provider, with a third-party CDN in front. Its traffic was spiky by nature: one story or campaign could multiply requests within minutes, and those were exactly the moments it degraded. Latency climbed, error rates followed, and the team spent its biggest days firefighting.

Two things made the spikes hurt more than they needed to. Far too much traffic was reaching the origin that the edge could have served, including images and pages that barely changed between requests. And capacity didn’t respond to load on its own: getting through a spike depended on someone spotting trouble and scaling things up by hand, fast enough.

Releases had their own problems. Deploying was a manual, multi-step process, run late at night or in narrow windows, with no fast and trusted way back. When a release did go wrong, recovery meant diagnosing under pressure with live traffic on the line.

Underneath all of it, the infrastructure had grown the way successful platforms often do: pieces added under pressure, workarounds nobody had time to revisit, and no single, current picture of how a request actually flowed through the system.

What we did

We started with load testing rather than assumptions. I built load tests that reproduced realistic traffic shapes, including sudden surges concentrated on a handful of hot pages, and ran them against a production-like environment. The results didn’t match what the team expected: several of the weakest points weren’t the components everyone had been worried about.

Then we worked through the findings in four strands:

• Edge caching. We tuned cache behaviour per content type so the CDN served the bulk of page and image traffic, and the origin stopped doing repeat work for every visitor. That alone took most of the pressure off the core system during a spike.
• Autoscaling. Workloads got realistic resource requests and horizontal autoscaling on the signals that actually tracked load, with the cluster able to add nodes underneath. We re-ran the load tests to confirm scale-up kept pace with a real spike rather than lagging behind it.
• Observability. We added monitoring and alerting on leading indicators (latency, error rates, saturation), tuned to fire while a problem was forming rather than after visitors noticed.
• Release pipeline. Deploys moved to automated CI/CD with blue-green and canary releases. A new version runs alongside the old one, takes a slice of traffic, and is only promoted if health checks and error rates hold. Rolling back means shifting traffic back, which takes minutes.

None of this was a rewrite. A platform with this much live traffic can’t afford one, so we found the weak points and reinforced them one at a time while the site kept serving readers.

Where things landed

Traffic spikes are now absorbed by the edge and by autoscaling, and the team hears about a story taking off from the analytics rather than from an alert. Growth in attention is something to welcome instead of something to brace for.

Releases happen during the working day as a matter of routine. With canaries catching problems early and rollback being a traffic switch, a bad release is an inconvenience rather than an incident.

The biggest change is where engineering time goes. Far less of it goes on firefighting and manual releases, and far more on the product itself.

The problem

This was a content platform doing more than 20 million visits a month, big enough that a bad hour costs real money and real reputation. The trouble was, “big traffic day” and “site’s about to fall over” had become the same sentence.

Certain stories or campaigns would send traffic through the roof, and instead of that being good news, it was a fire drill. Pages loaded slowly, some visitors got errors, and the team ended up firefighting on the exact days they most needed things to work. On top of that, releasing new code had become something people quietly dreaded. Deployments happened late at night or in narrow windows because nobody trusted them to go smoothly during the day. That’s a rough way to build software: your best people spending their energy managing risk instead of building things.

Underneath it, the infrastructure had grown the way a lot of successful products do. Bits added under pressure, workarounds nobody had time to revisit, and a general sense that nobody had a full picture of how it all held together. It worked, mostly, until it didn’t.

What we did

I started by figuring out where the platform actually broke under load, not where people assumed it broke. That meant properly stress-testing it against realistic traffic patterns rather than guessing, and it turned up a few surprises: some of the slowest, most fragile parts weren’t the obvious ones.

From there, the work fell into a few strands. We rebuilt how the platform handled sudden surges in visitors, so it could absorb a spike without buckling, rather than relying on someone spotting trouble and reacting fast enough. We changed how content and images got served to the busiest pages, taking load off the core system so it wasn’t doing unnecessary work for every single visitor. And we put proper monitoring in place, the kind that tells you a problem is forming before your visitors notice, not after your phone starts ringing.

Just as importantly, we rebuilt how new code got released. Deployments went from a nervy, manual process to something automated, tested, and reversible. If something did go wrong after a release, the team could roll it back in minutes rather than scrambling to work out what broke and why. That single change did more for morale than almost anything else, because it meant releasing software stopped being something people had to brace for.

None of this was about ripping everything out and starting again. Big platforms with real traffic can’t afford a risky rewrite, so it was a case of finding the weak points and reinforcing them one at a time, while the site kept running and kept serving readers.

Where things landed

Big traffic days stopped being an emergency. When a story took off, the platform handled it, and the team found out about it from the analytics rather than from an alert going off. That’s a genuinely different way to run a media business: growth in attention becomes something to welcome, not something to fear.

Deployments changed character too. What used to be a late-night, hold-your-breath process became routine enough that it could happen during the working day, with people confident enough in the rollback process that a bad release was an inconvenience rather than a crisis.

The wider effect was on the engineering team’s time. Less of it went on firefighting and reacting, more of it went on the product itself. For a business built on getting content in front of readers fast, having infrastructure you can trust rather than merely tolerate makes a real difference to what the team can actually get done.