Warcraft, DDoS and the cost of success

The biggest MMoRPG in the world nearly buckled under the combination of an influx of places, and a large scale DDoS. Scaling is the big, unsolved problem of the digital world.

Scaling is hard.

We make a lot of the reduced operational costs that a lack of physical infrastructure brings to digital business – while often forgetting the costs of the physical servers which ensure that all that digital business happens.

You only need to look back a couple of weeks to see what happens when that goes wrong. Activision Blizzard’s market-leading massively multiplayer online role-playing game, World of Warcraft, is celebrating its 10th anniversary – and the release of its latest expansion pack, Warlords of Draenor.

Now, the game is a distance down from its peak, averaging about 7m active monthly players before the expansion hit. However, 3m returning players, and intense interest in the new expansion after a year without a major game update brought the service to its knees. For example, a couple of hours after logging in on the weekend after the expansion hit, this is what I saw:

Lagnor

Given that I have maybe half an hour to play on the average evening – that’s not much use to me.

Now part of it was the increased pressure on the servers. Part of it was structural problems with the game.

But a pretty big chunk was the other problem that hits really big scale online businesses – and smaller ones to – a distributed denial of service attack:

It was significant enough for Blizzard to formally acknowledge it in the WOW forum. “We ran into a few other issues [in NA],” explained community manager Bashiok, “including a distributed denial of service attack, that resulted in increased latency.”

The dark side of the web

We’ve talked DDoS attacks before – and they’re an increasing problem for online businesses, as they can be mounted much more easily than they can be countered.

You can get some sense of the scale through various visualisations you can find around the net, including this one from DarkViking:

Under attack

It’s becoming an uncontrollable price of doing business, in what’s already an uncertain environment. Blizzard certainly made its best guess at the server capacity they’d need for the expansion – but you don’t want to over-provision, because that hits your bottom line. But unlike physical businesses, where when you under-estimate demand, you just sell out, with digital products, you can end up not providing a service customers pay for. And that’s exactly what happened here – and that’s cost the company money:

Blizzard has issued a formal apology for the server issues which plagued the launch of the Warlords of Draenor expansion. Posting on the official Battle.net forums, World of Warcraft executive producer and vice president J. Allen “Nethaera” Brack apologised for issues users encountered at launch and announced that five days extra subscription time would be added to active players’ accounts.

That’s almost certainly in the millions of lost revenue.

It’s not like Blizzard was completely unprepared – this is a shot of their Network Operations Centre, from the documentary Looking For Group, showing their monitoring and response to such attacks:

Blizzard's NOC for Warcraft

But still, they were, for a while, over-whelmed.

Every small startup dreams of massive scale – but how many are really prepared for it?