After last weekend's sustained outage at the BBC, a promised explanation has been published. It appears that several concurrent sporting events pushed application load over usual levels. Given that caching and CDNs are used extensively, this shouldn't be an issue. However:
At almost the same time we had a second problem. We use a caching layer in front of most of the products on BBC Online, and one of the pools failed. The products managed by that pool include BBC iPlayer and the BBC homepage, and the failure made all of those products inaccessible. That opened up a major incident at the same time on a second front.
Our first priority was to restore the caching layer. The failure was a complex one (we're still doing the forensics on it), and it has repeated a number of times. It was this failure that resulted in us switching the homepage to its emergency mode ("Due to technical problems, we are displaying a simplified version of the BBC Homepage"). We used the emergency page a number of times during the weekend, eventually leaving it up until we were confident that we had completely stabilised the cache.
Unfortunately, the root cause doesn't appear to have been identified or corrected because video streaming is equally unresponsive this weekend.
Related Stories
The BBC suffered a sustained partial outage over the weekend. Described as a "major incident", it affected the website and video on demand service, iPlayer. From an interim statement updated on Monday afternoon, BST:
Internal logs of the incident and how it was handled showed that database administrators, network engineers and system analysts were all called on to see if they could diagnose the problem and fix it.
Work continued throughout the weekend to try to stabilise the servers and database supporting the iPlayer and many other BBC services.
The final fixes for the problems were expected to be applied on 21 July when the vast majority of people should be able reach the web-based video services as normal.
The BBC said it would issue a statement when it knew more about the cause of the glitches.
(Score: 2) by present_arms on Sunday July 27 2014, @01:15PM
Used it over the last couple of days with no gltches, I'm on the Isle of Wight, off the south coast of England. Maybe people from other areas are affected.
http://trinity.mypclinuxos.com/
(Score: 1) by wantkitteh on Sunday July 27 2014, @01:46PM
...from North London.
(Score: 3, Informative) by present_arms on Sunday July 27 2014, @01:54PM
I checked again today, seems to be working fine, however I did notice I couldn't see the "view in HD" link for any I tried. I know that wasn't because of low bandwidth, 60Mb/s down 20 up on wifi (75 down, 35 up connected by ethernet cable).
http://trinity.mypclinuxos.com/
(Score: 2) by kaszz on Sunday July 27 2014, @10:25PM
The whole concept of a single pipe (multiple unicasts) to every viewer is technically flawed. The site and many other could do away with a lot of the non-simple fluff. It eats your equipment and power bill. And it eats the same at the provider. And sports doesn't seems that important to waste any raw materials or joule on.
(Score: 3, Informative) by isostatic on Monday July 28 2014, @10:16AM
The BBC has offered multicast live streams [bbc.co.uk] for years, very few ISPs (and certainly not the big ones - Talk Talk, BT, Sky) accept multicast traffic.
However most bandwidth is taken up by on-demand viewing, not live streams. Multicast doesn't help here.