Or, what probably happened behind the scenes at the 9th September Apple Keynote stream that continued to fail
Excited to watch Apple’s new innovative products, millions of people tuned in to the live webcast where Tim Cook revealed iPhone 6, Apple Pay and Apple Watch.
As someone following the live stream while it happened, monitoring twitter feeds live chats during the event, one thing became clear to me immediately as the stream started to play: something was totally wrong with the stream.
On the surface, everything seemed to go well when the stream started playing around 15 minutes before the event. We saw video of the hall the event took place in, in beautiful HD, but something was off. Not everyone noticed this immediately, but I did: two songs were playing on top of each other. It sounded horrible to my ears.
With 10-15 minutes to go before the event started, you can safely assume that at this point the streaming crew were scrambling to fix the problem so the start of Tim Cook’s talk wouldn’t be underlaid the otherwise great downtempo tracks playing in the background. Just before the event started, even 3 tracks were playing as the audio from the Flint Center was faded in.
We can only guess at the cause of the multiple music tracks: Extra audio playing in the background of the computer used to do the stream (which is the wrong way to add music, and probably not what happened), two audio streams for different regions or a backup stream were accidentally combined, music was added at two different points of the encoding chain. We just don’t know.
The event began, the initial TV ad started playing as Apple likes to do at the beginning of events, but most of it was missing because of several restarts, audio errors and dropouts: They couldn’t fix the problem in time, and tried restarting the stream just before T+0:0, but to no avail. The keynote started, and the intro ad started playing, still underlaid with two, now increasingly annoying, simultaneous music tracks playing in the background.
At this point the stream started getting increasingly unstable (not due to bandwidth, however, not yet), we saw the “Apple Special Event Flint Center Schedule” color bars multiple times as the crew tried restarting the stream. After this, the sound completely stopped working as we can assume the streaming crew made an emergency decision and put the sound on mute before trying a different solution.
The Chinese Lady to the Rescue
When the ad was over and Tim Cook went on stage, the streaming crew had fixed the music issue just in time, but unfortunately the issue was replaced with another one: The voice of Tim Cook was now underlaid the voice of a female chinese translator, translating the live stream for chinese users.
To me, it seems that since they could not resolve the music issue, they went ahead and used the Chinese stream as a back-up. (Update: it seems all audio streams; the japanese, chinese and english ones, were playing simultaneously throughout the start of the event, not that they switched to another stream – see the comments below) Unfortunately, this made the stream almost completely unwatchable for english-speaking users.
By now, at T+0:05, everyone watching the stream and participating on Twitter were aware that Apple had grave problems with their stream. The stream was restarted again, showing the color bars.
And so, with Apple executives quickly advancing the keynote, skipping their usual 15-minute section on sales figures and App Store download counts, went on to introducing the iPhone 6, just mere 10 minutes after the event had begun. The restarts and chinese translation continued for another 20 minutes or so, with the stream starting to lag out – this time due to lack of bandwidth – due to the, presumably, extreme amount of people trying to watch the stream by reloading the player.
At about T+0:25, something interesting happened; the stream started coming back, being increasingly more stable, but the chinese voice was gone – this time replaced by a much more quiet japanese (or still chinese, I couldn’t tell due to low volume) voice. Apparently the streaming crew now figured out a way to lower the volume of the translator.
But, by the time Philip Schiller was talking about how noone is using camcorders anymore, the japanese/chinese voice also was gone, and the stream started getting a little more stable, still lagging out every couple of minutes. Now, however, lots of people had simply left in frustration, going to bed, to work, wanting to catch up later instead of wasting their time watching a completely broken stream.
The TWiT crew, broadcasting their live commentary to the event, was at this point getting increasingly aggravated, you could feel the exasperation coming from Sarah Lane in the studio.
It wasn’t until the very last 30 minutes of the stream, when Apple introduced Apple Watch, the stream started playing relatively smoothly.
The Web site
While all of this was going on, lots and lots of users reported errors on apple.com, the whole site going down, coming from their underlying Akamai CDN. I’m guessing that this is stemming in part from users reloading the site and all the assets to their live-reloading liveblog Apple included as something new this time.
I’m sure that at this point Apple and their streaming partner has done a complete investigation of the causes of the many problems of the stream. Here’s what I think they have found:
- They didn’t notice errors in time, probably noticing the double-music error at the same time as they went live and the audience logging in to the stream. Now they had a measly 15 minutes to debug what, I assume, was a complex issue with the music that had a rather obscure fix, and were probably panicking, scrambling to find some solution. (Apparently, they were unable to simply stop the chinese translator and use the chinese stream as a backup, although I don’t know how big a part of the audience were chinese)
- Not enough streaming capacity. Truth be told, I and many others have experienced dropouts and lag every single time Apple has streamed an event. I understand why Apple is going with a do-it-yourself solution with the streaming platform, but they maybe should consider using a partner to handle the streaming (bandwidth) aspect of the events. This would open up for more platforms to see the stream as well.
- Not consulting with CDNs to optimize their Web frontend. Apple did something new this year, hosting their own liveblog relaying news while the video was playing. To be honest, I think they were expecting issues since they always have issues, and the liveblog mostly kept working as the stream went on. But I believe the continuously reloading liveblog together with users mashing Command+R killed the site. Apple’s Web site couldn’t follow along, triggering Access Denied errors from the CDN.
I’m very interested in figuring out exactly what happened, and so far this is just speculation, so if you think I’m wrong in any of this, don’t hesitate to correct me in the comments. You can also follow me on twitter here.