25
Sep 10, 2014
— https://sf.gl/1115
Conjecture on the livestream of the 9. sep Apple keynote
Excited to watch Apple’s new innovative products, millions of people tuned in to the live webcast where Tim Cook revealed iPhone 6, Apple Pay and Apple Watch.
As someone following the live stream while it happened, monitoring twitter feeds live chats during the event, one thing became clear to me immediately as the stream started to play: something was totally wrong with the stream.
On the surface, everything seemed to go well when the stream started playing around 15 minutes before the event. We saw video of the hall the event took place in, in beautiful HD, but something was off. Not everyone noticed this immediately, but two songs were playing on top of each other. It sounded horrible to my ears.
With 10-15 minutes to go before the event started, you can safely assume that at this point the streaming crew were trying to fix the problem so the start of Tim Cook’s talk wouldn’t be underlaid the otherwise great downtempo tracks playing in the background. Just before the event started, even 3 tracks were playing as the audio from the Flint Center was faded in.
We can only guess as to the cause of the multiple music tracks: Extra audio playing in the background of the computer/equipment used to do the stream (which is the wrong way to add music, and probably not what happened), two audio streams for different regions or a backup stream were accidentally combined, music was added at two different points of the encoding chain. We just don’t know.
The event began, the initial TV ad started playing as Apple likes to do at the beginning of events, but most of it was missing because of several restarts, audio errors and dropouts: They couldn’t fix the problem in time, and tried restarting the stream just before T+0:0, but to no avail. The keynote started, and the intro ad started playing, still underlaid with two, now increasingly annoying, simultaneous music tracks playing in the background.
When the stream was restarted, this screen was shown until the feed was buffered
At this point the stream started getting increasingly unstable (I don't think it was due to bandwidth, however, not yet), we saw the “Apple Special Event Flint Center Schedule” color bars multiple times as the crew perhaps tried restarting the stream. After this, the sound completely stopped working as we can assume the streaming crew made an emergency decision and put the sound on mute before trying a different solution.
Chinese Translation
When the ad was over and Tim Cook went on stage, the streaming crew had fixed the music issue just in time, but unfortunately the issue was replaced with another one: The voice of Tim Cook was now underlaid the voice of a female chinese translator, translating the live stream for chinese users.
To me, it seems that since they could not resolve the music issue, they went ahead and used the Chinese stream as a back-up. (Update: it seems all audio streams; the japanese, chinese and english ones, were playing simultaneously throughout the start of the event, not that they switched to another stream - see the comments below) Unfortunately, this made the stream almost completely unwatchable for english-speaking users.
By now, at T+0:05, everyone watching the stream and participating on Twitter were aware that Apple had grave problems with their stream. The stream was restarted again, showing the color bars.
With Apple executives quickly advancing the keynote, went on to introducing the iPhone 6, just mere 10 minutes after the event had begun. The restarts and chinese translation continued for another 20 minutes or so, with the stream starting to lag out and getting increasingly hard to get ahold of - this time, I suspect, due to lack of bandwidth - due to the extreme amount of people trying to watch the stream by reloading the player.
At about T+0:25, something interesting happened; the stream started coming back, being increasingly more stable, but the chinese voice was gone – this time replaced by a much lower volume Japanese voice. Apparently the streaming crew now had a way to lower the volume of the translator.
But, by the time Philip Schiller was talking about how noone is using camcorders anymore, the Japanese/chinese voice also was gone, and the stream started getting a little more stable, still lagging out every couple of minutes. Now, however, lots of people had simply left in frustration, going to bed, to work, wanting to catch up later instead of wasting their time watching a completely broken stream.
The TWiT crew, broadcasting their live commentary to the event, was at this point getting increasingly aggravated, you could feel the exasperation coming from Sarah Lane in the studio.
It wasn’t until the very last 30 minutes of the stream, when Apple introduced Apple Watch, the stream started playing relatively smoothly.
The Akamai CDN error users experienced on apple.com during the stream
The Web site
While all of this was going on, lots and lots of users reported errors on apple.com, the whole site going down, coming from their underlying Akamai CDN. I’m guessing that this is stemming in part from users reloading the site and all the assets to their live-reloading liveblog Apple included as something new this time.
Post-mortem
I’m sure that at this point Apple and their streaming partner has done a complete investigation of the causes of the many problems of the stream. Here’s what I think they have found:
- Not noticing the double-music issue at the same time as they went live, and the audience logging in to the stream. Now they had a measly 15 minutes to debug what, I assume, was a complex issue with the music that had a rather obscure fix, and were probably scrambling to find some solution. (Apparently, they were unable to simply stop the chinese translator and use the chinese stream as a backup, although I don’t know how big a part of the audience were chinese).
- Not enough streaming capacity. Truth be told, I and many others have experienced dropouts and lag every single time Apple has streamed an event. I understand why Apple is going with a do-it-yourself solution with the streaming platform, but they maybe should consider using a partner to handle the streaming (bandwidth) aspect of the events. This, perhaps, would open up for more platforms to see the stream as well as many Windows users want to watch the stream.
- Not consulting with CDNs to optimize their Web frontend. Apple did something new this year, hosting their own liveblog relaying news while the video was playing. I think they were expecting issues since they always have issues, and the liveblog mostly kept working as the stream went on. But I believe the continuously reloading liveblog together with users mashing Command+R to see the stream killed the site. Apple's Web site couldn't follow along with the swaths of users, triggering Access Denied errors from the CDN.
I'm very interested in figuring out exactly what happened, and so far this is just speculation, so if you think I'm wrong in any of this, don't hesitate to correct me in the comments below.
Updated 4/oct/14: Changed title and article to reflect that this post is speculation.
Sep 11, 2014 at 0:56
I have a feeling this is what was happening, too— my only correction: fourth paragraph—it was likely the sound bed from the Flint Center, not Moscone.
Sep 11, 2014 at 1:07
I’m curious why people keep saying Apple weren’t using a CDN – http://www.apple.com itself is delivered through akamai.
Sep 11, 2014 at 3:13
I also was trying to figure stuff out while it was happening.
I do think that there is something to how the stuff was being encoded for live – how they tied into where it was stored while encoding and then how it was streamed using the CDN.
A lot of system for encoding – cloud based or otherwise need some place to put the segments and the playlists as the storage for the CDN. If it was true that they were using AWS S3 which Akamai was pulling from I could easily see that this would cause problems.
We have done extensive streaming using S3 and cloudfront, using S3 and akamai and using net storage and Akamai. The performance of S3 and Akamai is not great and would not be a good fit for live events. Normally Akamai is highly tuned to use it’s own storage, Netstorage, and only in this scenario does geographic replication work properly. Something I think needed for a live event of this scale.
Given the stream would suddenly jump back to old segments also points to something messy with playlists in that suddenly they playlist is pointing to old segment but the player wouldnt know the difference.
Not an expert but for sure the core setup was just all wrong for a live event of this scale.
fun stuff!
Sep 11, 2014 at 4:01
The only thing I can conclude is that I’m glad that I am not the person who was in charge of this. It became the lead story for too many.
Sep 11, 2014 at 5:41
The liveblog was a nice idea in theory, but it kept crashing Safari on my iPhone 5s and my retina iPad while trying to load the content. Also a sign of not enough testing being done.
Sep 11, 2014 at 7:06
If they “restarted” the stream for other reasons, that probably caused the rest if the problems. As you may know, a HTTP Live Stream consists of multiple individually addressable (by URL) video clips (typically 9 seconds long), that are referenced from a chain of M3U8 playlists, also URL addressable. For such a large expected audience, the contents of each of those URLs must be replicated across a CDN, likely translating the URLs in the playlists. During the event, it appeared to me as if these parts, or URLs, were mixed up. The stream repeatedly jumped to show clips I had already seen. And while I could come up with possible causes in the generated URLs (case sensitivity, counter overflow), none of them seemed likely, given how these URLs typically look. But now, I suspect that the “restart” of the stream reused URLs already replicated across the CDN and cached by the clients. That is, the URL generating algorithm was reset to its initial state. Which of course was the biggest mistake. (But maybe CDN restrictions forced them to.) This also explains why the stream eventually became watchable, since they finally ran out of the old URLs.
Sep 11, 2014 at 13:28
Much of the internet went on a “go-slow” protest on Wednesday, as some of the world’s largest tech companies began a protest over proposals that could create fast web lanes for some companies.
http://www.theguardian.com/technology/2014/sep/10/tech-firms-go-slow-internet-net-neutrality
Sep 11, 2014 at 14:28
[…] folks are pointing to this article which is good and I even left a comment there but I don’t think it gets to the bottom of what […]
Sep 11, 2014 at 15:13
The Japanese voice didn’t replace the Chinese voice. It was there from the beginning, together with the much louder Chinese voice.
Sep 11, 2014 at 15:19
I do a lot of live streaming myself for my business and run into many issues with current streaming partners, Ustream & Livestream being the top two I use. Neither of them could have handled this event, honestly, they have issues handling 6,000 viewers on a live event, so the impact that this even would have had on their systems would be intolerable – and much worse than what we saw. Also, they don’t support HLS, so Apple are unlikely to support it anyway.
The biggest thing that I see is that Apple has the iTunes Music Festival going on right now as well, and that is being done by their primary team, and going off without a hitch. While the viewership is probably orders of magnitude less, the skill and talent were probably sent there, thinking this would be an easier gig. That’s my guess, anyway, on the social structures. As for the technicals, the URI issue is possible, but so is simply not having a stable connection to the CDNs. It’s the internet; anything can go wrong.
Sep 11, 2014 at 15:07
The live stream was always going to be heavily subscribed which is why I stayed well away and read a live blog elsewhere instead.
@Alex: The ‘Go Slow’ was no such thing. All any of the sites did was add a buffering widget on their pages to highlight the Net Neutrality issue to end users. At no point did anyone hobble themselves in this protest. Even Wikipedia gave users a way to access their site when they went ‘dark’ in protest of SOPA & PIPA back in 2012.
Sep 11, 2014 at 15:25
I have zero streaming experience. But I have 20 years live broadcast experience. It’s possible none of the audio issues were streaming related, but within the broadcast timeline. I’d be asking where the translation was done from. Were there people on site doing this, or was it going to another site to add multiple audio language channels. In either instance, at some point these audio streams were incorrectly patched (an video/audio stream could contain 64-tracks of audio) or multiple embedded audio streams were incorrectly de-embedded and incorrectly combined before streaming. I’m sure you’re right that the ‘access denied’ errors were caused by people constantly restarting the stream or reloading the page.
Sep 11, 2014 at 18:43
Great article/comments!
The audio on the quicktime replay of the event is good. So it would seem that their were extra audio inputs in addition to the quicktime recording that was going to the place that was sending out the live stream. They probably checked the quicktime recording and the audio was fine and when they went live they noticed the extra audio and they could not isolate it quickly…with all the stops and restarts trying to figure out the problem…
Sep 11, 2014 at 19:50
[…] Si una nota amarga tuvo esta presentación fue la presentación en vivo. El video nunca se vio bien y después se interrumpió por completo. Se pensó que estos problemas se debieron a que los servidores no pudieron con la carga y el interés generado por un evento mundial como este. Pero al parecer estábamos equivocados. La presentación en video fracasó por una pobre implementación por parte de Apple. Más en este articulo. Y en este otro. […]
Sep 11, 2014 at 22:55
Sure it wasn’t Samsung playing with bandwidth and giving their own translation of the events?
Sep 11, 2014 at 22:27
I don’t think I’ve seen such an Apple event livestream successfully for years. In fact, wasn’t there no livestream at all broadcast for a few years?
Anyhow, it looked to me like fewer people were interested in the Apple Watch because that’s when the livestream cleared up (that’s my perception, it’s entirely plausible that the broadcast team’s heads are on the block for reasons outlined in the article – who knows!?)
Pity the Apple Watch reveal wasn’t as cheeky or warm as the ‘hello’ from Macintosh and the iMac.
Full disclosure: I haven’t worn a watch for over 10 years, I just don’t do jewelry. There are of course, plenty of people who do – but unlikely to be nerdy enough to be watching a livestream of an Apple product reveal.
Sep 12, 2014 at 1:17
[…] spent the first 45 minutes in agony. I get why the streaming failed so miserably now — but for it to happen at this historical moment is truly unforgivable. A […]
Sep 12, 2014 at 12:18
I’m quite sure that I streamed the WWDC 2014 keynote without a hitch. I’m in Australia. I also don’t recall any real difficulty with the 5s launch, but memories fade after a year.
I second the comments that BOTH Japanese and Chinese translations were present at the start, and also Florian’s observation that the liveblog/video page (after about the first 45 minutes) would crash mobile Safari hard every time it was opened.
I also observed that the initial TV ad (“Perspective”, I think) was horribly non-deinterlaced, which is weird given that nobody needs to interlace anymore (PAL is dead) There was terrible scan line fuzzing. That cleared up when they finally got a decent feed of Schiller happening – perhaps sooner.
Sep 12, 2014 at 13:16
I’m interested to understand why Apple wanted to limit the audience to only those with Apple H/W and Safari.
– why not use existing, proven, open streaming tech/protocols?
– did this odd (only safari) choice contribute their streaming failure?
Any ideas?
Sep 12, 2014 at 17:01
Steve- Apple used HLS streaming, which is an open protocol. VLC will play it. Android devices will play it. I’m not sure which browsers support it natively, but several do. Probably not Chrome because Google isn’t supporting MPEG4 standards in it, but there is probably a plug-in that does support it on Chrome.
HLS has been around a long time, and is pretty proven and open.
Sep 12, 2014 at 23:51
Thanks to everyone for your insightful comments.
@Kev Hamm, you’re right, UStream was probably not the best example, I’ve removed it. But I’m thinking there must be providers that can guarantee skip free broadcast, and perhaps even availability on more platforms. Of course, Apple’s events continue to be the the most seen live-streamed events ever, it’s a hard task to solve. I really hope Apple will nail it next time, like they do with all their other products.
@David + @Ben Holmes, I didnt notice the japanese track was there at the beginning too. It seems like Ben is on to something here: the different audio streams weren’t separated. A likely explanation is were hearing all the audio streams at once. I’ve updated the post to reflect this.
Regarding jumping to front of the stream, I think @Markus Persson is on to something. I didn’t notice this myself, however, I was watching via a browser on my computer and didn’t notice a lot of ‘jumping back’, I kept reloading the site and never saw the beginning of the event twice.
Something caught my eye today also; Tim Cook went on Charlie Rose talking about how broadcast TV seems to be stuck in the 70’s.
Sep 13, 2014 at 3:05
The live stream crashed my Apple TV twice. I didn’t even know that was possible. I thought I heard a German translator under the Chinese lady at the beginning, but perhaps that was Japanese? And you didn’t mention the amazing skips backwards — like a 2 year old with the TiVo remote control. It was embarrassing. So much great tech happening on stage and such terrible tech streaming to their most devoted audience.
Sep 13, 2014 at 15:37
@ Steve
The stream worked equally shitty on my Mac, my iPad, and my Windows Phone. No cross platform issues, just big issues.
Sep 13, 2014 at 16:52
[…] Simon Fredsted (via John Gruber): […]
Sep 17, 2014 at 2:23
[…] Unfortunately, the stream had some serious issues, as documented by Simon Fredsted: […]