In my last post I introduced you to the release of my personal Roundcube skin. It's been a few months, and in the meantime a new version of Roundcube arrived bringing changes to the way skins are handled. As it turns out, my skin wasn't compatible with the new version.

Therefore I've updated the skin – now version 1.0 – with the necessary fixes, mainly compatibility with Roundcube 1.0 -- for example you can actually write emails now.

There's also a new beautiful login screen along with some UI improvements – in the Settings area, especially.

It's available on good ole github, but you can also just download the zipball if you prefer. Installation instructions are included.

Don't hesitate creating some pull requests or reach out about problems in the comments -- there's still lots more work to be done, and I'll make many more design tweaks in the coming weeks and cleaning up the code.

Excited to watch Apple’s new innovative products, millions of people tuned in to the live webcast where Tim Cook revealed iPhone 6, Apple Pay and Apple Watch.

As someone following the live stream while it happened, monitoring twitter feeds live chats during the event, one thing became clear to me immediately as the stream started to play: something was totally wrong with the stream.

On the surface, everything seemed to go well when the stream started playing around 15 minutes before the event. We saw video of the hall the event took place in, in beautiful HD, but something was off. Not everyone noticed this immediately, but two songs were playing on top of each other. It sounded horrible to my ears.

With 10-15 minutes to go before the event started, you can safely assume that at this point the streaming crew were trying to fix the problem so the start of Tim Cook’s talk wouldn’t be underlaid the otherwise great downtempo tracks playing in the background. Just before the event started, even 3 tracks were playing as the audio from the Flint Center was faded in.

We can only guess as to the cause of the multiple music tracks: Extra audio playing in the background of the computer/equipment used to do the stream (which is the wrong way to add music, and probably not what happened), two audio streams for different regions or a backup stream were accidentally combined, music was added at two different points of the encoding chain. We just don’t know.

The event began, the initial TV ad started playing as Apple likes to do at the beginning of events, but most of it was missing because of several restarts, audio errors and dropouts: They couldn’t fix the problem in time, and tried restarting the stream just before T+0:0, but to no avail. The keynote started, and the intro ad started playing, still underlaid with two, now increasingly annoying, simultaneous music tracks playing in the background.

When the stream was restarted, this screen was shown until the feed was buffered

At this point the stream started getting increasingly unstable (I don't think it was due to bandwidth, however, not yet), we saw the “Apple Special Event Flint Center Schedule” color bars multiple times as the crew perhaps tried restarting the stream.  After this, the sound completely stopped working as we can assume the streaming crew made an emergency decision and put the sound on mute before trying a different solution.

Chinese Translation

When the ad was over and Tim Cook went on stage, the streaming crew had fixed the music issue just in time, but unfortunately the issue was replaced with another one: The voice of Tim Cook was now underlaid the voice of a female chinese translator, translating the live stream for chinese users.

To me, it seems that since they could not resolve the music issue, they went ahead and used the Chinese stream as a back-up. (Update: it seems all audio streams; the japanese, chinese and english ones, were playing simultaneously throughout the start of the event, not that they switched to another stream - see the comments below) Unfortunately, this made the stream almost completely unwatchable for english-speaking users.

By now, at T+0:05, everyone watching the stream and participating on Twitter were aware that Apple had grave problems with their stream. The stream was restarted again, showing the color bars.

With Apple executives quickly advancing the keynote, went on to introducing the iPhone 6, just mere 10 minutes after the event had begun. The restarts and chinese translation continued for another 20 minutes or so, with the stream starting to lag out and getting increasingly hard to get ahold of - this time, I suspect, due to lack of bandwidth - due to the extreme amount of people trying to watch the stream by reloading the player.

At about T+0:25, something interesting happened; the stream started coming back, being increasingly more stable, but the chinese voice was gone – this time replaced by a much lower volume Japanese voice. Apparently the streaming crew now had a way to lower the volume of the translator.

But, by the time Philip Schiller was talking about how noone is using camcorders anymore, the Japanese/chinese voice also was gone, and the stream started getting a little more stable, still lagging out every couple of minutes. Now, however, lots of people had simply left in frustration, going to bed, to work, wanting to catch up later instead of wasting their time watching a completely broken stream.

The TWiT crew, broadcasting their live commentary to the event, was at this point getting increasingly aggravated, you could feel the exasperation coming from Sarah Lane in the studio.

It wasn’t until the very last 30 minutes of the stream, when Apple introduced Apple Watch, the stream started playing relatively smoothly.

The Akamai CDN error users experienced on apple.com during the stream

The Web site

While all of this was going on, lots and lots of users reported errors on apple.com, the whole site going down, coming from their underlying Akamai CDN. I’m guessing that this is stemming in part from users reloading the site and all the assets to their live-reloading liveblog Apple included as something new this time.

Post-mortem

I’m sure that at this point Apple and their streaming partner has done a complete investigation of the causes of the many problems of the stream. Here’s what I think they have found:

1. Not noticing the double-music issue at the same time as they went live, and the audience logging in to the stream. Now they had a measly 15 minutes to debug what, I assume, was a complex issue with the music that had a rather obscure fix, and were probably scrambling to find some solution. (Apparently, they were unable to simply stop the chinese translator and use the chinese stream as a backup, although I don’t know how big a part of the audience were chinese).
2. Not enough streaming capacity. Truth be told, I and many others have experienced dropouts and lag every single time Apple has streamed an event. I understand why Apple is going with a do-it-yourself solution with the streaming platform, but they maybe should consider using a partner to handle the streaming (bandwidth) aspect of the events. This, perhaps, would open up for more platforms to see the stream as well as many Windows users want to watch the stream.
3. Not consulting with CDNs to optimize their Web frontend. Apple did something new this year, hosting their own liveblog relaying news while the video was playing. I think they were expecting issues since they always have issues, and the liveblog mostly kept working as the stream went on. But I believe the continuously reloading liveblog together with users mashing Command+R to see the stream killed the site. Apple's Web site couldn't follow along with the swaths of users, triggering Access Denied errors from the CDN.

I'm very interested in figuring out exactly what happened, and so far this is just speculation, so if you think I'm wrong in any of this, don't hesitate to correct me in the comments below.

Updated 4/oct/14: Changed title and article to reflect that this post is speculation.