Streaming music from the Klipspringer hub requires three components to interact asynchronously – the Java process that encodes and transmits the data, the web browser with the client-side code that receives and plays it, and the HTTP server that ties them together – so no wonder it’s been stuck in deadlock many times during development. But one deadlock was particularly mystifying and turned out to be related to a whole chain of quirks along the stream. I’ll focus on that for this post.
It happened when playing EP7 by Autechre, an album that has a hidden pregap track, which means that when you play the physical CD it starts with track 1, and you have to skip backwards to hear the hidden track 0. Klipspringer mimics that behavior and also starts with track 1, but when I clicked the skip-30-seconds-backwards button to get back into track 0, the track player froze, apparently deadlocked. I expected to find a bug triggered by the unusual track number, but no, a thread dump revealed that the track player was in a state where it waited for playback notification from the client on the web browser.
The notification contraption has to do with a write-ahead limit that I’ve set to about ten seconds by default, needed because without a limit the server would read, encode and transmit a whole album in just a few seconds after it started playing, and send all of it over the local network to the client, where it would have to be held in a huge buffer in the web browser. Then if there was, say, a fast-forward request to skip a few seconds ahead, all the buffered data would be discarded and the server would again encode and transmit almost the whole album from the new starting position – quite a waste of resources! So instead, the server starts by only encoding and transmitting ten seconds of audio, and expects the client to call back every second or so to say that it has played some of the data (unless it’s paused) and is ready to receive some more to fill up to the ten-second margin. If the client skips to another position, at most ten seconds of audio data needs to be discarded.
A ten second margin may sound large, but it can’t be much smaller because (this is the first of the quirks) many web browsers refuse to start playing anything from an audio element until they have buffered up at least a few seconds. Sometimes the browser even expects to get the whole file – or at least to have its content length specified – before starting to play, which of course it can’t get if the audio is a stream and not a file. My reason to implement MP3 streaming support (using LAME) was that Safari (and also other browsers on iPhones) had problems with streaming in any lossless format. I speculated that this was because Apple only wants you to use their streaming services. But one day, FLAC streaming had suddenly started to work in Safari, which I later realized was because I had accidentally removed the “Content-Type: audio/flac” header from the HTTP response. Having the audio format declared prevents streaming?! Apparently, and no browser seems to have any problem detecting the audio format without that header anyway, so I just leave it commented out in audiostream.js, to the effect that FLAC streaming works on all the browsers I’ve tried.
But anyway, the track player was stuck waiting for notification that some of the Autechre pregap track had played, and this was because none of it had played, so why hadn’t it? It turned out to be due to another buffering limit. Apparently, some web browsers don’t just feel that they have to have several seconds buffered before starting playback: when playing FLAC, they also need a certain number of bytes in the input buffer. What’s special with the particular Autechre track is that it ends with about the three minutes of silence. Silence is of course highly compressible, so compressed with FLAC, ten seconds of silence was too small to reach the playback beginning limit.
To get around this, I added some slightly tricky logic in the streaming code to consider not only the number of seconds to write ahead but also assuming that the recipient may hold up to a certain number of bytes unplayed. An experiment indicated a limit of at least 30 kB was needed, so I set it to 49152 by default (a number that will bring back memories for old C64 enthusiasts).
Although the fix works, I actually don’t think it would if the audio was completely silent – that is, all zero sample values – because that would make it compress into almost nothing and the minimum number of bytes to play would never be reached. But the silent segment has a slight flutter in the lowest bits, which audio data almost always has, and I expect that’s for precisely this reason: to prevent audio software and hardware from panicking when there is nothing to process. For instance, if you ask SoX for some silence, it actually gives you very low – inaudible – random noise.
Now the final quirk: with the added byte buffering logic, unfortunately one of the streaming formats stopped working.
Klipspringer is actually supposed to support not two but three streaming audio formats: in addition to FLAC and MP3 also Ogg FLAC, which is FLAC wrapped in an Ogg transport layer to make it more suitable for streaming. It isn’t needed for the streaming that Klipspringer does, but since it was a very small extra effort to support with libFLAC, I included it, and you can specify it as "oga". Some web browsers support it.
The problem is, the byte buffering logic depends on the samples parameter of the libFLAC write callback function to provide the number of sample frames involved, and it doesn’t when libFLAC encodes Ogg FLAC: the parameter is instead always zero. It isn’t supposed to be, and I’ve submitted a pull request to libFLAC to change the behavior. Hopefully, my patch or something similar will be adopted in the next version of libFLAC not too many years from now, and Ogg FLAC will work, but until then, Klipspringer rejects requests for the oga format unless it runs with a patched version of libFLAC.
Update, three weeks later: My pull request was merged in, so if you build your libFLAC directly from the repo, oga streaming works. Also, its maintainer alerted me to the existence of a method in the API that sets the minimum bitrate to one bit per sample, which is an alternative to the tricky buffer-assumption code. However, that method seems to have been added in libFLAC 1.4, and the standard package installed with Ubuntu is still 1.3.3, so I’m keeping it the way it is for now.
Update, eight months later: With the latest release of libFLAC, oga works for streaming, along with ogg (lossy Ogg Vorbis) since the latest format additions.