Support for Vorbis and Opus, not MP3

I included MP3 encoding support in the Klipspringer platform when I worked on streaming, because at one point I had problems getting FLAC streaming to work with some browsers, and it was tempting to implement MP3 file playing support as well for symmetry. But since I don’t have any MP3s myself (mostly because I’m reluctant to rely on lossy compression methods) and nobody else has asked me to do it, I could spend my time on something more productive. There’s also the aspect that if you’ve written some code and just leave it lying around without using it, some update somewhere tends to cause it to stop working at some point without you realizing it. But… it would be fun to do, and there’s nothing more I need right now after finally creating a track splitting utility, so I started looking into MP3 decoding anyway.

Which library should I use for MP3 decoding then? No obvious choice stands out, but I realized I already had libMAD on all my computers, because it’s come along with SoX and other audio software I’ve installed, so it would be natural to use. Although it appears to have stopped being maintained decades ago, and the latest version is still numbered zero-something, it seems like the most commonly used library still, at least on Linux.

Decoding with libMAD didn’t seem difficult. I followed the minimad.c example that comes with libMAD, plus what I found in mad.h, and quickly got a simple C program working that could play MP3 files, except… it didn’t sound right: there was a strong stutter in the sound. Apparently, when I read the MP3 data in chunks (instead of mapping the entire file into memory like minimad.c does), I was either losing or inserting something in each chunk of data run through the decoder. Sure enough, others out there had had similar problems, and there was an excellent reply on Stack Overflow with code that solved them. But it was a bit of a red flag that this was needed, and fitting the solution into my code made it about three times as complex as before.

The next problem was that I couldn’t find any support for seeking in mad.h or in any simple examples around the internet. Many programs that use libMAD do support seeking, but it seemed difficult (which has reasons that we will get to). Looking for ways to seek, I also stumbled on information that several ambitious audio software projects have been using libMAD the wrong way for a long time, in a way that could lead to timing problems if nothing else, and it didn’t seem obvious that I could easily figure out how to do it right.

So I started looking for other libraries, and considered FFmpeg, which seems to be a well-maintained, widely used alternative. Its documentation contains a decode_audio.c exampe, which was easy to get working, except… it printed an error message:

Warning: the sample format the decoder produced is planar (s16p). This example will output the first channel only.

But the file I tested with was not “planar”, it was an ordinary stereo filie, and it was very clear from the sound that both channels were played! I searched for the error message on the web and found a thread from a mailing list where someone asked about it and received a remarkably dismissive reply that said it was because the official example program didn’t skip ID3 metadata in the file. So apparently, metadata was being played as if it was audio data? Okay, so how do you go about correctly skipping the metadata? I looked around and found no clear answers but was left with the impression that MP3 files can be a mess of various incoherent metadata.

But at least FFmpeg supports seeking? Yes, except… apparently it isn’t exact! Seeking to the correct position – which I’ve assumed that you can do in Klipspringer – requires some not very attractive workaround trickery. How can this be? It turns out, which is the final nail in the coffin for MP3 as far as I’m concerned, that apparently MP3 files are simply not inherently seekable.

To sum up, the MP3 format seems to be a mess, and available decoding libraries are not much help in cleaning it up. That, in addition to the fact that MP3 was never an open format, but has had problems with licensing, ownership, and legislation, leads me to the conclusion that the decision not to implement MP3 decoding was correct in the first place.

From the start of considering MP3 decoding, I was aware of the existence of the Vorbis format and had already figured that Klipspringer should support as well, because unlike MP3 it’s designed to be an open format, and from what I’ve heard it’s technically superior. In fact, I had already experimented with it before deciding to pass on MP3, and implementing Vorbis decoding with the Vorbisfile API took less effort than making the decision not to implement MP3 decoding. So Klipspringer now supports playing .ogg files, which for historical reasons is what Vorbis data wrapped up in an Ogg container (the normal thing for Vorbis) are called.

Encoding was slightly more difficult, partially due to a couple of deficiencies in the documentation, but going by encoder_example.c went fine, as well as integrating it in the Klipspringer codec package and the Klipspringer hub, and it was nice to see that streaming to Firefox just worked.

After I originally started writing this blog post, I happened to see the following on the Vorbis Wikipedia page:

Since February 2013, Xiph.Org has stated that the use of Vorbis should be deprecated in favor of the Opus codec, which is also a Xiph.Org Foundation project and also free and open-source. Compared to Vorbis, Opus can simultaneously achieve higher compression efficiency […] and lower encode/decode latency

I’m not convinced that Vorbis is “deprecated”, but anyway it felt wrong to have Vorbis and not Opus, so I looked at Opus as well. Decoding .opus files, where Opus is wrapped up in an Ogg container similarly to Vorbis, using the opusfile API was simple.

But encoding Opus turned out to be a problem. I haven’t been able to find any simple example for how to wrap up Opus data in Ogg, and the opusenc API, which provides exactly that service, lacks one thing I need to properly integrate it as a Klipspringer streaming format: the write callback doesn’t tell you how many samples there are in an encoded chunk. I managed to stop myself from reading lots of specifications and code to figure out how to do this, so for now Klipspringer only supports playing .opus files, not streaming in Opus format or writing .opus files.

With libFLAC 1.5.0, streaming in Ogg FLAC properly supplies the number of samples per packet (the same thing that the opusenc API lacks), which eliminates the problem with streaming in that format with Klipspringer. I submitted a patch for this myself, but in the end I’m not sure any its code was used, because when it had drawn attention to the problem, someone with more knowledge of Ogg API details supplied another patch that fixed it in a nicer way.

Anyway, this means that, four formats can now be used for streaming:

• FLAC (flac), lossless audio compression.
• Ogg FLAC (oga), FLAC encapsulated in an Ogg transport layer. I don’t know any good reason for using this with Klipspringer since it doesn’t take advantage of anything in the Ogg layer, and native FLAC streaming works in all browsers I’ve tried, but it works in case it’s needed.
• Vorbis (ogg), also known as Ogg Vorbis, lossy audio compression with high quality. By default, variable bitrate with quality level 0.9 is used.
• MP3 (mp3), well known older lossy audio compression format, with a bitrate of 320 kbit/second by default. (Encoding it with LAME isn’t nearly as messy as decoding.)

The klipcook format conversion utility can also use the same encoders, to write .flac, .oga, .ogg, and .mp3 files, in addition to uncompressed .wav and .raw.

For decoding (playing files), the following are supported (given that the respective codec APIs are installed:

• WAV (.wav), implemented in Klipspringer without any external codec API, and also by the standard Java audio system, which Klipspringer falls backs to in case it encounters some strange WAV variant it doesn’t recognize.
• FLAC (.flac), the lossless compressed audio that is my primary format of choice.
• Ogg FLAC (.oga), FLAC in an Ogg container.
• Vorbis (.ogg), also known as Ogg Vorbis because of its Ogg container, lossy compressed audio.
• Opus (.opus), newer versatile lossy compressed audio, also in an Ogg container.
• AIFF (.aiff or .aif), not implemented in Klipspringer code but supported by the Java audio system.

This is all available to the Klipspringer hub, as well as in the codec library to be used in other Java projects.

What about the problem with code lying around unused that stops working? I’m not too worried about it since the parts that deal with the individual codecs are very small, essentially just adapters, and most of the processing is done in common code that gets tested all the time. But admittedly, it is a risk.

Since Klipspringer version 4.3, I’ve made the ALSA module part of the codec library, to make it easier for Java projects in Linux to handle audio more efficiently. Check out the API documentation.