Ten standard ways of representing binary numbers

To convert a signed b-bit number to unsigned or vice versa, you add or subtract \(2^{b-1}\), or equivalently flip the sign bit with an xor operation. So to change x from signed to unsigned or the reverse, just do:

x ^= 1 << b-1

Beware that in C, these kinds of operations should always be done with values declared to be unsigned, because it’s undefined what happens when a signed integer overflows beyond its range.

CPUs often have instructions to reverse endianness, which in C on most Unix-like systems can be accessed through macros found in endian.h. For instance, be32toh converts (if necessary) a big-endian 32-bit integer to the host representation and htobe32 the reverse. In macOS, the corresponding macros have names like OSSwapBigToHostInt32, and can be found in architecture/byte_order.h.

Integers can be composed from bytes using shift and or operations. For instance the following reads a 24-bit value from position i in a byte array buf:

buf[i] << d0 | buf[i+1] << 8 | buf[i+2] << d2

If buf is little endian, d0 should be 0 and d2 16, and the reverse if buf is big endian.

In Java, ByteBuffer has an order method to specify the endianness of the underlying array, which guides methods like getInt and putInt. If you look into the implementation, you’ll find that native methods in jdk.internal.misc.Unsafe do the actual memory access. But there are no methods for 24-bit (3-byte) numbers, so they have to be composed with shift and or.

I think the most sensible choice is to place the significant bits in the most significant digit (MSD) positions, and set the padding bits in the least significant positions to zero. The sign bit (if the representation is signed) ends up where it’s supposed to be, and if data is sent to a recipient that can’t handle padding, it works to just lie by giving the number of bits after padding. Then to the recipient, it looks like the lowest bits just happen to always be zero.

The LAME API expects samples to be delivered this way. Also, it may work in WAV files, depending on what software reads them. WAV is a Microsoft format, and its specifications can make you think of the old joke about how many Microsoft employees it takes to change a light bulb. (None, they just define darkness to be the new de facto standard.) There is no requirement that the bit depth should be divisible by 8, but the assumption seems mostly to be that it is, except in this section on page 3-25 of the joint IBM and Microsoft specification version 1.0 from 1991:

Each sample is contained in an integer i. The size of i is the smallest number of bytes required to contain the specified sample size. The least significant byte is stored first. The bits that represent the sample amplitude are stored in the most significant bits of i, and the remaining bits are set to zero.

For example, if the sample size (recorded in nBitsPerSample) is 12 bits, then each sample is stored in a two-byte integer. The least significant four bits of the first (least significant) byte is set to zero.

However, the Microsoft “standards update” document from 1994, versioned as 3.0, makes no mention of this. Generally, you can’t expect that software can handle padding in WAV files, but MPlayer can, as well as Klipspringer. Later, Microsoft has defined an extensible format that requires the number of bits per samples to be a multiple of 8, but then adds a specification for the actual “valid” number of bits. The Klipspringer WAV decoding class supports the extension since version 4.0.7, but klipcook still produces WAV files without it. (Maybe I’ll change it some day, but for now the previous standard works for me.)

To move the a b-bit bumber from LSD to MSD positions in a variable x with a word size of w bits, just shift it w​−​b bits to the left (and again in C, make sure x is unsigned).

x <<= w-b;

Positioning the significant bits in the least significant digit (LSD) positions, with unused more significant bits set to zero, seems to be the standard with the ALSA API for instance.

The ALSA header file pcm.h has:

/** PCM sample subformat */
typedef enum _snd_pcm_subformat {
    /** Unknown */
    SND_PCM_SUBFORMAT_UNKNOWN = -1,
    /** Standard */
    SND_PCM_SUBFORMAT_STD = 0,
    /** Maximum bits based on PCM format */
    SND_PCM_SUBFORMAT_MSBITS_MAX = 1,
    /** 20 most significant bits */
    SND_PCM_SUBFORMAT_MSBITS_20 = 2,
    /** 24 most significant bits */
    SND_PCM_SUBFORMAT_MSBITS_24 = 3,
    SND_PCM_SUBFORMAT_LAST = SND_PCM_SUBFORMAT_MSBITS_24
} snd_pcm_subformat_t;

That indicates that the standard, “STD” is not the most significant bits, because the MSBITS constants serve as the exception to prove the rule. I haven’t seen it clearly specified anywhere, but both ALSA and the Java sound system seem to default to zero-padded LSD format, and therefore raw data files recorded by Klipspringer also have this format.

Usually nothing needs to be done to get this format, but obviously you need to shift right if going from MSD. Sometimes you may need to mask off the top w​−​b bits from a b-bit number in a variable with word size w, for instance like this:

x &= (1 << w-b)-1 << b;

If samples captured in LSD with zero padding are directly sent as input to the libFLAC encoder, the process crashes, apparently due to an unchecked out-of-range error. Surprising, but not altogether unreasonable. Signed b-bit numbers can have values from \(-2^{b-1}\) to \(+2^{b-1}-1\) and if negative numbers are padded with more significant 0s they go out of range when interpreted as having more than b bits. For instance, 12-bit signed numbers range from −2048 to +2047. As you can see in the zero-padded LSD illustration, −1397 viewed as a 12-bit binary number padded with four 0s makes the 16-bit pattern 0000101010001011. But interpreted as a 16-bit number, that bit pattern is 2699, which is out of range. In order to keep its value as a 16-bit number, it needs to be padded with four 1s instead. As mentioned above, it makes sense to regard negative numbers as padded with an infinite number of 1s to the left.

To pad a b-bit zero-padded LSD number with 1s if it’s negative, you can extend the sign bit like this:

x |= -(x & 1 << b-1);