Speech from first principles · Part 2
Running the tube backwards
The acoustic model of a speech recogniser, derived as the inverse of the one that makes the sound.
Part 1 built a vowel out of a buzz and a tube, and ended at a factorisation:
\[P_{\text{out}}(\omega) = U_g(\omega)\,V(\omega)\,R(\omega)\]Source times filter times radiation. The vowel lives in \(V\) — the shape of the tube. The pitch, the voice quality, whether the speaker has a cold, all of that lives in \(U_g\).
A recogniser gets \(P_{\text{out}}\) and needs the words. So its problem, stated in one line, is: recover \(V\) and throw \(U_g\) away. That’s the entire design brief for a speech front end, and nearly every strange-looking decision in one — the 25 ms window, the logarithm, the triangular filters, the discrete cosine transform nobody uses any more — is a consequence of it.
Everything analysed below is synthesised by the tract model from part 1, which means the true answer is always known.
Why you have to chop it up first
The factorisation holds only while the tract is standing still. Speech isn’t stationary — that’s the point of it — so before anything else, cut the signal into frames short enough that the tongue hasn’t moved.
That immediately collides with the uncertainty principle of the Fourier transform. A window of length \(T\) resolves frequencies no finer than about \(1/T\):
- Too short (5 ms → 200 Hz resolution): individual harmonics, spaced 100–200 Hz apart, blur into each other. What survives is the envelope. The formants are visible; the pitch is gone.
- Too long (40 ms → 25 Hz): the harmonic comb is beautifully resolved, and the tongue has moved during the window, so the formants smear.
Figure 1 · The window
The standard settings — a 25 ms window, stepped every 10 ms — sit deliberately in the middle. 25 ms is two to three pitch periods for a typical adult voice, long enough that a low-pitched speaker’s harmonics don’t dominate and short enough that the tract is nearly frozen. The 10 ms hop overlaps the windows so a transient can’t fall between two frames.
Those numbers have survived every architectural revolution in the field since the 1970s.
The cepstrum: undoing the multiplication exactly
Now the central move. We have a product and we want one of its factors. Take logs:
\[\log|P(\omega)| = \log|U_g(\omega)| + \log|V(\omega)| + \log|R(\omega)|\]The product became a sum. That’s already worth something, but the real trick is what these two terms look like as functions of \(\omega\):
-
$$\log U_g $$ is the harmonic comb — a rapidly oscillating ripple, wiggling up and down every f₀ hertz. -
$$\log V $$ is the formant envelope — slow, smooth, a few broad humps across 5 kHz.
They’re separated by rate of variation along the frequency axis. And the tool that separates a signal by rate of variation is a Fourier transform. So take the Fourier transform of the log spectrum — a spectrum of a spectrum. Its independent variable has units of time but indexes structure in frequency, so Bogert, Healy and Tukey in 1963 named it quefrency, and named the whole thing the cepstrum, by spelling “spectrum” backwards. The field kept the joke.
\[c[n] = \mathcal{F}^{-1}\left\{ \log|P(\omega)| \right\}\]And the separation is clean:
- The comb ripples at a rate of one cycle per f₀ hertz, so it lands at quefrency \(1/f_0\) — a sharp spike at 8.3 ms for a 120 Hz voice.
- The envelope varies slowly, so it lands at low quefrency, in the first millisecond or two.
Cut the cepstrum in half — “liftering”, because the field committed to the spelling gag — and you have separated the tube from the buzz.
Figure 2 · The cepstrum
Pitch detection falls out for free. The quefrency of that spike is the pitch period. Measured on the synthesised vowels, it recovers 100, 120 and 200 Hz exactly. Cepstral pitch tracking is still a competitive method sixty years later.
Formant frequencies do not fall out for free. You can see the envelope; reading reliable numbers off it is a different and much nastier problem. For /u/, where F1 and F2 are close together on a steep spectral rolloff, naive peak-picking on that orange curve is wrong more often than right. This is why, despite formants being the physically meaningful quantity, no production speech recogniser has ever extracted them. The envelope is kept as an envelope — a few dozen numbers describing a shape — and the classifier is left to work out what the shape means.
That decision, made for engineering reasons in the 1970s, is the reason speech recognition features look like a smoothed spectrum rather than a description of a vocal tract.
The mel filterbank: the same trick, done cheaper
The cepstrum separates source from filter by transforming twice. There’s a blunter route to the same place: just smooth the spectrum. Average energy over bands wide enough to swallow several harmonics and the comb disappears, leaving the envelope.
That raises the question of how wide the bands should be, and here the answer comes from the ear rather than the tube. Human frequency resolution is roughly logarithmic above about 500 Hz: we distinguish 500 from 600 Hz easily and 5000 from 5100 Hz not at all. The mel scale is one fit to that behaviour,
\[m(f) = 2595 \log_{10}\!\left(1 + \frac{f}{700}\right)\]and the standard front end lays out triangular filters at equal spacing on it. Near-linear low down, stretching out above 1 kHz.
Figure 3 · The mel filterbank
Then take the log of each band energy, which does three jobs at once: it matches the roughly logarithmic perception of loudness, it turns the remaining multiplicative terms into additive ones, and it compresses a dynamic range of many orders of magnitude into something a classifier can handle.
The DCT, and why it left
Classically there was one more step. Take the discrete cosine transform of the log-mel energies and keep the first 13 coefficients: MFCCs, the most-cited feature in the history of speech.
The DCT is there for a reason that has nothing to do with speech. Adjacent mel bands are highly correlated — a formant lights up several neighbours at once. The acoustic models of the day were Gaussian mixtures with diagonal covariance matrices, chosen because full covariances were computationally hopeless. A diagonal Gaussian assumes the features are uncorrelated, so feeding it correlated ones badly misfits. The DCT approximately decorrelates them and the assumption becomes tolerable.
So MFCCs are not a better description of speech than log-mel. They’re log-mel bent into a shape that suits a specific classifier from 1980.
A neural network has no diagonal-covariance assumption, models correlated inputs happily, and would rather have the extra information the truncation threw away. Modern systems feed log-mel filterbank energies — typically 80 of them — straight in. The DCT was dropped when the reason for it was dropped, and it took the field about a decade to notice.
The other half of the problem: nobody labelled the frames
Now there’s a second, independent difficulty, and it has nothing to do with acoustics.
The front end produces 100 feature vectors a second. The transcript says “cat”. Nobody has said which frames are the /k/. Training a frame classifier needs frame labels, and the labels don’t exist — the alignment between audio and text is latent.
The classical answer is a hidden Markov model. Each phone is a small left-to-right chain of states; each state can loop on itself (making the phone longer) or advance. Given per-frame emission probabilities, the Viterbi algorithm finds the single most likely path through the trellis, and that path is an alignment.
Figure 4 · Alignment as a path
Training then alternates: align with the current model, retrain on the alignment, realign. That loop — the forward–backward algorithm doing it properly, in expectation over all paths rather than just the best one — is how every recogniser was built for thirty years.
CTC: summing over every alignment
The HMM approach needs a pronunciation dictionary, phone states, and a lot of scaffolding. Connectionist temporal classification throws all of it out and keeps the one idea that mattered: if the alignment is unknown, sum over all of them.
CTC adds a blank symbol meaning “no output here”, and defines a collapsing rule: merge repeats, then delete blanks. So for four frames, caat, -cat, ccat and ca-t all collapse to “cat”. Training maximises the total probability of every frame sequence that collapses to the transcript — computed in one dynamic-programming pass, the same shape of recursion as the forward algorithm above.
Figure 5 · Every path to the same word
The blank is what makes the collapsing rule invertible enough to be useful. Without it, “cat” and “caat” would be indistinguishable, and a word with a genuine double letter could never be emitted.
What replaced the front end, and what didn’t
Modern encoders — wav2vec 2.0, Whisper and the rest — appear to have swept this away. Whisper still eats log-mel; wav2vec 2.0 goes further and runs a convolutional stack straight over the raw waveform, learning its own filters from data.
But look at what those learned filters converge to. Train a network on raw audio and its first layer reliably ends up with a bank of bandpass filters, more densely packed at low frequencies than high. It rediscovers a filterbank, roughly mel-shaped, because that is what the data is shaped like. What the learned version gets is the freedom to be wrong where mel is wrong — to put resolution where the task needs it rather than where a 1937 psychoacoustics experiment said the ear has it.
The pieces that survived every rewrite are the ones that came from the physics rather than the engineering:
| Choice | Where it comes from | Still there? |
|---|---|---|
| ~25 ms frames | the tract is stationary that long | yes, everywhere |
| Log-magnitude | products become sums; perception is logarithmic | yes |
| Mel-like warping | auditory resolution | yes, learned or fixed |
| Discarding phase | the filter’s magnitude carries the vowel | mostly |
| DCT / MFCC | diagonal-covariance Gaussians | gone |
| Explicit formants | — | never adopted |
| Latent alignment | nobody labels frames | yes — CTC, transducers, attention |
Where this becomes a person
Part 1 derived that formants scale as \(c/4L\). An adult male tract is about 17.5 cm; a child’s is far shorter. So every formant a child produces sits higher than an adult’s, by the ratio of the lengths — not shifted, scaled.
That is measurable. Taking Peterson and Barney’s 1952 recordings and comparing 15 children against 33 men, averaged over ten vowels, the children’s formants are higher by a factor of 1.34 for F1, 1.35 for F2 and 1.38 for F3 — a consistent multiplicative shift, exactly as a length change predicts, and implying a child vocal tract of about 13 cm.
The consequence for a recogniser is brutal. Children’s /i/ has an F2 between 2730 and 3610 Hz. Men’s /i/ has an F2 between 2000 and 2700 Hz. The ranges barely touch. A model that has only ever heard adults has learned a decision boundary in the wrong place, and a child’s perfectly well-formed vowel lands outside every region the model knows.
The classical fix follows straight from the physics: if the difference is a scaling, undo the scaling. Vocal tract length normalisation warps the frequency axis of a speaker’s features by a factor \(\alpha\) chosen to make them look like the training population.
Figure 6 · One number that fixes a lot
I find that figure genuinely satisfying: a single number, fitted to nothing but formant measurements, lands on the physical length of a child’s vocal tract, because part 1 says it must.
My MSc thesis was about a child with a congenital speech disorder, and VTLN does not fix that. A shorter tract is a uniform transformation, and uniform transformations are the easy case — that’s the whole reason one scalar works. A cleft palate or an Apert-syndrome craniofacial structure changes the tract’s shape, not just its scale: it couples the nasal cavity in permanently, moves constrictions to places the vowel space doesn’t have, and introduces zeros that an all-pole model cannot represent even in principle. There is no α for that.
The model in part 1 is good enough that fifty years of speech technology was built on it, and good enough that one scalar recovers a child’s anatomy from a scatter plot. It is also a model of a smooth, unbranched, undamaged tube, and the people most in need of speech technology that works are frequently the people whose tracts are not that. When a foundation model fails on non-normative speech, the training distribution gets the blame, and it deserves a lot of it. But some of the failure is older than the data: it’s built into the shape of the representation, and it has been there since we decided a voice was a buzz in a pipe.
Where to go from here
If you want to go deeper, the two books are Fant’s Acoustic Theory of Speech Production (1960), which is where the tube models come from, and Rabiner & Schafer’s Digital Processing of Speech Signals for the DSP. Stevens’ Acoustic Phonetics is the modern reference and much harder going.
The code behind every figure in both posts is lab.js, with a test that checks the acoustics against an independent NumPy implementation. If you want to poke at it, the most instructive thing to do is set every reflection coefficient to zero and watch the vowels disappear.