[Verse 1] In ninety-nine Tishby found the key Information bottleneck theory Compress the input, keep what matters most Throw away the noise, preserve the ghost Of relevant data flowing through X to Y with T between the two Mutual information guides the way What to keep and what to throw away [Chorus] Bottleneck principle, squeeze it tight Keep the signal, lose the white noise Sparse code dancing in the light Natural images make their choice Deep learning layers understand Information in demand Cortical columns compute and store Less is more, less is more [Verse 2] Olshausen and Field in ninety-six Showed how simple cells do their tricks Gabor filters emerge from sparse code learning Natural statistics, wheels are turning Receptive fields that match the world Edge detectors, patterns unfurled Minimize the neurons that fire Efficiency that we admire [Chorus] Bottleneck principle, squeeze it tight Keep the signal, lose the white noise Sparse code dancing in the light Natural images make their choice Deep learning layers understand Information in demand Cortical columns compute and store Less is more, less is more [Verse 3] Twenty-fifteen brought deeper insight Zaslavsky and Tishby shed new light Neural networks learn to generalize By finding the optimal compromise Between fitting data perfectly And keeping representations free From irrelevant complexity Information geometry [Verse 4] Mumford and Geman showed the way Bayesian priors hold the sway Sparse representations in the brain Help us see through visual rain Dictionary learning algorithms find The basis functions that define How cortical neurons decompose The world into what nature knows [Bridge] Beta controls the tradeoff game Compression versus prediction's flame Lagrange multipliers set the stage For learning's computational age Cortical columns distributed wide Process information side by side Each unit holds a piece complete Of nature's mathematical feat [Chorus] Bottleneck principle, squeeze it tight Keep the signal, lose the white noise Sparse code dancing in the light Natural images make their choice Deep learning layers understand Information in demand Cortical columns compute and store Less is more, less is more [Outro] From retina to visual cortex flowing Information bottleneck always knowing What to keep and what to release Mathematical beauty never cease
← Dancing Through the Neural Gate | Whispers Through the Noise Gate →