[Verse 1]
In ninety-nine Tishby found the key
Information bottleneck theory
Compress the input, keep what matters most
Throw away the noise, preserve the ghost
Of relevant data flowing through
X to Y with T between the two
Mutual information guides the way
What to keep and what to throw away
[Chorus]
Bottleneck principle, squeeze it tight
Keep the signal, lose the white noise
Sparse code dancing in the light
Natural images make their choice
Deep learning layers understand
Information in demand
Cortical columns compute and store
Less is more, less is more
[Verse 2]
Olshausen and Field in ninety-six
Showed how simple cells do their tricks
Gabor filters emerge from sparse code learning
Natural statistics, wheels are turning
Receptive fields that match the world
Edge detectors, patterns unfurled
Minimize the neurons that fire
Efficiency that we admire
[Chorus]
Bottleneck principle, squeeze it tight
Keep the signal, lose the white noise
Sparse code dancing in the light
Natural images make their choice
Deep learning layers understand
Information in demand
Cortical columns compute and store
Less is more, less is more
[Verse 3]
Twenty-fifteen brought deeper insight
Zaslavsky and Tishby shed new light
Neural networks learn to generalize
By finding the optimal compromise
Between fitting data perfectly
And keeping representations free
From irrelevant complexity
Information geometry
[Verse 4]
Mumford and Geman showed the way
Bayesian priors hold the sway
Sparse representations in the brain
Help us see through visual rain
Dictionary learning algorithms find
The basis functions that define
How cortical neurons decompose
The world into what nature knows
[Bridge]
Beta controls the tradeoff game
Compression versus prediction's flame
Lagrange multipliers set the stage
For learning's computational age
Cortical columns distributed wide
Process information side by side
Each unit holds a piece complete
Of nature's mathematical feat
[Chorus]
Bottleneck principle, squeeze it tight
Keep the signal, lose the white noise
Sparse code dancing in the light
Natural images make their choice
Deep learning layers understand
Information in demand
Cortical columns compute and store
Less is more, less is more
[Outro]
From retina to visual cortex flowing
Information bottleneck always knowing
What to keep and what to release
Mathematical beauty never cease