[Verse 1]
In the cortical maze where neurons compute
There's a principle tight that we can't refute
Sufficient statistics hold the key
Extract what matters, set the rest free
Data flows downstream but information stays
Processing inequality shows us the ways
What passes through cannot exceed
The bottleneck upstream indeed
[Chorus]
Minimize the input, maximize the task
Information bottleneck, that's all we ask
Sparse and selective, the neural code sings
Compression and function, the balance it brings
M minimizes X but preserves the Y
Tishby's principle reaches the sky
[Verse 2]
In the columns of cortex, sparsity reigns
Only few neurons fire through cognitive trains
Selectivity sharp like a crystalline blade
Each cell responds to patterns custom-made
Population vectors dance in high dimensions
While metabolic costs demand interventions
Grandmother cells or distributed schemes
The debate continues in neuroscience dreams
[Chorus]
Minimize the input, maximize the task
Information bottleneck, that's all we ask
Sparse and selective, the neural code sings
Compression and function, the balance it brings
M minimizes X but preserves the Y
Tishby's principle reaches the sky
[Bridge]
Rate distortion theory maps the trade
Between compression and the price we've paid
Variational autoencoders learn this game
Attention mechanisms do just the same
The bottleneck layer squeezes tight
While reconstruction keeps the target bright
[Verse 3]
From Shannon's theorems to modern machines
The mathematics flows through silicon dreams
Gradient descent finds the optimal squeeze
Between complexity and what the task needs
Cortical columns like distributed arrays
Process the world in computational ways
Each minicolumn a processing node
In the brain's magnificent neural code
[Verse 4]
When plasticity reshapes the neural wire
Hebbian learning sets synapses on fire
What fires together will wire the same
In correlation's computational game
But information theory guides the way
Mutual information has its say
Only patterns that predict survive
The bottleneck keeps the code alive
[Final Chorus]
Minimize the input, maximize the task
Information bottleneck, that's all we ask
Sparse and selective, the neural code sings
Compression and function, the balance it brings
M minimizes X but preserves the Y
In cortical columns where computations fly
[Outro]
From Tishby's wisdom to VAE design
The bottleneck principle draws the line
Between the data and the goal we seek
Mathematical beauty, elegant and sleek