[Verse 1] In the cortical maze where neurons compute There's a principle tight that we can't refute Sufficient statistics hold the key Extract what matters, set the rest free Data flows downstream but information stays Processing inequality shows us the ways What passes through cannot exceed The bottleneck upstream indeed [Chorus] Minimize the input, maximize the task Information bottleneck, that's all we ask Sparse and selective, the neural code sings Compression and function, the balance it brings M minimizes X but preserves the Y Tishby's principle reaches the sky [Verse 2] In the columns of cortex, sparsity reigns Only few neurons fire through cognitive trains Selectivity sharp like a crystalline blade Each cell responds to patterns custom-made Population vectors dance in high dimensions While metabolic costs demand interventions Grandmother cells or distributed schemes The debate continues in neuroscience dreams [Chorus] Minimize the input, maximize the task Information bottleneck, that's all we ask Sparse and selective, the neural code sings Compression and function, the balance it brings M minimizes X but preserves the Y Tishby's principle reaches the sky [Bridge] Rate distortion theory maps the trade Between compression and the price we've paid Variational autoencoders learn this game Attention mechanisms do just the same The bottleneck layer squeezes tight While reconstruction keeps the target bright [Verse 3] From Shannon's theorems to modern machines The mathematics flows through silicon dreams Gradient descent finds the optimal squeeze Between complexity and what the task needs Cortical columns like distributed arrays Process the world in computational ways Each minicolumn a processing node In the brain's magnificent neural code [Verse 4] When plasticity reshapes the neural wire Hebbian learning sets synapses on fire What fires together will wire the same In correlation's computational game But information theory guides the way Mutual information has its say Only patterns that predict survive The bottleneck keeps the code alive [Final Chorus] Minimize the input, maximize the task Information bottleneck, that's all we ask Sparse and selective, the neural code sings Compression and function, the balance it brings M minimizes X but preserves the Y In cortical columns where computations fly [Outro] From Tishby's wisdom to VAE design The bottleneck principle draws the line Between the data and the goal we seek Mathematical beauty, elegant and sleek
← Convergent Evolution Burning Bright | Dancing Through the Neural Gate →