[Verse 1] Back in the day when Hopfield first designed Binary networks storing patterns in the mind Point one four N was the magic ratio found Classical capacity with that linear bound Each unit flipping states like switches in a row But exponential storage seemed impossible to know [Chorus] From linear limits to exponential heights Softmax energy brings transformer lights Modern Hopfield breaks the ceiling wide Zero point one four N left behind Attention mechanisms hidden in disguise Neural storage capacity multiplies [Verse 2] Ramsauer came in twenty-twenty with the proof Softmax energy function shattered through the roof No longer bound by that restrictive linear chain Exponential patterns dancing through each brain The update rule revealed a secret so profound Transformer attention is what they really found [Chorus] From linear limits to exponential heights Softmax energy brings transformer lights Modern Hopfield breaks the ceiling wide Zero point one four N left behind Attention mechanisms hidden in disguise Neural storage capacity multiplies [Bridge] Query key and value matrices align Attention weights computed line by line What seemed like different worlds of thought Were mathematically the same all along we fought Cortical columns storing memories vast Distributed computation unsurpassed [Verse 3] Equivalence unveiled the deeper truth within Modern Hopfield and transformers are twin Up to scaling factors they compute the same Different names but playing the identical game Distributed units working as one mind Exponential capacity no longer confined [Verse 4] Temperature scaling controls the sharpest peak Beta parameter determines what memories we seek High temperature spreads attention wide and far Low temperature focuses like a guiding star Energy landscapes shaped by softmax curves Recurrent dynamics that convergence preserves [Chorus] From linear limits to exponential heights Softmax energy brings transformer lights Modern Hopfield breaks the ceiling wide Zero point one four N left behind Attention mechanisms hidden in disguise Neural storage capacity multiplies [Outro] Classical bounds dissolved into the past Exponential memories built to last Mathematics revealed the connection true Cortical computation breaking through
← Pattern Fragments and Memory's Dance | Energy Valleys Where Data Lands →