[Verse 1] MNIST digits on the screen, twenty-eight by twenty-eight Forward weights matrix W learns to classify their fate But backward signals use a random matrix B instead Can these misaligned pathways train a neural network's head? [Chorus] Track the alignment, watch it grow W and B start to show Feedback signals find their way Even when they're led astray Cortical columns compute and learn Even when the signals turn Random pathways still can teach Neural harmony within reach [Verse 2] Start with shallow, two or three, layers stacked up neat and clean Feedback alignment works just fine, performance sharp and keen But push it deeper, five or six, seven layers high Watch the random feedback fail as accuracy waves goodbye [Chorus] Track the alignment, watch it grow W and B start to show Feedback signals find their way Even when they're led astray Cortical columns compute and learn Even when the signals turn Random pathways still can teach Neural harmony within reach [Bridge] At what depth does backprop win the race? Test it layer by layer, find that breaking place Plot the curves and trace the lines Where feedback alignment declines [Verse 3] Sign-symmetric holds the key, positive and negative dance Forward weights go left, feedback mirrors that stance Same direction, opposite sign, a structured random play Closing gaps between the truth and feedback's roundabout way [Verse 4] Measure how the gradients flow through tangled pathways deep Does the sign constraint rescue what random feedback couldn't keep? Plot the learning curves and see how much the gap gets small When structure meets the chaos in cortical protocol [Verse 5] Initialize the weights with care, small and centered tight Watch the early epochs where alignments catch the light Layer by layer, epoch by epoch, the dynamics unfold Random matrices learning tricks that never were foretold [Bridge 2] Direct feedback, indirect paths Calculate the neural math Training time and convergence rate Which algorithm seals the fate? [Chorus] Track the alignment, watch it grow W and B start to show Feedback signals find their way Even when they're led astray Cortical columns compute and learn Even when the signals turn Random pathways still can teach Neural harmony within reach [Outro] From shallow nets to towers tall Test the limits, test them all Sign-symmetric bridges built Between the chaos and the gilt
← Messy Wires Still Learn | Breaking the Weight-Sharing Chain →