[Verse 1] Two neural architectures on my bench tonight Same dataset flowing through their hidden layers Linear probe connecting what they learned inside Testing if one network predicts the other's flavors Architecture alpha builds its feature space While beta constructs representations differently Can I map between them with a linear trace? Or do they encode knowledge independently? [Chorus] Linear probing, CKA measuring Representational similarity Networks learning, patterns turning But can they speak the same language? Alignment searching, kernels working Centered correlation's the key Some representations can't be bridged That's the mathematics we need to see [Verse 2] Centered Kernel Alignment takes the stage Computing similarities across the divide Normalize the features, center every page Then calculate how representations coincide Gram matrices dancing in the kernel space Frobenius inner products tell the tale High CKA means networks share their grace Low scores reveal where alignment starts to fail [Chorus] Linear probing, CKA measuring Representational similarity Networks learning, patterns turning But can they speak the same language? Alignment searching, kernels working Centered correlation's the key Some representations can't be bridged That's the mathematics we need to see [Bridge] Now construct two networks that solve the same task But their representations cannot be aligned What conditions make this paradox last? Different dimensional manifolds intertwined Nonlinear transformations twist the space Orthogonal subspaces living separate lives When linear maps cannot find their place That's when unalignment truly thrives [Verse 3] Provably unalignable, what must be true? The feature spaces live in different worlds Maybe one network learned a rotated view While the other's representations are unfurled Task performance identical on the surface But internal geometry tells another story Linear probes will fail to bridge the purpose Different paths can lead to the same glory [Verse 4] Random initialization sets the course Stochastic gradients carve their unique paths Each network follows its own driving force Creating representations that never match Weight decay and dropout add their noise Different architectures bend the solution space Every hyperparameter makes its choice Leading networks to their separate place [Chorus] Linear probing, CKA measuring Representational similarity Networks learning, patterns turning But can they speak the same language? Alignment searching, kernels working Centered correlation's the key Some representations can't be bridged That's the mathematics we need to see [Outro] Train your networks, test the bridges Mathematics reveals the hidden truth Some connections live on distant ridges That's computational learning proof
← Platonic Shapes in Hidden Storms | Convergent Evolution Burning Bright →