[Verse 1]
Forward mode sweeps through the computation graph
Dual numbers carry values and their slopes
Each operation propagates derivatives mathematically
While the primal trace unfolds what the function hopes
Griewank shows us how the chain rule multiplies
Through every node from input to the end
Automatic differentiation never lies
It computes gradients we can truly depend
[Chorus]
Forward flows from source to sink
Reverse mode travels back upstream
Computational graphs that link
Every partial in the scheme
AD makes the gradients flow
Backprop's just a special case you know
Tangent linear, adjoint mode
That's how the mathematics explode
[Verse 2]
Reverse accumulation starts from the output
Adjoint variables sweep backward through time
Each intermediate gets its bar computed
The transpose Jacobian falls into line
Baydin's survey maps the landscape clearly
Machine learning needs those gradients fast
Higher order derivatives cost us dearly
But second-order methods hold contrast
[Chorus]
Forward flows from source to sink
Reverse mode travels back upstream
Computational graphs that link
Every partial in the scheme
AD makes the gradients flow
Backprop's just a special case you know
Tangent linear, adjoint mode
That's how the mathematics explode
[Bridge]
Computational complexity tells the tale
Forward mode scales with parameters out
Reverse mode flips it, makes the math prevail
When outputs are few, there's never doubt
Cross-country elimination solves the puzzle
Vertex elimination cuts the work
Optimal sequences prevent the shuffle
Where computational bottlenecks lurk
[Verse 3]
Cortical columns process information streams
Distributed units compute in parallel ways
Each minicolumn realizes different schemes
Like forward and reverse in neural displays
The brain might use both modes simultaneously
Different pathways for prediction and error
Automatic differentiation fits seamlessly
Into biological computational terror
[Verse 4]
Source transformation builds the augmented code
Operator overloading makes it clean
While checkpointing helps when memory's slowed
Trading computation for storage routine
JAX and PyTorch implement these dreams
TensorFlow's eager execution flows
Differentiable programming redefines schemes
As gradient-based optimization grows
[Chorus]
Forward flows from source to sink
Reverse mode travels back upstream
Computational graphs that link
Every partial in the scheme
AD makes the gradients flow
Backprop's just a special case you know
Tangent linear, adjoint mode
That's how the mathematics explode
[Outro]
From Griewank's foundation to modern days
AD transforms how we optimize and learn
Neural networks benefit from these ways
As cortical mysteries slowly turn