Perf Tools for Kernel Performance Profiling

Linux Kernel Debugging and Analysis · 4:44

Listen on 93

Lyrics

[Verse 1]
Deep inside the kernel where the hardware meets the code
There's a tool called perf that lights up every node
CPU cycles spinning while the cache begins to miss
Branch prediction stumbles and we need to diagnose this
Install the linux-tools package on your running machine
Then perf record minus a for the system-wide scene

[Chorus]
Perf stat shows the counters
Perf record captures all
Cache miss ratio climbing
Branch prediction's fall
Hardware performance counters
Tell the story deep inside
Kernel space profiling
With perf as our guide

[Verse 2]
Start with perf stat minus e for the events that you need
Instructions per cycle shows if your code's got the speed
L1 cache misses hitting when your data's not there
L2 and L3 follow with their latency to bear
Branch misprediction ratio tells you when the CPU guessed wrong
Use perf top for real-time view to see what takes too long

[Chorus]
Perf stat shows the counters
Perf record captures all
Cache miss ratio climbing
Branch prediction's fall
Hardware performance counters
Tell the story deep inside
Kernel space profiling
With perf as our guide

[Bridge]
Flamegraphs visualize where all your cycles go
Call stack sampling shows the hottest paths below
Annotate the assembly with perf annotate command
See which instructions cost the most to understand
KVM guests and containers need minus minus guest for the view
Kernel symbols require root access to see it all through

[Verse 3]
TLB misses happen when virtual memory translation stalls
Context switches costly when the scheduler makes its calls
PMU events vary depending on your hardware type
Intel and AMD have different counters for the hype
Export to FlameGraph dot pl for the visual display
SVG output shows you where performance goes astray

[Chorus]
Perf stat shows the counters
Perf record captures all
Cache miss ratio climbing
Branch prediction's fall
Hardware performance counters
Tell the story deep inside
Kernel space profiling
With perf as our guide

[Outro]
From user space to kernel space
Perf reveals it all
Hardware counters never lie
Performance stands or falls
When bottlenecks are hiding
In the silicon below
Perf tools light the pathway
To the answers you need to know

← Kernel Panic Analysis & Core Dumps | Memory Debugging with KASAN & SLUB Debug →