perf <command> [options]
sudo perf top
perf is the Linux performance profiler. It uses hardware counters and kernel events to
show where CPU time goes, which functions are hot, why a program is slow, and what the
kernel is doing. It can profile one program, one process, or the whole system, with
much lower overhead than strace or ltrace.
Install it with: sudo apt install linux-tools-common linux-tools-$(uname -r)
or: sudo dnf install perf
The perf version should match your running kernel. Most commands need root, or a relaxed
kernel.perf_event_paranoid setting (see the permissions section below).
The main commands
perf list List the events you can measure
perf stat Count events for a command (cycles, cache misses ...) and print totals
perf record Sample where time is spent and save it to perf.data
perf report Browse the recorded data
perf top Live view of the hottest functions (like top for code)
perf annotate Show source or assembly with hot spots marked
perf trace System call tracing with much lower overhead than strace
perf sched Scheduler analysis: latency, run queues
perf diff Compare two recordings
perf script Dump the raw recorded events (for flame graphs and scripts)
perf bench Built-in micro-benchmarks
perf probe Add dynamic tracepoints to the kernel or programs
perf mem / c2c Memory access and cache-line contention analysis
perf lock Lock contention analysis
Count events with perf stat
perf stat ls # run a command and print counters
perf stat -r 10 ./myapp # repeat 10 times, show averages and variance
perf stat -p 1234 sleep 10 # a running process, for 10 seconds
sudo perf stat -a sleep 5 # the whole system for 5 seconds
sudo perf stat -a -A sleep 5 # per-CPU numbers
perf stat -e cycles,instructions,cache-misses,branch-misses ./myapp
perf stat -d ./myapp # more detail (cache and TLB events)
perf stat -dd ./myapp # even more detail
perf stat -o stats.txt ./myapp # write to a file
perf stat -I 1000 -a sleep 10 # print counters every 1000 ms
perf stat --timeout 5000 -p 1234 # stop after 5 seconds
Sample output to read:
task-clock CPU time used (msec) and how many CPUs it kept busy
context-switches How often the scheduler moved it off the CPU
cpu-migrations Moves between CPUs
page-faults Memory pages touched for the first time
cycles, instructions CPU work done; the ratio is instructions per cycle (IPC)
insn per cycle Above 1 is decent for many workloads; well below 1 means stalls (often memory)
branch-misses Mispredicted branches (percent of all branches)
cache-misses Data not found in cache (a common cause of slow loops)
Find hot code with perf record
sudo perf record ./myapp # profile a command
sudo perf record -F 99 ./myapp # sample 99 times a second (default 4000)
sudo perf record -g ./myapp # also record call stacks (call graphs)
sudo perf record --call-graph dwarf ./myapp # better stacks for programs without frame pointers
sudo perf record -p 1234 -g -- sleep 30 # a running process for 30 seconds
sudo perf record -a -g -- sleep 30 # the whole system for 30 seconds
sudo perf record -e cycles:u ./myapp # user space only
sudo perf record -e cache-misses ./myapp # sample on a different event
sudo perf record -o my.data ./myapp # output file name
sudo perf record -F 99 -a -g -- sleep 60 # a good general-purpose system profile
perf writes perf.data in the current directory. Then look at it:
perf report # interactive browser
perf report --stdio # plain text
perf report --no-children # self time only, not callers
perf report --sort comm,dso,symbol # group by command, library, function
perf report -i my.data # a specific file
Keys in the report: arrow keys move, Enter zooms in on a function, a annotates it,
h help, q quit, / filter, E expand all call chains.
Live view with perf top
sudo perf top # hottest functions right now, system-wide
sudo perf top -p 1234 # one process
sudo perf top -g # with call graphs
sudo perf top -e cache-misses # another event
sudo perf top -F 500 # sampling rate
sudo perf top --sort comm,dso # group by process and library
Look at the Overhead column and the Symbol. [k] marks kernel functions, [.] user functions.
Annotate: which lines are slow?
sudo perf annotate # from perf.data, choose a symbol
sudo perf annotate --stdio -s main # one symbol, as text
Build with -g (debug info) to see source lines. Without it you see assembly only.
Trace system calls with perf trace
sudo perf trace ls # like strace, far lower overhead
sudo perf trace -p 1234 # a running process
sudo perf trace -s ls # summary: counts and time per syscall
sudo perf trace -e openat,read ls # only these calls
sudo perf trace --duration 10 -p 1234 # only calls slower than 10 ms
sudo perf trace -a sleep 5 # the whole system
Trace scheduler behavior
sudo perf sched record -- sleep 10 # record scheduler events
sudo perf sched latency # per-task scheduling delay
sudo perf sched map # which tasks ran on which CPU
sudo perf sched timehist # timeline of every wakeup and run
Good for "why does my process wait so long to run?" questions.
Events
perf list # everything available
perf list hw # hardware events (cycles, instructions ...)
perf list sw # software events (page-faults, context-switches)
perf list cache # cache events
perf list tracepoint # kernel tracepoints (sched:*, syscalls:*, block:*)
perf list | grep -i cache # search
Event modifiers: :u user only, :k kernel only, :p precise (PEBS/IBS where supported)
Examples: cycles:u instructions:k cache-misses:pp
Names vary by CPU. Inside VMs most hardware events are missing: only software events work.
Trace points and dynamic probes
sudo perf record -e sched:sched_switch -a -- sleep 5 # every context switch
sudo perf record -e block:block_rq_issue -a -- sleep 10 # disk I/O requests
sudo perf record -e 'syscalls:sys_enter_openat' -a -- sleep 5 # a specific syscall
sudo perf probe --add tcp_sendmsg # dynamic kernel probe
sudo perf probe --list
sudo perf probe --del tcp_sendmsg
sudo perf script # print the recorded events
Use tracepoints when you want exact event counts, not statistical samples.
Flame graphs (a popular way to see stacks)
sudo perf record -F 99 -a -g -- sleep 30
sudo perf script > out.perf
git clone https://github.com/brendangregg/FlameGraph
./FlameGraph/stackcollapse-perf.pl out.perf | ./FlameGraph/flamegraph.pl > flame.svg
Open flame.svg in a browser. Wide boxes are functions where the time goes. Read from the bottom (callers) up.
For stacks to look right, programs should keep frame pointers, or use --call-graph dwarf.
Compare two runs
perf record -o before.data ./myapp
perf record -o after.data ./myapp
perf diff before.data after.data # what got faster or slower
Practical one-liners
# Is this program CPU-bound? (look at task-clock, IPC and page-faults)
perf stat ./myapp
# Where is my program spending its time?
sudo perf record -F 99 -g ./myapp && sudo perf report --stdio | head -50
# What is my whole server busy doing right now?
sudo perf top
# What is a slow, busy process doing?
sudo perf top -p $(pgrep -o myservice)
# Profile a running service for 30 seconds
sudo perf record -F 99 -p $(pgrep -o nginx) -g -- sleep 30
sudo perf report --stdio
# Which system calls does it spend time in?
sudo perf trace -s -p 1234 -- sleep 10
# Which processes cause the most context switches?
sudo perf stat -e context-switches -a -A sleep 5
sudo perf record -e context-switches -a -- sleep 5 && sudo perf report --sort comm
# Are cache misses a problem?
perf stat -e cache-references,cache-misses,instructions,cycles ./myapp
# Find who is causing disk I/O
sudo perf record -e block:block_rq_issue -a -g -- sleep 10
sudo perf report --sort comm,dev
# Count page faults for a program
perf stat -e page-faults,minor-faults,major-faults ./myapp
# Check a quick benchmark of the scheduler
perf bench sched pipe
# Kernel-only profile of the whole system
sudo perf record -a -g -e cycles:k -- sleep 10
Permissions and safety
cat /proc/sys/kernel/perf_event_paranoid # current level
sudo sysctl -w kernel.perf_event_paranoid=1 # allow user-level profiling of your own processes
sudo sysctl -w kernel.kptr_restrict=0 # show kernel symbol addresses
-1 no restrictions 0 allow most 1 user + kernel for your processes 2+ user space only
Lowering these settings exposes kernel information to local users. Change them only on machines
you control, and put them back afterwards. On shared or production systems, use sudo instead.
Recording a whole system can create large perf.data files. Use a short -- sleep N and check the disk.
perf vs other tools
top / htop Which processes use CPU (whole-process view)
perf top Which FUNCTIONS use CPU, including kernel code
strace/ltrace Calls with arguments, heavy overhead
perf trace System calls with far lower overhead
valgrind Memory errors and leaks, very slow
gprof Instrumented profiling (needs recompiling with -pg)
bpftrace / bcc / eBPF tools Custom, production-safe tracing and statistics
sar / vmstat / iostat / mpstat System-wide counters over time
Common problems
"perf not found" or wrong version Install linux-tools for your exact kernel: uname -r
"Permission error" / "perf_event_paranoid" Run with sudo, or lower perf_event_paranoid
"Not supported" / "<not supported>" events Virtual machines and containers often hide hardware counters
No kernel symbols Run as root, or set kptr_restrict=0
Report shows raw addresses or [unknown] The binary has no symbols. Build with -g, avoid stripping
Call graphs look wrong or short Add --call-graph dwarf, or build with -fno-omit-frame-pointer
perf.data is huge Lower -F, shorten the run, or drop -g
Results change each run Sampling is statistical: use longer runs or perf stat -r 10
Containers Needs SYS_ADMIN or a perf-enabled seccomp profile (--cap-add SYS_ADMIN)
Tips
Start with perf stat to see the big picture: CPU-bound, memory-bound, or waiting.
Use perf top for a quick live look, then perf record -g and perf report to dig in.
Sample around 99 Hz (-F 99) for long recordings. It is light and avoids lock-step with 100 Hz timers.
Add -g so you can see who called the hot function, not just the function.
Build your own code with -g and -fno-omit-frame-pointer so reports show real function names and lines.
Read a flame graph from the bottom up: the widest boxes are where the time goes.
Watch instructions per cycle (IPC). A low value usually means the CPU is waiting on memory.
On virtual machines expect fewer hardware events. Software events and tracepoints still work.
Keep the perf tool version matched to the running kernel, and delete large perf.data files when done.
Profile only systems you are allowed to instrument, and revisit any sysctl settings you relaxed.
