Perf

perf

Reference: cnblogs post

Prerequisites: Permissions

By default, many distros set perf_event_paranoid=4, which blocks hardware counter access for unprivileged users. Check and fix:

1
2
3
4
5
6
7
cat /proc/sys/kernel/perf_event_paranoid   # 4 = very restricted, -1 = fully open

# Temporarily lower it (until next reboot):
sudo sysctl -w kernel.perf_event_paranoid=0

# Make it permanent:
echo 'kernel.perf_event_paranoid=0' | sudo tee -a /etc/sysctl.conf

Basic Workflow

1. Compile with -g to embed debug symbols. This increases binary size but does not affect runtime performance.

2. Record and convert the trace:

   1
    perf record -F 99 -a -g -- MY_EXE && perf script -i perf.data &> perf.unfold
   

3. Visualize with one of:

   • FlameGraph

     1
     2
     3
       git clone https://github.com/brendangregg/FlameGraph.git
       ./FlameGraph/stackcollapse-perf.pl perf.unfold &> perf.folded
       ./FlameGraph/flamegraph.pl perf.folded > perf.svg
     

   • Speedscope (recommended) — drag perf.unfold onto the page.

Perf in a Docker Container

Running perf inside Docker has a few quirks, especially with custom kernels (e.g., System76 / Pop!_OS).

1. Missing kernel-matched perf binary

If you see:

1
2
3
WARNING: perf not found for kernel 6.9.3-76060903
  You may need to install the following packages for this specific kernel:
    linux-tools-6.9.3-76060903-generic

Custom kernel builds (System76, etc.) often have no matching linux-tools-<version> package in the default Ubuntu repos, so apt install linux-tools-$(uname -r) will fail inside the container.

Fix: Perf’s ABI is very stable (perf_event_open(2) is backward/forward compatible), so copy the host’s perf binary directly into the container:

1
cp /usr/lib/linux-tools-$(uname -r)/perf ./perf-copy

Do not copy /usr/bin/perf — it is a shell wrapper, not the real binary.

2. Required docker-compose settings

1
2
privileged: true      # or at least cap_add: [SYS_ADMIN, PERFMON]
pid: "host"

• privileged / SYS_ADMIN / PERFMON are required for perf_event_open.
• pid: "host" lets perf trace real host PIDs and resolve symbols correctly.

3. Lowering the paranoid level

If perf complains about permissions, run inside the container:

1
2
sudo sysctl -w kernel.perf_event_paranoid=0
sudo sysctl --system

Profiling

Timing and Resource Metrics

Clock Types

1
2
3
4
5
6
using Clock = std::chrono::steady_clock;
auto t0 = Clock::now();
// do work
std::this_thread::sleep_for(std::chrono::milliseconds(100));

auto t1 = Clock::now();

your timer will report about 100 ms, even though CPU time is almost zero. For actual CPU time, you would use things like: std::clock()

• std::chrono::steady_clock : monotonically increasing clock, doesn’t change if system clock changes
• std::chrono::high_resolution_clock: walltime
• clock_gettime(CLOCK_MONOTONIC, ...): wall time
• clock_gettime(CLOCK_PROCESS_CPUTIME_ID, ...) cpu timee of the whole process

Is CPU time always shorter than wall time?

No. For a single-threaded program, CPU time is usually: CPU time <= wall time, But for a multi-threaded program, process CPU time can be greater than wall time, because CPU time is summed across cores.

CPU Performance Counter Metrics

These require hardware perf counters to be enabled (may show “n/a” otherwise).

Collect all at once with:

1
2
3
perf stat -r 5 \
  -e task-clock,cycles,instructions,branches,branch-misses,cache-references,cache-misses \
  -- ./your_benchmark_binary --your-flags

Hands-on Example

The program below exercises all the metrics above in one binary:

• Sequential sum — cache-friendly, high IPC, low cache-miss rate baseline.
• Random scatter — cache-unfriendly, drives up LLC miss rate.
• Unpredictable branches — values are random so the branch predictor is ~50 % wrong.
• Large allocation — 256 MB array → visible Peak RSS.
• std::chrono timer — prints elapsed time from inside the program.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
// perf_demo.cpp
#include <algorithm>
#include <chrono>
#include <cstdlib>
#include <iostream>
#include <numeric>
#include <random>
#include <vector>

static const size_t N = 64 * 1024 * 1024;   // 64 M elements = 256 MB (int)

// Cache-friendly sequential sum
long long sequential_sum(const std::vector<int>& v) {
    long long s = 0;
    for (size_t i = 0; i < v.size(); ++i) s += v[i];
    return s;
}

// Cache-unfriendly random scatter read
long long random_sum(const std::vector<int>& v,
                     const std::vector<size_t>& idx) {
    long long s = 0;
    for (size_t i : idx) s += v[i];
    return s;
}

// Unpredictable branches (random input → ~50 % misprediction)
long long branch_heavy(const std::vector<int>& v) {
    long long s = 0;
    for (int x : v) {
        if (x & 1) s += x;     // odd/even depends on random data
        else        s -= x;
    }
    return s;
}

int main() {
    std::mt19937 rng(42);
    std::uniform_int_distribution<int> dist(0, 1 << 20);

    std::cout << "Allocating " << N * sizeof(int) / (1024 * 1024) << " MB...\n";
    std::vector<int> data(N);
    for (auto& x : data) x = dist(rng);

    // Random index permutation for scatter access
    std::vector<size_t> idx(N);
    std::iota(idx.begin(), idx.end(), 0);
    std::shuffle(idx.begin(), idx.end(), rng);

    auto t0 = std::chrono::steady_clock::now();

    volatile long long r1 = sequential_sum(data);
    volatile long long r2 = random_sum(data, idx);
    volatile long long r3 = branch_heavy(data);

    auto t1 = std::chrono::steady_clock::now();
    double ms = std::chrono::duration<double, std::milli>(t1 - t0).count();

    std::cout << "Results (prevent DCE): " << r1 << " " << r2 << " " << r3 << "\n";
    std::cout << "Elapsed: " << ms << " ms\n";
}

• sequential_sum gives high IPC (~2–4) and low cache-miss rate (<1 %).
• random_sum drives LLC miss rate to 20–40 % and drops IPC below 1.
• branch_heavy pushes branch-miss rate toward 50 %.

Compile and run:

1
2
g++ -O2 -g -o perf_demo perf_demo.cpp
./perf_demo

Timing + RSS (/usr/bin/time -v):

1
/usr/bin/time -v ./perf_demo 2>&1 | grep -E "wall clock|Maximum resident"

CPU counters (perf stat):

1
2
perf stat -e task-clock,cycles,instructions,branches,branch-misses,cache-references,cache-misses \
  -- ./perf_demo

Expected Output (actual run, hardware varies)

1
2
3
4
5
6
7
8
9
10
11
12
13
 Performance counter stats for './perf_demo':

    33,189,508,178      task-clock         #    0.998 CPUs utilized
    43,898,685,981      cycles             #    1.323 GHz
    67,699,556,400      instructions       #    1.54  insn per cycle
     9,385,632,421      branches           #  282.789 M/sec
       103,063,756      branch-misses      #    1.10% of all branches
       223,694,419      cache-references   #    6.740 M/sec
       165,054,219      cache-misses       #   73.79% of all cache refs

      33.271095641 seconds time elapsed
      32.689158000 seconds user
       0.504431000 seconds sys

Time breakdown

Cycles and effective frequency

• 43.9 B cycles / 33.2 s ≈ 1.32 GHz effective frequency.
• This is below typical turbo speeds (2–4 GHz). Possible causes: power/thermal throttling, container CPU quota, or frequent stalls causing the CPU to clock-gate.

IPC — Instructions Per Cycle

1.54 IPC — the CPU retired 1.54 instructions on average every cycle.

Cross-check: 1.32 GHz × 1.54 IPC ≈ 2.03 billion instructions/sec, which matches 67.7 B instructions / 33.3 s.

Branch misses

A branch misprediction occurs when the CPU’s branch predictor guesses the wrong direction for a conditional jump. Modern out-of-order CPUs speculatively execute instructions along the predicted path. When the prediction is wrong the pipeline must be flushed and re-executed from the correct target — wasting ~15–20 cycles per miss.

Common causes:

• Data-dependent conditions on random input (e.g., if (x & 1) where x is random → ~50 % miss rate).
• Indirect calls / virtual dispatch — the target address is computed at runtime.
• Loop bounds that vary — the predictor can’t learn a fixed pattern.
• Correlated branches deep in call chains.

This run: 103 M misses / 9.39 B branches = 1.10 % — normal range for a mix of predictable and unpredictable code.

Cache misses

• 223 M cache-references, 165 M cache-misses → 73.79 % looks alarming, but perf cache-misses measures Last Level Cache (LLC / L3) misses, not L1/L2.
• The denominator (cache-references) counts only the subset of accesses that reach L3, not all memory accesses, so the ratio can appear inflated.
• In absolute terms: 165 M LLC misses / 33 s ≈ 5 M misses/sec — not unusually high.
• The random_sum workload (random scatter over 256 MB) is the main driver; it defeats hardware prefetchers and forces most accesses all the way to DRAM.

Flame Graph (perf record)

1
2
3
perf record -F 99 -a -g -- ./perf_demo
perf script -i perf.data &> perf.unfold
# drag perf.unfold to https://www.speedscope.app

• Previous
  C++ Profiling and Speed Ups
• Next
  C++ Tool Websites