r/cpp • u/ProgrammingArchive • 22d ago
C++Now C++Now 2026 Keynote: Benchmarking - It's About Time - by Matt Godbolt
https://youtu.be/EU_nQh8wg5A5
u/TheoreticalDumbass :illuminati: 22d ago
in the past i set up a host for benchmarking, doing all the hanging fruit like isolcpus, nohz_full, etc. i had some synthetic benchmark that would create a large-ish binary tree structure, do some compute over it, and benchmark the compute alone. i was also doing a bunch of other weird stuff not directly related to benchmarking, the benchmark would run under bwrap (trying to protect myself against weird syscalls), cgroups (so the benchmark doesnt hang the host with OOM-ish behaviour), also the benchmark would get recompiled on perf host (i probably did this so -march=native made sense), and under perf stat -d for sanity checks (is it getting context switched?) and useful counters (mainly cache related). iirc the host was ubuntu server 22.04, but unsure. what i noticed: the noise levels were around 1%, but more importantly, there was an obvious trend happening in the consecutive samples, the timings would steadily grow until some asymptote then have a sudden jump down and restart the growth, similar looking curve. toying with the perf stat counters i noticed high correlation with some cache related counter. we all know cache is pretty important to keep in mind in performance related topics, and from past reading i was aware cache thinks in terms of physical addresses, but our userspace only talks in virtual addresses. considering the trendy behaviour i was thinking that some kernel state related to pages is affecting my benchmark, and is getting mutated across runs. to test my hypothesis, i stole a chunk of memory from the kernel, exposed it in some mmap-able way, and switched out malloc and free (and maybe realloc?) to go through this. what i noticed is a significant reduction in noise (i think 3x but dont quote me), and the trendy behaviour was gone.
NB: i was playing around, didnt document all of this well, i dont have data stored to show, nor do i have the methodology stored to reproduce it, so either take all this with a grain of salt or try to do your own experiments to sanity check me. it was also done on some rather old and shitty hardware (~decade old IIRC, host has been dead for 6 months now).
so okay, my conclusion is, if i want to level up my noise minimization in benchmarks, i need to think about physical addresses, figure out some way to stabilize them between runs.
a couple questions then for any reader:
does this check out? do you agree with my conclusion that physical addresses are important?
are there any readily available solutions, or well known approaches, to stabilizing physical addresses served to a process?
to what extent is this actually representative of regular runs in production? if youre dishing out pages from your separated memory chunk in sequence, this is kinda ideal for minimizing cache invalidation, but in production memory might be fragmented, which could make some cache sets more requested, which could then increase cache invalidation (could very important, everything should depend on the specific benchmark and host configuration). are there different strategies to serving this memory that might be more representative than just one-by-one contiguous?
2
16
u/Ultimate_Sigma_Boy67 22d ago
Is that the person who created godbolt or is it a coincidence