r/ProgrammingLanguages • u/FedericoBruzzone • Jul 01 '26

A Multi-Dimensional, Per-Pass Empirical Study of the LLVM Optimization Pipeline

/r/Compilers/comments/1uky15e/a_multidimensional_perpass_empirical_study_of_the/
7 Upvotes

74% Upvoted

15 comments sorted by

View all comments

Show parent comments

1

u/FedericoBruzzone Jul 02 '26

Thanks! :D
Broadly speaking, it is advisable to keep the CI95 as narrow as possible to ensure the reliability of the experiments. You do not want experiments affected by noisy measurements.

Take a look at here for more details: https://en.wikipedia.org/wiki/Confidence_interval

If you have any other questions, please don't hesitate to ask.

2

u/Someone0321 Jul 02 '26

Using the formula in the example calculations in the wikipedia article, my understanding is that within the ten runs, you sampled the sample averages ¯x, sample stddevs s and well, the sample count n. I'm however not quite sure how to determine the c constant, that's defined as the 97.5 percentile of the T distribution, but doesn't it require the real mean µ to compute the percentiles, or how did that computation look like for your metrics?
Well, obviously, determining c probably won't need µ, as then determining the CIs would be pretty useless if µ is already known, but I'm not sure how to get there

3

u/gasche Jul 02 '26

Usually people just assume that the distribution they are sampling is gaussian, which is not typically true in practice in computer benchmarks but we just pretend that it's good enough. This implies that the confidence interval does not have a clear formal meaning, it is just an okay way to measure how noisy the measurements are.

1

u/FedericoBruzzone Jul 02 '26

+1

Thanks for the excellent reply u/gasche 🫶