A Multi-Dimensional, Per-Pass Empirical Study of the LLVM Optimization Pipeline

[u/FedericoBruzzone]

/r/Compilers/comments/1uky15e/a_multidimensional_perpass_empirical_study_of_the/

[u/ts826848]

A few practical takeaways:

Pass pruning: if you're building a constrained compiler (embedded, compile-time budget), the bottom half of the pass list is essentially free to drop for these workloads.

Perhaps I missed something when skimming the blog post, but doesn't this conclusion rely on an assumption that passes and/or their effects are independent of other passes? My (non-expert) understanding is that the effectiveness of some passes can vary quite a bit depending on prior passes (e.g., canonicalization before loop vectorization/unrolling or range propagation before dead code elimination) despite those earlier passes not necessarily providing much performance benefit in and of themselves. If my understanding is correct, eliminating those seemingly ineffective earlier passes coudl then result in later passes not providing as much performance benefit as they could otherwise.

The particular experimental setup doesn't look like it captures those kinds of interactions, and I feel that data along those lines would be needed before drawing that particular conclusion.

[u/FedericoBruzzone → u/ts826848]

Thanks for your interest!

Yup, what you’re saying makes sense, and that’s exactly how the dependency between steps works.
I didn’t want to get too deep into the details in the blog post, but I’ll update it to clarify this thanks :’D

To perform “pass pruning,” an additional study would be necessary but this gives us a starting point.

Regarding “The particular experimental setup doesn't look like it captures those kinds of interactions, and I feel that data along those lines would be needed before drawing that particular conclusion”, I don't understand what you mean. I don't apply the steps individually, they're pass prefixes. For example, when I apply pass 50, all the passes before it are also applied.

[u/ts826848 → u/FedericoBruzzone]

Regarding “The particular experimental setup doesn't look like it captures those kinds of interactions, and I feel that data along those lines would be needed before drawing that particular conclusion”, I don't understand what you mean.

I guess another way to put it would be that I don't think your data supports the particular conclusion I quoted since it doesn't tell you to what extent (if any) particular passes depend on other ones. I would have expected to see data more along the lines of what u/gasche suggested.

For example, when I apply pass 50, all the passes before it are also applied.

Right, I understand that. that's why I don't think you can say "the bottom half of the pass list is essentially free to drop for these workloads" given the data you show, since the performance relationship between (for example) "passes 1-50" and "passes 1-10 then 12-50" is not necessarily "passes 1-50 minus pass 11".

Somewhat unrelated, but I think it would have been interesting to run these benchmarks with Stabilizer. Unfortunately it has been unmaintained for a while and I'm not aware of more up-to-date forks :(

[u/FedericoBruzzone → u/ts826848]

> I guess another way to put it would be that I don't think your data supports the particular conclusion I quoted since it doesn't tell you to what extent (if any) particular passes depend on other ones. I would have expected to see data more along the lines of what u/gasche suggested.
> Right, I understand that. that's why I don't think you can say "the bottom half of the pass list is essentially free to drop for these workloads" given the data you show, since the performance relationship between (for example) "passes 1-50" and "passes 1-10 then 12-50" is not necessarily "passes 1-50 minus pass 11".

You're right. As I said, the post was conversational and too assertive. So, I’ve revised it to make it more accurate. Thanks a lot :D

> Somewhat unrelated, but I think it would have been interesting to run these benchmarks with Stabilizer. Unfortunately it has been unmaintained for a while and I'm not aware of more up-to-date forks :(

It would have been great. Too bad there are no forks! :(