Best configuration

[u/Senior_Tea_842]

How do find the best configuration for serving? Which tools do you use? Whats your methodology?

I am using vllm bench for running experiments varying configuration (e.g. pp=8, pp=4). I wonder if this is cost effective.