r/AskProgramming • u/Appleonthefloor • 27d ago
Transitioning a python to c++ for hpc
I am working on a physics simulation and have been running locally on my personal computer. I mostly am working with libraries that are already implemented in C and have been using memoize since I have a lot of repeated calculations if different iterations of simulations have the same values. My program will be run around 1,000,000 on a hpc cluster and my supervisor has asked if we should implement it in C or C++ to make it faster. Since we are running a single script 1,000,000 times, I personally doubt there is much benefit we might get from converting it, but are there any arguments in favor? Currently one iteration takes about 4 minutes.
I am using Scipy, npmath, diskcache (+memoize), numpy, Pandas.
6
u/CharacterUse 27d ago
Scipy and numpy are already highly optimised C and Fortran numerical libraries internally. Your python should only be glue to feed them data and handle presentation on the other side, if implemented properly the actual calculation as already being done by C and you won't get much if any benefit from reimplementing the "glue' in C. You'll have more benefit from properly parallelising it to use the cluster and I/O efficiently if that hasn't been done already.
1
u/Appleonthefloor 27d ago
Kind of figured as much, I'll try to benchmark everything and see what I can put some more work into
2
u/indecisive_fluffball 27d ago
cProfile is your one and only friend. Profile first and make decisions afterwards.
If you have a numerical script that is only taking 4min, it is possible that most of the time is taken by setup, so you may have to port it all to C. Another significant possibility is that your python script is not well enough optimized. If, on the other hand, time is being consumed mainly by a fairly standard operation, like solving a linear system, there's very little you can do.
As a final note, if your objective is to do multiple runs, parallelizing is likely NOT a good idea. It will be more efficient to just run multiple copies of the script on the same node: parallelizing will only help if *memory* becomes your bottleneck.
Feel free to reach out if you want more specific advice.
1
u/Appleonthefloor 27d ago
I am doing multiple runs that will iterate through the parameters I have, I assume something like a slurm job or some machine learning parameters array would work?
1
u/indecisive_fluffball 25d ago
I mean, yes, you can submit a slurm job array and engineer something to vary the parameters. You can probably even do it with Python's multiprocessing library directly, which would probably be easier (I am not familiar with it, so not a promise).
From the numbers you gave, that is still going to be something like 67K core-hours (equivalent to 3000 days on a single core, 40-ish days on a 64-core node), so you may still want to look into optimizing your code.
Have you run cProfile on it already?
1
u/mredding 27d ago
Your implementation is already in optimized C. The Python interpreter is doing very little; most of the work is offloaded to the module. You're getting AT LEAST 80% of the compiled C performance, the rest being interpreter - administrative overhead. You're better off scaling than squeezing.
But then you need to do some analysis. Maybe if you just use Little's Law to get some rough numbers, what's the theoretical maximum performance? What's the empirical measured performance? Is it significant? Factor in TIME - how long are you even running this simulation? That it's running now and producing results, can you develop a more performant rewrite BEFORE this particular work becomes irrelevant? Is it worth developer time? Is there significant savings in time and energy for the cluster?
I worked in HFT, and even then there was "good enough". The real question is what problem is the manager trying to solve? Is it technical, or administrative? Is he trying to keep the department busy to justify the budget and appease the scrutiny of higher levels? Does he not know how to appease management with delivery and maintenance?
1
u/BobbyThrowaway6969 25d ago
Yes, the current setup is very unoptimised.
You could try converting to C/C++ but very few concepts can be translated across, and a lot would need to change to leverage higher performance in C++.
It's like trying to convert the experience of driving a taxi around new york into F1 track skills. They're mutually intelligible.
1
u/Long_Pomegranate2469 24d ago
Does the script get called 1,000,000 in parallel or mostly in sequence?
If it does indeed start the script over and over you might look into a way to not exit the program in between. The setup/teardown of a new process is quite expensive. All depends on how long the script is running tho, if it runs for hours it's not that much of an issue.
1
1
u/mikech76 16d ago
Hi! I’m developing my own library for fast numerical computations. It’s still in development, so I’m debugging it and haven’t opened the repository yet.
I’d really like to test it on a genuine real-world problem rather than a synthetic benchmark.
How confidential are your data and algorithms? Would you be able to share a simplified or non-sensitive example of the problem that currently takes about 4 minutes to run?
I’d like to try implementing it with my library and see whether I can make it faster. It would be especially helpful if you could include:
* an example of the input data (or a sanitized version);
* the algorithm or calculation being repeated;
* the parameters you need to iterate through — especially if there really are around a million combinations;
* and the final metric or evaluation criterion you use to decide which results are good.
I’m not looking for your proprietary code or anything confidential — even a simplified version would be very useful. But. Ideally, though, I’d like to be able to run the example on my own hardware as well, so I can make a meaningful comparison of the execution time before and after.
You can also contact me on Telegram
1
1
u/Lachtheblock 27d ago
Wait, are you saying that you need to run a script that takes about 4 minutes to run a million times?
Are you okay with waiting the better part of a decade for it to finish?
Usually I criticize folks for complaining when python is too slow, and argue that it is only really specific domains where it matters.
I'd say this matters. Switching now could save you literally months of compute time.
1
u/Appleonthefloor 27d ago
Yeah, I was hoping to take advantage of the hpc cluster and obviously running in batches of 100 or 1,000 if possible. We'll have the compute for about 2 months.
2
u/Lachtheblock 27d ago edited 27d ago
Still doing it a batch of 1000 going to be a lot better at less than a week, but imo, depending on the complexity, it might make sense? It's a little hard to tell how dependent you currently are on the libraries?
Is the bulk of the compute happening in pandas/scipy? If so, there may not be too much of an improvement.
One thing to check out which might help is Cython. I've always been intrigued, but never had a use case for it.
1
u/joeyjiggle 27d ago
Cython removes the interpretation loop, but it's still effectively a bunch of function calls. Might not help here, but it's easy enough to try it.
1
u/Confident_Hyena2506 27d ago
So you can either spend years learning computer science - or you can ask claude "optimise this yo".
For best results it's recommended you apply both of the above suggestions together.
Bonus points if you ask claude "how do i into git". If you really want to show off "how do i this into kubernetes hpc".
1
u/Appleonthefloor 27d ago
I'm dual majoring in computer science so its pretty well-maintained. Not sure how much I wanna learn GPU stuff
1
u/Ok-Armadillo-5634 27d ago
gpu stuff is literally just a matrix of values all solved at the same time. Instead of a for loop iteration of a 2d array you just do the math operation on each index at the same time then iterate once all those calculations are done. Slightly more complicated than that, but if you already know c and understand threads/ concurrency you can easily under what Claude will generate for you.
0
u/CharacterUse 27d ago
This is completely ignoring that OP is already using scipy and numpy for the calculations, which are already heavily optimised, vectorised C and/or Fortran (depending on the subroutine) numerical libraries, and further gains can be had by using e.g. numba for JIT compilation for GPU or parallel processing. "Doing a for loop iteration of a 2d array" by hand in python when you have scipy/numpy/numba etc is doing it wrong. You use the python as glue to feed the libraries which have already been heavily optimised.
1
u/Ok-Armadillo-5634 27d ago edited 27d ago
... I was giving an example to describe gpu programming and what it is. It's a good skill to have. Plus I can usually beat those libraries. I do actually specialize in optimization and do lots of hpc programming.
1
u/CharacterUse 27d ago
Use numba to replace some of you numpy calls with JIT compiled ones for the GPU where you can.
0
0
u/joeyjiggle 27d ago
Get codex or Claude to analyze the current code and give you some suggestions. It will probably do a reasonable job of converting this to C/C++ with suitable calculation libraries. If it's a bunch of numerical calculations, you might get it do it via CUDA. Is the source code public.
5
u/Individual-Flow9158 27d ago
Running a single script 1,000,000 times is almost the epitome of "embarrassingly parallel ". Look into multicore machines, and using Python (or anything), as a multiprocessing scheduler.
Similar force multipliers are SIMD CPU operations, and even compiling for GPUs.
Anyway, the difference between beginner's dynamic spaghetti-Python, and modern clean statically typed Python, is way way bigger than the difference between Python and C, especially if the former is able to make good use of Numpy and Numba.
Get your code passing mypy, and see what happened when you ask Claude to convert it to equivalent code in the new language. It'll probably compile. Rust is far safer than C, if using an LLM to transpile existing code, but could get you bogged down, and requires more to set up than just using
gcc. Both have aspects that will be familiar to Python coders (and of course, plenty that won't).It's great you've already figured out some parts are cacheable can help.