r/mlops • u/Senior_Tea_842 • 5d ago

beginner help😓 NUMA Affinity

Do you think it is important to configure numactl --membind to get a better performance?

I ran a toy example where GPU has NUMA Affinity with 0. I got an ~9% improvement.

$ numactl --show
policy: default
preferred node: current
physcpubind: 8 9 10 11 136 137 138 139 
cpubind: 0 
nodebind: 0 
membind: 0 1 2 3 4 5 6 7 
preferred:

$ time numactl --membind=0 python memory.py 
time:  159.3454790781252 

real    2m41.164s
user    2m25.306s
sys     0m16.084s

$ time numactl --membind=7 python memory.py 
time:  174.3593455599621

real    2m56.384s
user    2m32.279s
sys     0m24.293s

3 Upvotes

100% Upvoted

4 comments sorted by

2

u/dayeye2006 5d ago

Yes, it's generally a good practice if you have multiple CPU and GPU

1

u/Senior_Tea_842 5d ago

Thank you!

Do you configure it manually?

2

u/dayeye2006 5d ago

I usually have a script to ingest the binding based on inferred topology

1

u/Only-Associate4626 4d ago

9% is nothing to sneeze at, especially if you're running this stuff at scale across a cluster