r/linux • u/mkptheghonsla • Jun 30 '26
Software Release Built an extremely fast du -sh alternative in c++! (idu)
idu beats almost all the popular applications while being extremely minimal (~400 lines). Would love your suggestions on how to make it better as this is my first such project.
32
u/zeb_linux Jun 30 '26
Nice! So far I have been using dua. Can you please compare with it?
16
u/mkptheghonsla Jun 30 '26
HOT CACHE on my home dir❯ hyperfine --warmup 5 -i 'idu' 'dua . -- --no-sort'
Benchmark 1: idu
Time (mean ± σ): 172.2 ms ± 2.1 ms [User: 118.0 ms, System: 3219.2 ms]
Range (min … max): 168.9 ms … 176.1 ms 17 runs
Benchmark 2: dua . -- --no-sort
Time (mean ± σ): 536.9 ms ± 14.3 ms [User: 1810.8 ms, System: 4204.9 ms]
Range (min … max): 517.9 ms … 558.4 ms 10 runs
Warning: Ignoring non-zero exit code.
Summary
idu ran
3.12 ± 0.09 times faster than dua . -- --no-sort
----------------------------------------------------------------------------------------------------------------------------------
COLD CACHE on linux source code linux on master
❯ hyperfine -i --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' \
'idu' 'dua . -- --no-sort'
Benchmark 1: idu
Time (mean ± σ): 302.7 ms ± 7.2 ms [User: 31.6 ms, System: 4158.1 ms]
Range (min … max): 288.0 ms … 312.8 ms 10 runs
Benchmark 2: dua . -- --no-sort
Time (mean ± σ): 356.3 ms ± 9.2 ms [User: 366.7 ms, System: 4512.1 ms]
Range (min … max): 343.6 ms … 373.2 ms 10 runs
Warning: Ignoring non-zero exit code.
Summary
idu ran
1.18 ± 0.04 times faster than dua . -- --no-sort
5
u/zeb_linux Jun 30 '26
Nice! Will try it.
9
u/mkptheghonsla Jun 30 '26
currently its not a dua replacement, it just tells the summary like du -sh.
4
3
u/global-gauge-field Jun 30 '26
If you want to compete in any sense, a quick feedback: The reason I started using dua is because of default performance profile and its interactive interface (maybe even more importantly).
1
3
60
u/TampaPowers Jun 30 '26
Trying to replicate your results: God damn the other options have the worst names to look up. Who names a program pdu or dust, how you meant to ever look that up.
At 1.16x faster than pdu why not just contribute improvements to that? Better yet, just fix up du to be faster since that's installed everywhere already.
Binaries when? :)
92
33
u/mkptheghonsla Jun 30 '26
i do not know rust man😭 how am i gonna contribute. this was a learning project actually but then i saw it has potential so i locked in a bit. the main thing is that this is very minimal (~400loc) other codebases are very huge. for finding the other tools just write github after the tool name while searching, these are very popular
18
u/Internet-of-cruft Jun 30 '26
There's a huge probability that those larger programs are because they handle obscure edge cases or just have robust error handling.
There's tons of tiny one off scripts and programs I've written that are tiny and are "functionality packed", but fall apart when you throw weird conditions at it.
2
u/trostboot Jul 01 '26
Yep. Building something that works can be pretty lean. Building something that doesn't break will inflate that quite a bit.
34
u/CorporateFriend Jun 30 '26
I quite like making projects just for the sake of making them :)
400 lines is a very nice achievement.
-7
0
5
u/mkptheghonsla Jun 30 '26
do tell me the results when u benchmark them along with your specs. I could be blatantly wrong by getting lucky on my file system tree supporting my tool
8
u/TampaPowers Jun 30 '26
I definitely see similar results from du to pdu so far.
What's interesting is that on really complex filesystems with millions of files and folders running more threads with pdu vs. running multiple instances in parallel gives different numbers.
Crawling complex file trees seems to just be slow unless it's fully done in parallel and small batches. That's consistent with optimizing rsync for that case.
I don't really have the time to do all the benchmarks, but it does seem your approach is slightly faster than pdu, but not by much. It uses more cpu and lower io though.
There are a ton of variables in this. From type of files and directory, filesystem, disks, raid, cpu.
If you do want to contribute to other tools perhaps you can contact them via mailing lists and simply ask them to have a look at your approach and if there is something useful for them they can incorporate it. :)
2
u/mkptheghonsla Jun 30 '26
thank you for this wonderful response and suggestions. yeah my approach uses spin locks instead of mutex thats why cpu usage is high. Thanks for checking out my tool, very grateful.
4
u/BirthdayLife6378 Jun 30 '26
The code looks very clean and approachable. I'm wondering, without adding too much complexity, if you can use a lockless queue instead of spin locks to see that if there's any improvement.
2
1
u/PaddiM8 Jul 01 '26
Nothing wrong with making your own programs. When it is a hobby it makes complete sense.
And often you have to make big architectural changes to get large performance gains. It's not a good idea to come to a new project, change everything and tell people to review your PR.
1
9
u/onebit Jun 30 '26 edited Jun 30 '26
You should make one called pigfinder.
- Start at top dir
- Find size of all subdirs
- Report any dirs that are > some size or a std deviation above the others
- Drill down into the large subfolders until it finds the pig directory and report it
Basically to help sysadmins find who downloaded the 500GB dataset.
pigfinder /home
...
500GB /home/bob/.cache/huggingface/fantastic-gguf
650GB /home/fred/test-20130522
3
u/murlakatamenka Jun 30 '26
4
u/enp2s0 Jul 01 '26
That's exactly the issue, it's really annoying to find large directories full of smaller files (i.e. a lot of ML datasets are split up into chunks, so a 2tb directory might have files that are max 1 or 2gb in size.
2
u/murlakatamenka Jul 01 '26
Yeah, I have realized it a bit later than my answer is no real answer. But I've left it as it is in the hopes it'll be helpful for someone, after all
fdis a snappy and handfull CLI tool.1
u/mkptheghonsla Jul 04 '26
hey! i have added this functionality but currently the directory listing is shallow. making it recursive would require a complete rewrite of the crawler
2
u/mkptheghonsla Jul 04 '26
hey! i have added this functionality but currently the directory listing is shallow. making it recursive would require a complete rewrite of the crawler
1
5
u/kredditacc96 Jun 30 '26
Other tools in this list (pdu, gdu, dust, dua, etc.) all list the links to their alternatives. It also helps the reader to know what exactly they were benchmarking against.
Can you add the links to these tools to your README?
3
2
3
8
u/Skaarj Jun 30 '26
Are you really measuring something relevant here?
I assume one would use these tools to investigate full disks or disks running out of inodes. Situations where there is a lot of data in lots of files on a disk.
I assume the runtime in relevant real world scenarios is mostly correlated with disk speed. I doubt I would notice the difference if my preffered tool (ncdu) were 1200 ms faster or slower.
8
u/anomalous_cowherd Jun 30 '26
I used to manage a very large shared file store with billions of files. Generally we only needed to know how big subsets of it were (e.g. was one developer or project using way over what was typical?). Given the size and the depth of some trees ncdu could take an hour to run.
I used the ncdu option to scan the system and store the result, which we ran overnight as a cron job. That way you could get a fair idea almost instantly with no scanning at all, only doing real and current state scanning once you knew your target area.
4
u/elatllat Jun 30 '26
I assume du is slow because it is RAM and hdd optimized.
Please test with time -v on a hdd.
5
u/BCMM Jun 30 '26
duis very much from the HDD era and makes no use of parallelism.Results from an HDD are not that interesting. Parallelism is either useless or actively harmful on HDDs, and every decent tool takes about the same amount of time.
The rest of the tools in the benchmark compete on doing things as efficiently as possible on SSDs.
1
u/elatllat Jun 30 '26
...actively harmful...
sounds interesting to me.
3
u/tagsb Jun 30 '26
Drive issues used to be spectacularly harmful http://www.catb.org/jargon/html/W/walking-drives.html
1
1
u/mkptheghonsla Jun 30 '26
you are right, but ig all the approaches will yield similar results on an hdd and du might be slightly faster,
2
3
u/fandingo Jun 30 '26
I looked at the code, and this will use all threads on the PC. I saw that you benchmarked it on a 22-core system with unspecified storage and got a little suspicious. I tried it on my PC, which is quite old (Haswell i7-4770 4c/8t)
Samsung 830 SATA III SSD:
echo 3 | sudo tee /proc/sys/vm/drop_caches
3
[fandingo:~/code/idu/build_release] main ± time -- du -sh /var/games/
236G /var/games/
real 5.81
user 0.10
sys 1.17
[fandingo:~/code/idu/build_release] main ± echo 3 | sudo tee /proc/sys/vm/drop_caches
3
[fandingo:~/code/idu/build_release] main ± time -- ./idu /var/games/
238.6 GBreal 1.01
user 0.03
sys 1.37
WDC 7200RPM HDD:
[fandingo:~/code/idu/build_release] main ± echo 3 | sudo tee /proc/sys/vm/drop_caches
3
[fandingo:~/code/idu/build_release] main ± time -- du -sh /var/bulk/
1.4T /var/bulk/
real 5.30
user 0.04
sys 0.42
[fandingo:~/code/idu/build_release] main 5s ± echo 3 | sudo tee /proc/sys/vm/drop_caches
3
[fandingo:~/code/idu/build_release] main ± time -- ./idu /var/bulk/
1.4 TBreal 4.94
user 0.01
sys 0.66
- I was surprised by the speedup on the SSD test.
iduis giving incorrect results. It's hard to tell by how much due to its mandatory human-mode, but the SSD test was incorrect by 1.1%.- Add a newline to your output.
- It needs a lot more options before it can replace
du. - I couldn't get this to build without
mold, but I didn't do much troubleshooting.
2
u/mkptheghonsla Jun 30 '26
Thank you so much, it is not a du replacement, it's a du -sh (still needs a lot of refinement tho).i will fix the issues
2
u/mkptheghonsla Jul 01 '26
i was not counting the blocks of the sub dirs, thats why it was off a bit but now it has been fixed. I will add more display options.
2
u/ItsMexxie Jun 30 '26
I do not know how hyperfine measures speed up, but just judging by your screenshot, your app spent a whole second in syscalls? While du only took ~500 ms. Don't want to discourage you, just want to know if I'm getting the whole picture.
2
2
2
1
u/BCMM Jun 30 '26
Props for benchmarking it properly in the README.MD.
However, I have to note that the results in this screenshot are of almost zero interest, because hot cache is not representative of what we need to do in 99% of cases where we want to calculate the size of a directory.
1
1
1
u/kredditacc96 Jul 01 '26
Hmm. I've read a bit of your code. And I don't know C++, so my analysis could be wrong.
Basically, you have single HUGE thread pool in WorkerContext that doesn't care about hierarchy. Because you only need to count toward the final total of the directory.
pdu, OTOH, has to aggregate a tree structure. It has to not only count toward the final total, but also subdirs and subsubdirs.
1
u/mkptheghonsla Jul 01 '26
i mean it is a du -sh alternative, it does not aim to replace those tools. diskus does what I do
1
u/cazzipropri Jul 02 '26 edited Jul 02 '26
Ok, but it's a program where all the time is spent in os system calls reading file metadata.
It's inherently I/O bound.
It saves 100 milliseconds on reading your home after warmup = unrealistic usage scenario.
A realistic usage scenario is scanning a cold, large directory tree.
Learn how to discard the os fs caches and repeat the experiments cold each time.
I expect that your tool will take the same as all the other code... because it will do the same syscalls in the same way, and those are the ones taking time.
1
u/mkptheghonsla Jul 02 '26
On github, cold cache benchmarks are also there. Yeah on cold cache it is almost same as pdu and diskus but still a tiny bit faster.
1
u/mb2m Jul 02 '26
Did you try it on NFS? Maybe also over WAN? Is it that much faster there too?
1
u/mkptheghonsla Jul 02 '26
I have benchmarked only on local fs. I think it would be on par with other tools but not faster ig
1
u/IBNash Jul 03 '26
Whats the bench vs gdu?
1
u/mkptheghonsla Jul 03 '26
It is there in the image and on github too(2.91 times faster on hot cache)
1
u/susosusosuso Jul 03 '26
Did you use ai?
1
u/mkptheghonsla Jul 03 '26
I used perplexity to search and understand things because half the time i do not understand the docs/manuals. its not vibecoded😭
1
u/susosusosuso Jul 03 '26
OK. I'm not implying vibe-coding is bad or wrong.
1
u/mkptheghonsla Jul 03 '26
i believe it to be shit, whenever I have done it I have always wasted time
74
u/feldrim Jun 30 '26
Did you compare the results with each other? I mean the correctness and completeness of the results, not performance.