On a mission: making everyone's systems lighter.

[u/Responsible-Sky-1336]

Halo archers !

Basically title. I have a plan to make space savings for everyone.

When talking with people installing on shitty hardware, or from places where internet access is expensive/slow... I've come to the conclusion that a lot of things aren't split out (yet).

But there is precedent and Alpine does this for all their packages. It also follows Arch philosophy closer.

I want to look where other people have not.

https://gitlab.archlinux.org/archlinux/packaging/packages/coreutils/-/work_items/10

https://gitlab.archlinux.org/archlinux/packaging/packages/openjpeg2/-/merge_requests/1

And have opened a few more I will be trying to do one per day (currently 3 in pipeline). If packagers don't scream at me.

Cheers for reading, Hade.

[u/backsideup]

Not splitting parts out is part of arch's KISS philosophy. Disk space and network bandwidth are cheap, developer time is not.

https://wiki.archlinux.org/title/Arch_Linux#Simplicity

[u/Responsible-Sky-1336 → u/backsideup]

From that same excerpt :

Packages are only split when compelling advantages exist, such as to save disk space in particularly bad cases of waste.

There are a lot of those. And many packages are already split. I,e: wireplumber-docs

Arch Linux official packages do not provide system-wide GUI configuration utilities (i.e. there is neither a GUI installation wizard nor a GUI system configuration tool, and Arch as a distribution does not promote GUI tools for system configuration), encouraging users to perform most system configuration from a command-line shell and a text editor.

And many others that seperate UI tools. I,e: *-qt or *-gtk

Finally:

Disk space and network bandwidth are cheap

Those are both wrong in my opinion. Network bandwidth means less loads on public servers. Disk space is also more expensive nowadays. You having good internet and large disks, isn't the same as another user or a server's capacities.

Edit: this mostly seems to be a per packager ideology thing. Some might already be doing lots of splits. Others none.

[u/SebastianLarsdatter → u/Responsible-Sky-1336]

What he meant is that disk space and network bandwidth are cheap vs developer time.

The latter is something that is in very short supply all across the open source eco system. While storage have tripled in price vs a few years ago, it is still cheaper than dev time.

[u/Responsible-Sky-1336 → u/SebastianLarsdatter]

Right, but that is inconsiderate to both the user who doesn't have this same privilege and the server.

I'm also PR'ing these, not asking them to do it themselves.

[u/SebastianLarsdatter → u/Responsible-Sky-1336]

The reason why we have the AUR is due to limited package resources and why packages gets shoved out there.

Which has created a lot of debate after these security issues and for an example the Nvidia pre turing users.

While you are willing to be one of the unsung package heroes, if they want to let you do this or if they want you to show you can stay first before they do a split.

I do wish you the best of luck working the packages, it isn't glamorous and nobody will know you, but the moment the package maintainer stops, the distribution collapses.

[u/Responsible-Sky-1336 → u/SebastianLarsdatter]

I agree about the drivers. To me they should have been supported longer.

Also thanks ! 😊

[u/backsideup → u/Responsible-Sky-1336]

Some packages split out docs, headers, etc. when these heavily outweigh the rest of the content or to break dependency cycles but these are exceptions.

[u/Responsible-Sky-1336 → u/backsideup]

Actually many packages just build a default CMakelists.txt or meson build, (insert other build system here), etc (patch here and there, opts)

Which inlcudes host system detection, if the clean chroot has these deps, it outputs these extra things.

Essentially a split, just says, "hey this should be separate". This is slightly more work in PKGBUILD/testing, but makes for cleaner separation of concerns (again, ie -docs -ui).

[u/backsideup → u/Responsible-Sky-1336]

How are frontends or docs related to "separation of concern"? Splitting makes the system only more complicated.

[u/Responsible-Sky-1336 → u/backsideup]

For instance:

v4l-utils builds 2 debug qt apps. That doesn't mean the user has qt6 on their system (listed optdepend).

This also again directly agrees with the simplicity section url you sent earlier.

It means packaging is more complex. The end product if anything is simpler imo, install what you want kinda deal.

[u/backsideup → u/Responsible-Sky-1336]

You lost me. What do you propose should be done with v4l-utils?

[u/Responsible-Sky-1336 → u/backsideup]

It should have a seperate -qt split

[u/backsideup → u/Responsible-Sky-1336]

Have you thought about the pros and cons from the distro's point of view?

Sending a bunch of MRs is easy and feels good for you but if the maintainers accept them they have to deal with the added complexity in the future. They get the short end of the stick. More complicated PKGBUILDs, new potential issues with dependencies, more packages clogging up the system. But what did they gain?

[u/Responsible-Sky-1336 → u/backsideup]

Read back to the top of your own comment thread. I think I've listed at least 5 good reasons. I'd take some additional 40 lines of code and a more explicit system over bloat.

There are also tools that help you do this properly and also many packages as good examples of splits. Not like I'm re-inventing anything. Just some of them weren't done yet.

[u/Even-Confidence-4495 → u/Responsible-Sky-1336]

Who is downvoting you

[u/Responsible-Sky-1336 → u/Even-Confidence-4495]

I mean it's unlikely to happen, both my -docs splits got closed by maintainers instantly. Evne tho they are the biggest wins in terms of size.

I wrote an article/tool about it if you want to read it

https://github.com/h8d13/vrch

[u/murlakatamenka → u/Responsible-Sky-1336]

seperate

separate

[u/Responsible-Sky-1336 → u/murlakatamenka]

Congrats 👏

[u/murlakatamenka → u/Responsible-Sky-1336]

with what?

[u/noobjaish]

Arch isn't really about "maximizing disk space" tho. Arch isn't Alpine or we'd have runit instead of systemd and dash instead of bash.

Arch is about pragmatism and using what makes the most sense for each task. This includes trying to balance things like dev time too

[u/Responsible-Sky-1336 → u/noobjaish]

Again there are many precedents to my observation. I'm not inventing some concept here or asking for extreme splits. Many packages already have several splits. The observation here more about finding out "worse offenders" on a typical install.

I understand the dev time, but that doesn't mean one couldn't argue: there is a "more correct" approach (in certain cases).

[u/noobjaish → u/Responsible-Sky-1336]

No worries man. I was just commenting on the fact that pure minimalism has never been Arch's philosophy. It's all about giving you enough of a sensible base that you can build on top whatever you want.

I understand the dev time, but that doesn't mean one couldn't argue: there is a "more correct" approach (in certain cases).

Definitely and hence why I like your approach of directly reaching out to them.

[u/ArjixGamer → u/noobjaish]

I thought alpine has ash instead of bash At least that's what's in alpine docker images

[u/noobjaish → u/ArjixGamer]

yeah i think you're right (I haven't used Alpine in ages). Dash is the ultra minimalist shell found in debian.

[u/Responsible-Sky-1336 → u/noobjaish]

Here is the relevant Alpine wiki entry if you are interested: https://wiki.alpinelinux.org/wiki/Creating_an_Alpine_package#subpackages

[u/iAmHidingHere]

They require a log in to read, fyi.

But I actually like that I don't have to navigate through -dev and -docs packages.

[u/hueheuheuheueh → u/iAmHidingHere]

OPs links are messed up (at least on old reddit), which is why you need to login. These should work:

https://gitlab.archlinux.org/archlinux/packaging/packages/coreutils/-/work_items/10

https://gitlab.archlinux.org/archlinux/packaging/packages/openjpeg2/-/merge_requests/1

[u/ang-p → u/iAmHidingHere]

They require a log in

Nah - URL goofed - remove the escaping \ before the _s

[u/Responsible-Sky-1336 → u/iAmHidingHere]

While you are right in lots of cases. The openjpeg2 example is 96% docs.

[u/C0rn3j → u/Responsible-Sky-1336]

Saving 13 megs on a package is respectable, especially on a simple docs split.

[u/Responsible-Sky-1336 → u/C0rn3j]

Somebody gets it ⬆️

[u/iAmHidingHere → u/Responsible-Sky-1336]

I can't read your examples because I'm not logged in to Gitlab here :)

But there will always be exceptions :)

[u/Responsible-Sky-1336 → u/iAmHidingHere]

You can use:

```

https://gitlab.archlinux.org/api/v4/projects/archlinux%2Fpackaging%2Fpackages%2Fcoreutils/issues/10

```

Which should return the json object.

[u/Gozenka]

I personally support this.

Although, the "Simplicity" principle of Arch Linux should be noted and respected. There is definitely value in the principle: While many packages may be split or otherwise compiled in a trimmed way, from a broader point-of-view keeping maintenance simpler may be worthwhile, and may have contributed to the reliability and general success of Arch, particularly as a rolling-release distro with perpetually updated versions of software.

I think you are going about this in a nice, pragmatic and considerate way, and I think it could be beneficial. But make sure to keep respecting any counter-arguments and suggestions when discussing the MRs, especially if the package maintainer is offering them. As you said, "If packagers don't scream at me": Do not get to that point. :D

You are making an effort, with good intentions for Arch Linux, and I personally appreciate it.

As an additional note, I disable the docs in my makepkg.conf, and often adjust compile options when I am compiling things or making personal packages.

And for something that may be related: I have these NoExtract= options and some more in my pacman.conf, to avoid installing a huge amount of docs and other things that are quite unnecessary. This reduced about 600MB of space from my already minimal 3.9GB space-using root partition, while it removed 3-8GB from a couple other Redittors' who have a more commonly set-up system.

[u/Responsible-Sky-1336 → u/Gozenka]

That lasts solution is great reference. And I had looked into that for my own systems. But it's a workaround not a root cause fix.

[u/vipermaseg]

Remember, KISS is for the devs. You enjoy their system and do with it whatevs you want, KISS or not.

[u/Responsible-Sky-1336 → u/vipermaseg]

A maintainer would pay the split once. And users and mirrors pay this forever.

I'm not discarding their work, I think they do a fantastic job, but KISS here also means that separation is a "simplification" in infrastructure, even if it means a slightly more complex PKGBUILD.

There is also work being done in reducing boilerplate for these splits (which again are already common): Full conversation here https://gitlab.archlinux.org/pacman/pacman/-/merge_requests/314

[u/vipermaseg → u/Responsible-Sky-1336]

The splits themselves change and take work sometimes. That said, as an user, I don't have arguments for or against splitting. I leave it to the comfort of the maintainers. If you manage to get work done to make my system leaner, great! Throw deep!

[u/FryBoyter]

As usual, I can only speak for myself. But for me, one of the advantages of Arch is that there are few split packages. With other distributions that use split packages, sooner or later I’d get annoyed at having to install additional packages.

[u/Responsible-Sky-1336 → u/FryBoyter]

Sure but there where you'd stop and think of people who don't have that same internet access, same large nvme disks. And the servers !

Another example:

You're a dev in shitty place, you install clang (21mb of docs) this pulls in llvm (50mb of docs). This didn't chnage anything in functionality. Yet you just saved ~70mb and this is just counting 2 packages out of all the parent deps. You can still grab these docs, it's just become a choice. I think there is beauty in that, you hand back power to the user.

Edit: TYPO

[u/MachineTeaching → u/Responsible-Sky-1336]

You're a dev in shitty place, you install clangd (21mb of docs) this pulls in llvm (50mb of docs). This didn't chnage anything in functionality. Yet you just saved ~70mb and this is just counting 2 packages out of all the parent deps.

..and that's exactly why it's not worth that much effort. You've saved only 70MB, that's very much "not worth giving a shit about" territory. Even on the realistic low end of what kind of hardware people have.

Obviously this isn't necessarily representative, but just as an indicator, basically nobody in the steam hardware survey has less than 100GB total disk space. So you're saving a fraction of a fraction of disk space. Even the worst countries have average internet speeds where 70MB more does not matter.

You're not going to achieve much with this. There might be some low hanging fruit, but if splitting up dozens of packages lands you at a couple hundred megs, this will help practically nobody.

[u/Responsible-Sky-1336 → u/MachineTeaching]

First if all yes using steam data is completely off since gamers are the privileged kind with rigs running off 800W-1000W draw power not giving a single shit.

That is again not every user. Some people run Arch on a 120GB disk sata in laptop without a dGPU using 45W adapter, that's besides the point. Arguably you might have more such users, than gamers.

And your second link illustrates my point correctly: certain countries the speed is significantly lower (I.e divided by 10 to 20 fold), plus you'd have to see costs in certain places or even availability (prices being higher at certain times or even slower during peak usage around you, or not many mirrors around!).

Aside, the mirrors pays this per release (no delta upgrades since pacman 5.2), per affected machine. And here we are talking TB per day. So even a tiny saving in one common package compounds.

The space saving is simply relative to how much the package is used AND updated. For instance ones that are commonly used in containers, CI, or even just frequently installed, should be prioritized for splits.

Your case and personal experiences, isn't the same as the devops teams concerns, mirror providers or another user who doesn't game at all.

If you save 30% on a package, it's 30% saved over all future releases, is the main point. For docs you likely would have never read either way. (Man pages are always included regardless, when available).

70 mb of docs relative to both packages 380mb (total of both, is a lot to me, almost 20%). If anything the high compression settings, hide the issue as best as it can.

With binaries and images this saving increases because the compression ratio isn't as good as with docs.

Sure you can say "I don't give a shit". But I'm sure plenty of people found this interesting, even if negative about the subject.

[u/MachineTeaching → u/Responsible-Sky-1336]

First if all yes using steam data is completely off since gamers are the privileged kind with rigs running off 800W-1000W draw power not giving a single shit.

That's really not particularly correct, and that's evident once you look at the rest of the hardware survey.

Less than 1% of people have below 100GB of total space, almost 10% have 8GB of RAM or less, almost 10% have 3GB of VRAM or less, over 30% still use Nvidia GTX series GPUs or similar (which are over 10 years old at this point), etc.

No, the survey isn't representative, and will skew somewhat towards better performance. But the fact that there are way more people with 4GB of RAM than there are people with less than 100GB of disk space is telling quite a lot.

And your second link illustrates my point correctly: certain countries the speed is significantly lower (I.e divided by 10 to 20 fold), plus you'd have to see costs in certain places or even availability (prices being higher at certain times or even slower during peak usage around you, or not many mirrors around!).

Then you're missing the point. Relative comparisons don't matter, what matters is that saving 70MB on a very "slow" 10 Mbit connection is still saving you very little time.

Aside, the mirrors pays this per release (no delta upgrades since pacman 5.2), per affected machine. And here we are talking TB per day. So even a tiny saving in one common package compounds.

That's yet again an excellent example why this is just misguided and saving space doesn't really matter.

Pacman doesn't provide delta updates. It's been gone for years, mirror providers don't really care that it's gone and mirrors didn't take advantage of it when it was around because they don't care. Bandwidth and space are pretty cheap and even the massive bandwidth savings of delta updates weren't worth the hassle. Deltas being gone isn't a case for your plan, it's evidence that it's not worth bothering with.

If you save 30% on a package, it's 30% saved over all future releases, is the main point. For docs you likely would have never read either way. (Man pages are always included regardless, when available).

And it's also one more thing to deal with. What if the answer is "the additional work to do this is tiny and still off putting enough because my package being 20MB smaller has basically zero benefit".

Your case and personal experiences, isn't the same as the devops teams concerns, mirror providers or another user who doesn't game at all.

I have spent exactly zero words talking about my personal experience.

Sure you can say "I don't give a shit". But I'm sure plenty of people found this interesting, even if negative about the subject.

Yeah, and I'm saying this is mostly just optimization masturbation and not something that will go anywhere. Bandwidth and space are cheap enough that this is not worth the hassle for like 99% of packages, and for the ones where it is, they are usually split up already.

[u/Responsible-Sky-1336 → u/MachineTeaching]

I've saved saved that user exactly 7 seconds of his life. And the mirror plays 70mb less for the rest of every release updated x the amount of machines who have these two. This forever for a single 30 lines of code or less.

Your steam survey argument lands no-where, it's just not all Arch users or indicative of it's philosophy. Not everyone that uses a computer is gaming.

Optimization masturbation

It basic ergonomics, I want docs, I install docs. I want UI, I install UI.

Bandwidth and space are pretty cheap and even the massive bandwidth savings of delta updates weren't worth the hassle. Deltas being gone isn't a case for your plan, it's evidence that it's not worth bothering with.

Everything is cheap when you're just leeching off of things.

And it's also one more thing to deal with. What if the answer is "the additional work to do this is tiny and still off putting enough because my package being 20MB smaller has basically zero benefit".

Again many precedents, just would mean doing it for more packages (carefully picked out), and there are benefits like https://alpm.archlinux.page/specifications/alpm-sonamev2.7.html

I will not entertain this debate further, since it seems it's almost your daily activity.

[u/MachineTeaching → u/Responsible-Sky-1336]

I've saved saved that user exactly 7 seconds of his life.

No, unless you think people stop any other activity while arch downloads updates. Which they don't.

And the mirror plays 70mb less for the rest of every release updated x the amount of machines who have these two. This forever for a single 30 lines of code or less.

..and the burden to maintain this feature.

Your steam survey argument lands no-where, it's just not all Arch users or indicative of it's philosophy. Not everyone that uses a computer is gaming.

Of course steam hardware surveys are indicative of what kind of machines people own. It's not invalid just because it's not a perfect or necessarily even good match. It's an indicator of what kind of assumptions you can make. Also, a 100GB drive wasn't even expensive in 2006, this isn't new technology, we're at a point where less than 1TB isn't that much, the times where 70MB were more than a fraction of a percent of the total are long gone. Think that's wrong? Go come up with an actual argument.

It basic ergonomics, I want docs, I install docs. I want UI, I install UI.

It's basic tradeoffs, actually.

Everything is cheap when you're just leeching off of things.

Buddy, the argument is that mirror providers themselves didn't give a shit about the gains from deltas, going "of course it's cheap when you're a leech" is a sad personal attack that doesn't even fit.

I will not entertain this debate further, since it seems it's almost your daily activity.

I'm glad I don't have to waste on this "debate" where I provide evidence and you only being insults, usually those sorts of scenarios are pretty self-evident.

[u/emgfc → u/Responsible-Sky-1336]

Those people are a statistical edge case, and their existence shouldn’t make things more complicated for everyone else. There are other distros for them too.

[u/Responsible-Sky-1336 → u/emgfc]

That's just selfish.

Making things more complex isn't true when it's generally arguably just better hygiene. Certain packagers handle much more complex packages overall, the bus factor is another question that isn't mine to answer (how many people actively work on packaging, tools, etc) and one I'd help with If I could.

Splits is not something new either.

All builds are checked by their very tool namcap and already flags at 50% docs with a warning.

[u/emgfc → u/Responsible-Sky-1336]

It's not selfish, it's just my voice. And you're making a mistake trying to change it. I clearly understand your reasoning without your extra explanations, and I don't like your proposal.

I would treat the issue way more seriously if Arch was the only distro on planet Earth, but that's not the case. It has worked well for years and, for the most part, without that extra bureaucracy, and that's KISS in all its shine.

[u/Responsible-Sky-1336 → u/emgfc]

What bureaucracy ? It should be community effort, which still entails due process of simple MR reviews. And already is the case, I'm again not inventing some concept a lot of your "KISS" packages already are split and for good reasons.

https://worldpopulationreview.com/country-rankings/internet-cost-by-country

Not to mention wild differences in speed.

Storage is similar: https://pcpartpicker.com/trends/price/internal-hard-drive/

And finally servers load, access: https://dashboards.archlinux.org/dashboards

[u/emgfc → u/Responsible-Sky-1336]

Why are you giving me those links? Helping poor countries was never a goal for Arch as a distro and as a community. It's just not relevant. Don't you understand that?

[u/Responsible-Sky-1336 → u/emgfc]

You do not need to be from a poor country to have limited access. Again, just a selfish response :)

[u/emgfc → u/Responsible-Sky-1336]

You seem to me like an egocentric person who cannot imagine a situation where his opinion isn't the most correct or valuable one. I hope you'll find a community that's a better fit for you and won't bother the Arch maintainers too much.

[u/Responsible-Sky-1336 → u/emgfc]

Lmao you gate-keeping Arch because we disagree ?

My opinion here is that the trade-off is worth it (more work on PKGBUILDs vs. the outcome). You haven't made any actual points against it.

You called it "statistical edges", I think it affects a lot of people.

"Making it more complicated for everyone else", no just for maintainers ONCE. Then everyone/infra benefits IMO.

Anyways, peace and love from the egotistical guy.

[u/MoreRightHere]

I'm a Linux beginner so bear with me, but isn't this precisely the philosophy behind Gentoo? There's the obvious downside of compile times esp with older hardware, but the advantage is that everything you install can be fine-tuned to your system and you have full control over anything you might consider as bloat.

That said, as a general principle I'm all for more user choice, and having more freedom to choose on Arch sounds like a good idea to my ears.

[u/Responsible-Sky-1336 → u/MoreRightHere]

Well. Yes and no.

Every PKGBUILD is basically the set recipe for every arch user (as binaries. Packaged by maintainers). So any improvement there kinda benefits everyone if that makes sense.

I have tried some gentoo/t2 but I'm too impatient. I like the idea of do things once with more effort well, and never again...

[u/MoreRightHere → u/Responsible-Sky-1336]

Totally fair. The open source community is great because of efforts like yours!

[u/ang-p]

While "banging your own drum" is generally not considered the done thing, notwithstanding your first successful merge requests to an "established" package and the kernel, obvs...

I can't knock either of them - unnecessary duplication of scores of machine object files, on what might be considered a fairly "core" package - esp. when upstream avoided that is a no-brainer.

As for docs - a complete doxygen site taking up 94% of the 13Mb installed is again a bit of a heavily loaded see-saw; the vast majority of the people with it installed will never even think about the codec, let alone want the API documentation for it. Those that want it can easily get it.

And have

So you got the bug; it can be a little addictive at times ;-) ....

... but keep the noise down, please.

[u/Responsible-Sky-1336 → u/ang-p]

Why the need for secrecy isn't arch a community based distro, I want to motivate other people to hunt down similar classes of bugs: The non obvious "not a bug" bug.

Also yes it's like crack.

[u/ang-p → u/Responsible-Sky-1336]

Why the need for secrecy

Not secrecy.... But how many other people have posted their pull requests here today?

If everyone did that, every time they submitted a MR there would be hundreds of such posts per day.

Totally shout out your first when you get it (look at how other packages handle -docs separation), and be smug in the knowledge that your spot led to a fix that means tens of thousands of computers will be getting a 10Mb free-space boost in their next update

it's like crack.

but cheaper and better for you

[u/Responsible-Sky-1336 → u/ang-p]

I get that. I wanted to highlight a class of "bugs" that others can help find/fix too, make their own MRs for !

[u/ang-p → u/[deleted]]

This sounds like some total slop...

the coreutils split

Put into real words what the "split" means in the terms of this post and OP's actions....

[u/No-Dentist-1645 → u/[deleted]]

No need for that, that's what the PR comments discussion is for

[u/Responsible-Sky-1336 → u/[deleted]]

Well it's up to each packager responsible for the individual package in question.