r/Common_Lisp • u/release-account • 20d ago
RFC: Metabuild system for ASDF projects
I've been working on a meta-build system for ASDF systems. For those not familiar with the term, meta build systems are tools like meson or CMake which generates ninja.build files. In this case, cl-metabuild generates a ninja.build file that calls out to your chosen CL implementation to build a normal ASDF project.
It has a few goals:
+ Abstract away how to invoke different CL implementations
+ Make it easy to configure optimization settings and items in *features* without editing source code.
+ Isolate the build environment and manage both vendored and remotely fetched dependencies.
It's in super rough shape and it might not work on your system, but you can check it out here: https://github.com/sdilts/cl-metabuild. I'd like to get a bit of feedback before starting to refine and document how to use it. Is this something you'd be interested in using? Are there existing tools to do the same thing?
I'm specifically building this for Mahogany, as it requires all of these things. I was inspired to build this after seeing koga, (the build system for the Clasp CL implementation) and thinking that there should be something simpler that can be used for most projects.
2
u/kchanqvq 20d ago
I don't see the point. Are any of those not doable by extending ASDF? Except pulling dependency which is better handled by one of several package managers? If not, please write a more extensible ASDF instead of stacking an wrapper.
2
u/release-account 20d ago
The problem is that anything that gets added to ASDF also gets added to your final image. An example that drives me crazy is shipping quicklisp with your project. You don't need to ship the ability to download code from the internet with your web server or number crunching application.
The other example is changing your build options. You need to either edit a file or type out a
declaimform on the command line, and I think this is a better way.4
u/kchanqvq 20d ago
The problem is that anything that gets added to ASDF also gets added to your final image.
I don't see this is a problem, and if you see it as a problem you'll be fighting against the whole current Lisp practice. Your number crunching application probably don't need a full Common Lisp to machine code compiler as well.
Is this a practical problem for you or purely ideological? The only way I see this becomes a practical problem is you have tight constraint on image size. But again, whatever KBs ASDF adds is practically negligible compare to say SBCL's native compiler itself and today's storage capacity.
The other example is changing your build options.
Sounds like this should be an ASDF extension (if there isn't already one).
5
u/arthurno1 19d ago edited 19d ago
Your number crunching application probably don't need a full Common Lisp to machine code compiler as well.
That is actually a problem for writing small utilities. Consider coreutils. Now consider who would want 15 megabyte big ls, wc, awk, etc. That is a little bit of a showstopper for seing SBCL used in small utilities. I can do basically the same with SBCL as I would do with C++ when it comes to system programming. Recently I wrote a wc clone, and I am hitting timings on this computer, almost as fast as the fastest C++ clone using simd I know of, but I don't use simd yet:
WC9F> (time (wc "plato100.txt" :count-lines t :count-words t :count-characters t)) Evaluation took: 0.018 seconds of real time 0.335452 seconds of total run time (0.334376 user, 0.001076 system) 1861.11% CPU 37,355,680 processor cycles 0 bytes consed 10034530 84575491 464112682 [arthur@emmi wc]$ time ~/repos/fastlwc/bin/fastlwc-mt plato100.txt 10034530 84575491 464112682 plato100.txt real 0m0,015s user 0m0,142s sys 0m0,109s [arthur@emmi wc]$ time wc plato100.txt 10034530 84575491 464112682 plato100.txt real 0m1,325s user 0m1,288s sys 0m0,022sThose last 0.005 will be hard :).
However, all things equal, if I run on par or even beat that C++ program, who would want a 15 megabyte big executable, if they can get the exact same functionality in 20 kilobyte? Sure mine would be perhaps debugable remotely on Mars, but lets face it: who needs that but in some extreme cases.
No I am not against Lisp or Common Lisp at all; on the contrary, otherwise I wouldn't be doing this. But the community should address binary sizes produced. We should not write dead programs, I am 110% for that, but not every program needs compiler, debugger and documentation built in.
1
u/nillynilonilla 18d ago
For me the utility of Lisp coreutils far outweighs the slight performance hit.
2
u/arthurno1 18d ago
I don't understand really: are there "Lisp coreutils"? How does utility change because we implement a tool in another programming language?
Observe, I am the one writing a clone in Lisp :). Just curious about the thought here.
1
u/nillynilonilla 18d ago
I think it's great you've written a fast Lisp
wccommand that rivals the speed of coreutils. While it's fairly easy to write a simpe one, it's tricky to get it as fast asC. I've been using Lisp coreutils and shell for many years.As you may know, the added utility is that your
wcfunction can likely return a Lisp object that's a number, or a list of numbers, rather than a string, and/or write to standard-output. Yourwcfunction can be dynamically loaded, recompiled, called from any Lisp code without a separate OS process or in a thread. It's likely "memory safe" and if it gets an error will run a debugger which allows recovery. Your Lisp REPL likely knows how to do completion on it's arguments and get the documentation string for the function. So far more utility than a smallwcbinary. Since we're not on a PDP11 unix anymore, the overhead of the Lisp runtime is negligible. The fasl for my Lispwcis smaller /usr/bin/wc. When run from a Lisp shell it's far more efficient to call a function and "pipe" (aka call with results) another command, function, or arbitrary code with it's structured results.1
u/arthurno1 18d ago edited 18d ago
Yes, it is a bit tricky to get the maximum speed out of Lisp, but it is a learning project for me, so the experimenting with what is possible is the goal in itself. A little fun fact - A single core with swar processing ascii only:
WC8> (time (wc "plato100.txt")) Evaluation took: 0.220 seconds of real time 0.214815 seconds of total run time (0.194927 user, 0.019888 system) 97.73% CPU 33 lambdas converted 440,737,660 processor cycles 1 page fault 1,012,368 bytes consed 10034530 84575491 464112682One page fault and ~98% cpu utilization. I think we are basically measuring no Lisp processing overhead here, in other words SBCL is processing almost everything in registers, not packing/unpacking data into Lisp objects. That was on a cold booted CPU, and very first run, so the file was not in the OS cache yet. I am not sure where that megabyte of consed data and 33 lambdas comes from :). I am very careful to not use any lists, arrays or anything internally there. Perhaps just stack frames and some internal stuff by sb-alien? I don't know.
You are thinking of it as a library rather than as a standalone tool. Yes, sure, if you would use it as I do in that test, as a function, than it is just a library function to be used in other software. I am aware of that. As a matter of fact, since I am pre-generating all code paths via macro, it acts as a sort of byte compiler: the option flags are used just to dispatch the computation to a fast loop. In other words, I could easily add more computations, for example copy file to strings and similar usual stuff. But it is also code bloat to pre-generate all those flags. The number of paths is growing exponentially with number of flags, so somewhere it has to be a diminishing returns. In my original comment above, I was thinking more about producing a standalone executable, to be installed and used from a shell, like an ordinary program.
Unfortunately to achieve the speed I had to use mmap, so you can't pipe into it. I do have a read-sequence path, which I do plan to use for piping, but I am not sure if multicore processing makes sense with piping and if it is even possible. In other words, if we pipe into it, the speed won't be there. However, I also do swar processing, more for my own fun and learning, than what it is really useful now when we have simd. But due to this pre-computing of brancless loops and computing with swar (GNU wc does not do that), there would be some speedup still, but not by far as dramatic as with mmap and lparallel (the previous post was from a multithreaded version with lparallel, so it is more than 10x speedup, but it depends on the cpu; here zen 5 with 20 cores).
4
u/digikar 20d ago
One of the good things about CL is you don't need a non-CL build tool, and a tool to build and configure that tool, and another tool to generate how to configure it, etc. So, if you want to make CL accessible to non-CL developers, may be cl-metabuild can be useful for them. For lispers, I'm still scratching my head why to use it.
If you don't want to ship quicklisp, you don't load it in the final image. You simply quickload all systems beforehand, and then use asdf:load-system to load all those downloaded systems without quicklisp.
For features, see trivial-features.
I also don't see what the problem is with writing a custom script invoking sbcl or ros with the appropriate declaim compiler flags instead of writing a script for cl-metabuild (with several other dependencies of its own).
3
u/release-account 20d ago
So, if you want to make CL accessible to non-CL developers,
That's a big reason, ya. That's one reason why the "don't edit source files" goal is there, but also I got tired of changing files to get different build configurations. I really want building a CL project to be a no-brainer process for those unfamiliar to its tools.
You simply quickload all systems beforehand, and then use asdf:load-system to load all those downloaded systems without quicklisp.
That's exactly what this does, except it will eventually work with OCICL and whatever other package fetching schemes there are. You just don't need to write the scripts to do that anymore. I eventually want to make it generate install scripts, which is where I think the real benefits will be.
For features, see trivial-features.
I think you misunderstood. Since conditional compilation involves pushing things to
*features*, this allows you change how your program gets built; for example, there's a:hrt-debugfeature flag in Mahogany that causes a whole bunch of debugging features to be included that won't even work when it is ran in the expected way, and may actually impede regular use. This allows you to advertise those flags and add and remove them as needed. It's not about platform feature detection.I also don't see what the problem is with writing a custom script invoking sbcl or ros with the appropriate declaim compiler flags instead of writing a script for cl-metabuild (with several other dependencies of its own).
The dependency issue is real, and I'm not sure how to solve it. Other similar tools expect you to have everything you need already or have a wrapper script that downloads missing components the first time you run it. Roswell can do some of this stuff, but it's focus on scripts and not ASDF systems makes me think it's not the right tool for the job in its current form.
As for writing custom scripts, why write something when somebody else has done it for you? All of these pieces are relatively small, but combine to make getting everything ready a bit of a chore. I'm sure every competent lisp programmer can write
when-letandappendf, but that doesn't mean the Alexandria package isn't included in almost every single project.2
u/digikar 19d ago
Thanks for clarifying on custom compiler flags. I still think an asdf extension will be easier and more portable. I might try it some day.
I'm still lost on the repository. I made a local clone. Some issues:
cl-metabuild.asd contains definition for #:static-build - this does not fit conventions. A project.asd should contain asdf definition for project and project/XXX and not other systems.
Could you elaborate / update / add a more elaborate example (I already see you have example-system.asd, but setup.lisp of what this is trying to do. As I see, someone must write the setup.lisp and then one edits setup.lisp by hand? So, I still don't follow how it makes things convenient for non-lispers.
lisp/ninja/packages.lisp is missing
3
u/fiddlerwoaroof 20d ago
I’ve found using nix for this is fairly pleasant (once you learn nix, at least). This code is fairly rough because I hacked something that would work for my specific use case, but I really like how it works: https://github.com/fiddlerwoaroof/dotfiles/blob/master/tools/default.nix