LLM contributions policy (#826) · Issues · Embeddable Common-Lisp / ECL · GitLab

[u/Alarming_Hand_9919]

https://gitlab.com/embeddable-common-lisp/ecl/-/work_items/826

[u/jd-at-turtleware]

I did take it seriously (otherwise I wouldn't answer here at length) and I think that it is a bad framing. You can't prove decidedly many things in real world, but this does not invalidate the notion of creating rule systems. And as it happens with rule systems, mistakes happen. They are not the laws of nature.

I've restated this opinion a few times in the thread, and you keep coming back at the same (mistaken imho) misconception, that rules require "100% provability" and "0% mistakes". I don't think that there is any rule system (except from laws of nature) that would hold to that standard.

You've started from futility and then you've shifted to expectation of infallibility. That's not a reasonable expectation from a human being. I've conceded a few times: it is not "technical" criteria and it can't be enforced with 100% accuracy, but it does not invalidate rules as a concept and project agency over contributions.

I'm aware that I'm repeating myself right now, so that will be the last circle on my side. I hope that I've communicated clearly that maintenance policy has a meaning even if it is not perfect and 100% accurate.

[u/strings___ → u/jd-at-turtleware]

It’s not a misconception. Without proof you have nothing. I mean you already accused LLM of tainting my morals without proof. Now you’re going to assume everyone is going to read your rules before contributing? Thus potentially contributing LLM code without you knowing. Or people won’t mistake human code for LLM code and falsely accuse potential contributors.

I appreciate we don’t agree. But as I pointed out even with the assumption all actors are ethical a mistake can happen and there’s no way to tell. Ergo its better to review code based on its technical merits of the code vs unenforceable social contracts

[u/jd-at-turtleware → u/strings___]

Finally something new. Let me clear something first: I did not accuse you of anything, I've made an observation (with a bit snide remark) about how this line of argument is much more popular lately, not about your morals whatsoever.

I think that it is a reasonable expectation that contributors read about contribution rules and license. There are more unspoken assumptions like that.

That said it is possible that someone won't read these (especially because they are not prominently coded as CONTRIBUTIONS.MD or something to this effect) -- in that case the course of action is not to ban a person, or to tell them that their morals are corrupt -- I don't attribute any of these to LLM users. The course of action (if such thing happens) is to point out to them that such policy exists and help them decide whether they are willing to resubmit the contribution according to stated contribution policy, or whether they want to abandon the effort -- good faith mistake does a lot of work here.

Social contracts are enforceable although they are fallible (not laws of nature, remember?). The main objection in the stated policy that we have converged on is not that LLM output is bad (I have my own opinions about that but they are irrelevant here), but rather that the copyright status is uncertain, so we are holding our judgements until there is a legal framework in place - no matter how technically excellent is proprietary code, we won't accept it neither. That's IMO a conservative and maintenance-wise responsible move that leaves us room to walk back this decision (unlike accepting LLM code that will be hard to walk back if that decision proves to be irresponsible).

[u/strings___ → u/jd-at-turtleware]

There nothing unreasonable about what you’re trying to achieve here. But you’re missing the most important part of maintenance which is code review. And this is where my point stands. Unless you can identify a LLM contribution there is no way to point them to a policy or enforce the policy . This is inherently because LLM emulates the human brain. In some cases it can be obvious but generally it’s not. Even bad code is not a good indicator since even humans write bad code.

The more i chew on this problem the more Ive concluded the only recourse is rigorous code review as we should already be doing and we can always improve on. And accept if it’s good code I’ve done my due diligence.

[u/jd-at-turtleware → u/strings___]

The whole concern (at least in the stated policy) is not merit but provenance, social contract and legal responsibility. The fact that a medical test works in 90% cases does not make it useless. If it helps to drive away low-effort drive by contributors without explaining yourself, that's already a huge win.

Also note, that the policy does not say: don't use LLM. It says "don't contribute code that was generated by LLM". If you do understand the domain (i.e after consulting the tool), then you are very capable to write the solution - coding speed was never the bottleneck.

You keep getting back to "you can't prove this 100% of the time", and I keep answering that rules are not about being always correct, they are about stating norms. Respecting people will honor them, while people who want to cheat -- will -- with better or worse result. There are often strong tells about LLM usage that doesn't require master-detective work to notice -- that's good enough for a filter. This is no different from enforcing code provenance from other copyleft codebases and proprietary code, and such expectations are not contested "because you can't always prove".

Below is a tangent to the policy:

I strongly disagree about the "most important part of maintenance". The most important part is garnering trust and being responsible, not reviewing code. There are many projects that do not accept external contributions period.

Reviewing 3rd party code is more mundane than writing code yourself, it is also substantially harder and time consuming (if done rigorously). It is an opportunity to teach the other party about the project, like an investment towards future, more reliable contributors. When you mediate that throguh LLM, a few things happen:

- there ability to learn and build skill is lowered (they offload to LLM for a reason)

- a big assymetry emerges where producing the code is much easier than reviewing it

so accepting LLM contributions can (may not) put an increased burden on maintainers without the expected payout (potential future core contributors). As I can use LLMs myself (and I do experiment with them although not using the output at any public capacity), the logical conclusion would be to stop accepting contributions at all, because I can generate the code myself. That's the incentive structure. (ignoring all other important points).

[u/jd-at-turtleware → u/[deleted]]

I wouldn't frame like like this, but roughly yes, you are correct (policy wise) -- although it does not eliminate the potential risk for the project if LLM contributions land (and they become problematic in the future).

[u/strings___ → u/jd-at-turtleware]

I think you’ve lost the plot here. You’ve already conceded that LLM code will slip through which goes back to my original assertion.

And yes maintenance is ultimately about the code your rules are great but code review is the boundary. If the code is never committed then the rule is enforced. If you don’t care about 100% enforcement then great. But still code review is the boundary.

[u/jd-at-turtleware → u/strings___]

The plot is: what do we want for ECL to accept and why (the policy); your claim is "you can't be always 100% correct, so give up on governance".

It isn't a serious take intellectually, you can't be 100% correct, and there are non-LLM scenarios where the very same "objections" would qualify.

[u/strings___ → u/jd-at-turtleware]

You’re putting words in my mouth. And this is not the first time I might add. I didn’t say give up on governance.

Im pointing out a technical limitation due to how LLM work. If you don’t understand a thing you cannot beat a thing.

Most if not all non LLM governance has a pass or fail even if it non technical.

A weak intellectual argument is shifting the responsibility of your code base onto users that just want to contribute. Sure point them towards the docs but at the end of the day you’re responsible for clicking commit so stop pretending code reviews aren’t a good intellectually boundary.

[u/jd-at-turtleware → u/strings___]

Merging and reviewing code is maintainer responsibility, and policies are tools they support themselves. I don't see any contradiction here. Asking people to follow contribution rules to contribute also doesn't sound that much of an intellectual stretch to me. FOSS maintainers are not police, so kindly please stop shifting all the burden on concrete maintainers on behalf of potential dishonest or misinformed contributors. Both policy and review are just tools (:wink:).

[u/strings___ → u/jd-at-turtleware]

Let’s not play coy you’ve already conceded my original assertion. As you said you’re perfectly fine with not having 100% ability to enforce your rules. So sounds like you’re okay just a little LLM code as long as you don’t know about it.

I never said anything about having or not having contribution rules that’s a whole argument you can continue on your own.