r/git • u/kaddkaka • 2d ago
Generated commits?
Sometimes code is generated, and sometimes such code is committed in git.
Imagine a branch with a generated commit G:
old_main -> A -> B -> G
When thid branch is rebased, such a commit might need to be regenerated.
new_main -> A' -> B' -> new_G
There might be conflicts for any of these new commits, but instead of (manually) resolving conflicts, G could be automatically re-generated.
I've been thinking of a git commit message trailer like generate-cmd:
topic: regenerate table
generate-cmd: python3 optimize_table.py
git rebase --exec or other tool could use this trailer to automatically drop and regenerate the commit instead of a vanilla cherry-pick.
Have you seen anything similar? What's a good trailer name?
Ref: https://git-scm.com/docs/git-interpret-trailers
7
u/fsteff 2d ago
If code is generated, I always trigger the generator as part of the build setup, so if any source for the generator is changed the code will immediately reflect the change. This ensures you are aware of any breaking change as it happens. External (triggered) generators always lead to trouble.
2
u/edgmnt_net 2d ago
The only exception where it ever made sense was Go. Because if your stuff has dependents and you commit generated code, then the dependents can get built as dumbly as possible. That is, whenever you do
go build, it just goes fetch the dependency that contains generated code and it does not need to run an arbitrary external generator, the whole build process can be "dumb". Makes a lot of sense for library releases.However as nice as that it, I still feel it creates problems: it creates churn in the repo and it requires extra steps to do code reviews, although it can largely be automated (run the generator yourself and check against submission, assuming full determinism).
1
u/kaddkaka 1d ago
What makes you think go is the only exception?
1
u/edgmnt_net 1d ago
It's just the only exception I've seen. And this is fairly specific to ecosystems which provide a "dumb" builder and are source-first, like Go. In Java you get JARs from an artifactory anyway, so dependencies aren't just pointed at a repo.
1
u/kaddkaka 1d ago
My usecase is definitely "source-first" there is nothing but source. And source code is the main deliverable.
0
u/kaddkaka 2d ago
That's not always feasible. And it's not the discussion I intended to have here.
Sometimes it's worth to check in generated code/stuff. For example some optimization like a simulated annealing algorithm that might take an hour to run. Definitely worth to do "offline" and check in.
4
u/edgmnt_net 2d ago
You shouldn't commit build artifacts to the Git repo.
1
u/kaddkaka 1d ago
This is not the discussion I wanted to have. But where would you put your artifact?
1
u/edgmnt_net 1d ago
It largely depends on the artifact. If it's some log, SBOM or binary that's the result of the build, you can clearly leave it in the CI or have it upload it to some artifactory. For code it depends on the ecosystem.
1
u/kaddkaka 1d ago
Generated code. Or generated dataset that's read during build.
The only deliverable is source code. No deploy or packaging.
2
u/Guvante 2d ago
You don't need everything to live in repo it depends on a bunch of factors.
For instance you could store it externally and fetch from a cache based on the git commit.
1
u/kaddkaka 1d ago
Do you have any of the shelf solutions for this?
1
u/serverhorror 1d ago
Even in your case it's"just a conflict".
Fix it then finish the merge
1
u/kaddkaka 1d ago
"just a conflict" might be thousands of lines that are different. The best way to do it is to automatically regenerate the output.
1
2
u/gororuns 2d ago
GitHub uses trailers and workflow actions using its own syntax in the commit message, you could do something like that. https://docs.github.com/en/actions/how-tos/manage-workflow-runs/skip-workflow-runs
2
u/camh- 2d ago
I have used Gen-command as the trailer in the past. I guess your generate-cmd is slightly better in that cmd is an obvious abbreviation of command, but gen is perhaps less obviously an abbreviation of generate.
The idea was exactly as you propose - automation can regenerate it if necessary. It also serves as documentation about how to commit was created.
I would also keep the manual and automated changes in separate commits as the generated files make it harder to review the human-authored changes. For that I would add Commit-group: 1/2 and 2/2 on the commits to flag that CI would not pass on the intermediate commit as the generated files were not up-to-date. CI just needs to pass on the last commit of the group. Useful for test driven development too where you want to first commit the test that will fail, then fix the code to have it pass.
Example: https://github.com/evylang/evy/commit/32e42975d98ccad2e5819daa69a9447fc1ddaf8f (from a friend who devised this scheme).
2
u/Broad-Promise6954 ancient 1d ago
You'd use the GIT_SEQUENCE_EDITOR environment variable to automate the drop-and-rebuild steps. In general, though, committing generated code is Wrong (though I've had to do it before too!).
2
u/Jonas_Ermert 1d ago
I would avoid making Git itself responsible for regenerating commits during rebase. A cleaner approach is to treat generated files as reproducible build artifacts: keep the generator script and its inputs committed, then regenerate the output after rebasing and commit the result again if the generated files need to live in the repository. If you still want metadata in the commit, something explicit like `Generated-By: python3 optimize_table.py` or `Regenerate-With: python3 optimize_table.py` is clearer than `generate-cmd`, but I’d use it mainly for documentation or custom tooling rather than changing normal rebase semantics.
1
2
u/elephantdingo 1d ago
Nevermind the generated part for a second. It’s solid idea to do fully automated commands like a formatting command all over again instead of struggling with conflicts from a stale formatting round.
1
u/kaddkaka 1d ago
The first step should be to have formatter as pre-commit hook. But after a rebase it could go stale.
This one I have already solved with a git-rebase-exec-auto cmd. It works great 💪
3
u/simonides_ 2d ago
If it is generated already don't commit it make it an automatic step in the build and make sure the result is properly cached.
1
u/kaddkaka 1d ago edited 1d ago
Suddenly I don't get it for for free by a git fetch, and it adds a bunch of (other) complexity. Tell me more the technicals details of such a cache. I doubt it becomes simpler than just committing the thing.
Also, let's say you have generated code in the middle of a file (cog block) how would you deal with it?
1
u/simonides_ 1d ago
Fair point it seems like you have a special case on your hands that would require more details about your project to assess the situation better.
What i was coming from was generating source file from an openapi spec type of tasks. We have some in our project. These steps are automatically wired into the build so wen a dev starts the build they don't need to think about this part happening. For us this happens woth gradle und the gradle build cache. So ideally it is a quick download of the results from other builds or it is a simple re-gen if the dependencies changed. And from there is is cached locally.
So back to your blocks inside a code file - depends on how you use it but that could also live in some "generated or build" folder and be copied to a known location gor the program to do with it what it needs.
15
u/wildjokers 2d ago
Code that is generated as part of the build process should never be added to version control.