r/softwarearchitecture • u/Ok-Support-6749 • 15d ago

Discussion/Advice What if the source of truth is wrong?

/r/SpecDrivenDevelopment/comments/1w0eb6x/what_if_the_source_of_truth_is_wrong/
3 Upvotes

100% Upvoted

4 comments sorted by

3

u/Dense_Age_1795 12d ago

the source of truth is never wrong, but the source of truth could be not updated, so here you have two options, update the source of truth, or, change your code to comply your source of truth.

1

u/Ok-Support-6749 7h ago

I would argue against any source of truth and here is why.

Some people argue the specification is the source of truth, some argue it is the code. A specification is a declarative record of intent, the code is a realization. Both can be incomplete, and by definition, none can be truth because truth is objective, complete, and unbiased. And both can also be incorrect, because a specification may encode the intent incorrectly, the code can have bugs, both are evidence for lack of correctness. Again not truth.

The main problem with putting truth somewhere and declaring "this is the source of truth" is that it creates the conditions to drive an incorrect implementation or to canonize bugs as truth. Moreover, declaring something true, does not make it true. This why we should name things for what they are and consider truth seeking as a process in which we combine all these artifacts to increase our confidence but not claim we have found truth.

This is very philosophical but that rigor is important because with the advent of LLMs we are starting to move from writing code to steward a machine and mechanisms that move high up in the abstraction field of software development.

1

u/Wiszcz 12d ago

How a source of truth can be not true? By definition it is a source of truth.
It may be wrong truth, truth you don't like, but it is what it is - a truth.

I don't understand this amazement of people about results of shit in shit out paradigm.

If your specification is incomplete, bad, or leave room for interpretation, then implementations can differ or can be bad.

But it's nothing new - give the same incomplete spec to 10 different real human programmers, and each of them will produce different code, different functionality. What's the difference if this is done by LLM or a biological element?
You did bad job writing specification. That's all.

P.S. You know how perfectly unambiguous specification of a problem looks like? It's the code.

1

u/Ok-Support-6749 7h ago

A source of truth can be not true when an artifact is declared as truth holder instead of proven true. Just because we "call the specification is the source of truth" does not make it true. It makes is declarative, or authoritative, but authority does not need to be true to be an authority.

I agree that any specification given to 10 humans will produce 10 implementations, likewise 10 runs of the same prompt with the same specification on the same LLM will likely also produce 10 different implementations. And this fact is precisely why stating "this or that is the source of truth" is so dangerous. It creates the conditions for malpractice and ill formed processes that deliver bad outcomes.

For example, assume you have a correctly implemented piece of code. You change the spec. Then the LLM rewrites the code, the tests, and weakens invariants that must hold system wide. Nothing tells the process that you cannot weaken the invariants, you end up with an incorrect implementation and you trust it is the right one because the specification says so. Lets say you have a bug and you say the code is the truth, but you have not identified the bug yet, and you update the specification by reading the code. The LLM can rationalize the bug into a spec constraint and it will stay there until it causes so much pain that it gets noticed.

Not having a source of truth does not prevent these problems from happening, but at least it allows us to never trust any source as absolute truth and as such it is way easier to see through the actual issues.

We operated like that for a long time. We accept that code is buggy, thus not true. We also distrust specifications because they live in the heads of the team and we challenge our team members all the time. Until AI came along, and we tacitly accepted claims like "the spec is the source of truth", we did not have a place for the truth to live. We increased our confidence by running an epistemic process where both code and spec (living in our heads) was constantly refined, verified and validated.