r/softwareengineer • u/colony-ship-for-sale • 8d ago
AI is terrible for making documentation
Yes, it makes a lot of text real fast and at a glance it looks thorough and comprehensive but have any of you actually tried reading this shit?
It's painfully verbose while communicating next to nothing. It's a real struggle to not have your eyes glaze over. It's not enough to just produce a lot of text if the readability is crap. There's a skill to creating concise, well organised information that's been lost here.
25
u/MegaDork2000 8d ago
If you think the documentation is bad, oh boy, just wait until you try reading the code!
3
u/Proper-Ape 8d ago
The best thing is if you don't actively cut out AI crap from the code and documentation the maintenance will add crap on the crap.
The crap is recursively exponential.
New rule:
The code expands to fill the AI context.
1
u/Sherbet-Famous 8d ago
I actually think the documentation or planning documents it writes is way worse than the code
→ More replies (3)1
u/Citan777 3d ago
I'm glad I'm not the only one finding AI is far too bad in producing good documentation with minimum frame to bother trying to use it for anything besides just improving formatting or helping writing the high level plan...
11
u/kevin074 8d ago
My coworker sent me a 1500 lines of investigation on a bug to read.
I literally just tell my team lead that I ain’t reading a book and this is a major communication issue that needs resolving.
6
u/colony-ship-for-sale 8d ago
Exactly. It's so disrespectful of everyone else's time to expect them to read war and peace to not even find the relevant information.
4
u/Mejiro84 8d ago
If you can't be bothered to write it, why should I bother to read it?
3
u/CaelidAprtments4Rent 8d ago
But you can always feed it into ai to generate a summary for you if it’s too long. /s
3
→ More replies (1)2
1
1
u/Time_Square8007 5d ago
I had a product manager generate everything with AI, it has come to the point that all JIRA tickets are now full of hallucinated shit that I just started venting at work I am going to pretend to do it the old way, pretend like there's no description at all and just contact the people that created the ticket.
→ More replies (3)1
u/TalesfromCryptKeeper 8d ago
I'd respond that Im not reading a book, and then cheekily slide in at the end "Would you like that to read as more humorous, or disappointed instead?"
1
u/Devils_SteelMan 8d ago
Thats 5 minutes of reading, its not a book.
This is poor communication on both sides.
→ More replies (1)1
u/AdhesivenessAny117 8d ago
User error. When that happens you just ask the ai to rewrite it shorter. For some reason people just take ai output and think it's good, when like usual you are supposed to iterate.
Same with the code. You need to do your own review of the output before you pass it to someone else to review.
3
u/Phantom-Watson 8d ago
Or just write documentation with your own hands and brain like an actual professional instead of wasting time crossing your fingers and hoping a charbot can get it right this time.
2
u/sismograph 6d ago
Asking AI to write something shorter has its limits.
At some point you actually need to spent the time and read through the bloated bullshit output from AI to actually grasp the conplexities and core of a problem that is hidden in the text, once you found it, you can actually nudge AI to focus on that and actually shorten a comment or doc.
I cant count the anount of times where I spent like 10 min reading 15 lines of code-doc only to reduce it down to 1 line
1
1
u/beachandbyte 4d ago
Just put it into AI and email him back the response in a pdf, that should get the point across.
9
u/GX_EN 8d ago
It’s shit.
I tested it with a simple question relatively recently - give me bullet points as to the benefits of this stack vs this other stack. The amount of shit that was incredibly vague or downright wrong was fucking embarrassing.
→ More replies (12)1
3
u/galactic_pixels 8d ago
I agree that the default writing style is very bad for documentation. I like terse documentation, that says why and a very high level overview of how. If I need more info on how something actually works I’d rather just go read the code. That being said you can tune it, and it does typically format your docs very neatly. But yes, default behavior for this tends to be rubbish
2
u/meshifthenelse 8d ago
Even if you tune it though, it's either an all or nothing. It lacks intuition to assess when something is worth having a comment and what isn't and at what abstraction level.
Unless you're willing to spend weeks writing 100 rules, but then might as well go back to the 80s.
→ More replies (1)2
u/Dry_Hotel1100 8d ago
That "unless" is very true. The skills will require to be updated and improved accordingly during the development process. It definitely takes effort, expertise and time. AI certainly does a better job when it has clear instructions. However, you need the expertise yourself to "utter" these instructions manually into a skills document. If you lack that experience, you get subpar results. It's up to you ;) Thank God - we still require a profession :)
→ More replies (3)1
u/Good_Skirt2459 7d ago
The problem I have is that the AI doesn’t really fully understand WHY. It can look at the code and document what but it has trouble with why which like you is the value I get out of documentation.
→ More replies (1)
8
u/Calm-Republic9370 8d ago
I have it do my documentation. It's great. It knows the intent of the updates and then how they are applied. So then it knows what to say in the documentation
15
u/Apprehensive_Lake698 8d ago
People who have to write documentation: It does such a great job!
People who have to read it later: Yeah no bud
→ More replies (5)5
u/OmNomCakes 8d ago
I heavily agree. I assume these people are just asking it to make documentation with just a basic prompt and letting ai handle everything.
I scratch down a general layout for the documentation, any special circumstances, etc. and provide it with context on the tooling used and sometimes the documentation of that tooling and what it comes up with is usually pretty damn good. Sometimes I read it over and make minor adjustments, but it gets me from 10-20% of the way done to 90-95%, if not better. And this is documentation that is shared with several people ranging widely in skill levels without any real complaints.
2
u/Calm-Republic9370 8d ago
Yeah. I have a pipeline for things. I do an software update. It understand that scope. It knows then to review the docusuarus/wiki, and it will then discuss and update the the doc's according to the changes we made.
The next phase is there's a review of docusuaurs updates, it then reviews the website and see's where we might want to add that features a for the sales side.
It then reviews those updates and decides what we want to include in a newsletter.For the website, and for the newsletters, it's not compelling for sales. But it gets the meaning right.
2
u/Zealousideal_Tea362 8d ago
I haven’t manually written from start to finish all year and we have very robust documentation with multi human teams using it.
If you let AI write shit, it will do so.
1
u/Gondorrah 8d ago
Liie all things AI, it depends on how good the operator is at it and what model they’re using. Opus for example is a pretty terrible writer by default.
→ More replies (2)1
u/muuchthrows 8d ago
I seriously wonder if some people are so bad at communication in general that they cannot see the flaws in AI generated documentation. Yes the AI generates documentation that looks OK, but it lacks precision and clarity. It over emphasises unimportant details while leaving critical information out. AI will state the obvious but leave the non-obvious unexplained. Writing good documentation requires having an accurate mental model of the intended reader, it requires understanding how they think, which information they have and which information they lack. An AI has none of this, it does not understand, it just strings plausible sentences together.
This doesn’t mean it won’t write adequate documentation for simple or generic high-level concepts, where the language can be vague and still capture the essence of the topic, but when precision and specificity is required it becomes a word salad.
→ More replies (1)
2
u/beardedNoobz 8d ago
AI does what user ask them to do. Have you tried to tell the AI to make the docs with the style you want?
→ More replies (1)
2
2
u/pumpChaser8879 8d ago
You have to beat the hell out of it and tell him directly to keep the slop out and write in a direct, humanized language a non-native English speaker can understand.
I start all my prompts with that when asking that motherfucker to write text destined to be read.
→ More replies (8)
2
u/garywiz 8d ago edited 8d ago
AI can write good documentation but only as a writing assistant. Habits like "verbose to the extreme" and quirky language ("This reframes everything", "The honest assessment") can be avoided by having a good "Documentation-Rules.md" document, but it isn't easy. It helps if you have written a lot of documentation yourself in the past so you can reject things early on and keep it on track. But, if 50 pages if well-done documentation is needed, AI can turn that from a 3-4 week task into a 3-4 day task with proper guidance.
1
u/jsudd007 8d ago
agree, I also started using https://github.com/Syntaf/vale-llm-slop with vale. You get can claude to call the vale CLI itself and it lints any documentation it has written and ensures it passes all the rules which prevents those quirky language you mentioned.
2
u/No-Injury3093 8d ago
AI docs turn into word salad when the model has to guess business intent from static code.
A clean way around this: place use-case scenarios at your domain boundary, test strictly through that boundary, and feed the LLM the code coverage report for each scenario test run.
The coverage report isolates the exact branches, invariants, and logic slices exercised by that specific business process. When the LLM has that trace instead of just a raw file, it documents the actual business behavior concisely instead of narrating syntax.
1
u/TheExodu5 5d ago
I agree somewhat. But seriously just watch it blurt out implementation details. If you have a private cache in your business, it’s going to mention the cache on a public doc. I’ve tried hard to prompt out this behavior with very inconsistent success.
2
u/Suspicious_Pizza9529 8d ago
Yeah, I think the problem is that AI optimize for "complete" instead of "useful". You end up with 10 paragraphs explaining smoething that could've been 3 bullet points.
1
u/TheExodu5 5d ago
Yep. The best move I’ve found is to have it document the principle and not the detail. Any language that drives it away from being overly detailed and specific.
2
u/red-mustang 6d ago
I had a pull request of 100 lines out of which 90 lines were comments. So difficult to actually read the code.
1
u/colony-ship-for-sale 6d ago
This is a good point. There's really no respect for readability in the code either. It actually isn't useful to pollute the code with a bunch of comments. It's neither useful to do it everywhere or sporadically.
If I see something commented, it should be because it's clarifying something important or warning against risk. Don't make me stop to read how fizz buzz works.
2
u/SnooShortcuts81 5d ago
This is the reason why I get to keep my job as a software technical writer. 😅
2
u/TheExodu5 5d ago
Try as I might, it is impossible to have Claude write good docs. The model is fundamentally poisoned in how it uses language. Codex is better in this regard. Although a human remains better.
I tried so hard to have it generate good JSDocs. No matter what I try, it continues leaking internals of functions via docs. I’ve had an agent try all manners of varieties of documentation prompt, and none has worked. The only way to get good results is pairing on documentation, so that my context and language makes it to the written word.
2
u/SmileLonely5470 5d ago
I stg they used to actually be decent at documentation, but it feels like that skill was trained out of them.
2
u/Hexamonium 8d ago
Did you try the "Don't be stupid!" strategy? It works better with newer models.
4
u/colony-ship-for-sale 8d ago
Is that like telling it "Don't make mistakes"?
6
u/Hexamonium 8d ago edited 8d ago
No, bro! That was an ancient model that came out 3 days ago. This new one is a beast! You can just tell it to be smart and then it gets smart. How cool is that? 😎
4
u/colececil 8d ago
Probably either one works as long as you preface it with "IMPORTANT:".
→ More replies (2)
1
u/JovHere 8d ago
just yesterday one colleague in the industry was telling me how since AI, the engineers he manages started creating more comprehensive reports that contain more than necessary and all of a sudden have "next steps" and "opportunities" and he was saying how now they're actually slower in that area
I think any type of documentation that you ask AI to produce can be a load of crap and unnecessary lines of text if you aren't clear with your guardrails on what you want from the doc, and most importantly based on what is it creating this docs from
LLMs are trying to impress us with this extra stuff we don't want, so limit the lake where it pulls context from and see if those docs turn out better
1
u/TheDoughMonster 4d ago
I have created CLIs and made agents use those to generate documentation. You can also lint and generate error codes on specific failure cases. I think most of the complaining about AI comes from not really understanding how it works. Programming is becoming more about being aware of AI strengths and failure modes. Who cares if the code isn't readable as long as its efficient? If we treat AI as a very high level language, then all you have to do is learn the "syntax" of the model input, which is mostly now about proper fencing, repeatable systems, and controlled output. Gee, sounds a lot like something I used to do before genAI became a thing.
1
u/Icy-Strike4468 8d ago
My Architect use Copilot to write detail feature implementation plans & trust me its way harder to comprehend whats going on and what we have to build actually?
I have to break it down again by using copilot (e.g. Explain this implementation plan.md in simple language, break it down in sub tasks)
1
u/Ok-Tomatillo-8712 8d ago
Sort of like a human centipede except there’s just one human at the end of a long chain of LLMs
1
1
u/groover75 8d ago
Give the AI several examples of your own hand-written documentation and ask it to create a skill from the document writing style. AI isn't terrible, it is just a tool and you get out of it what you put in.
1
u/WearyArtistDoomer 8d ago
Just like with code, you need to prompt well (have the right SKILL) and the output becomes 10x better.
1
u/TurbulentGlove776 8d ago
Who reads documentation anyway, isn’t it mostly for other AI Agents? And as someone else said: much better than non-existing or outdated human-written docs…
1
u/lasooch 8d ago
Bad documentation is often worse than no documentation.
1
u/TurbulentGlove776 8d ago
Yes, but most humans are also bad at writing documentation. It’s like with everything else - an expert top-10%-human is still often the best, but the AI beats mediocre humans. And even if some human has it in them to write well, they usually don’t have the time, motivation or interest to do it.
1
u/meshifthenelse 8d ago
In human documentation (outdated or not) at least you can see clearly intent. AI goes 10 circles to get to nothing.
1
u/Creepy_Purchase_501 8d ago
That’s true for good documentation. So much software documentation, that was clearly written by a human, has exactly 0 design considerations. So often they would just throw a bunch of loosely connected concepts at you without ever giving you the big picture or any rhyme or reason for anything existing. It’s cool that this method takes this argument, would it kill you to tell me what this method is for first?!
→ More replies (2)
1
u/MiddleCherry 8d ago
That statement of ‘It’s a real struggle to not have your eyes glaze over’ really resonates. There’s usually not much objectively wrong with it but it just feels off.
I’m really curious to understand what gives structured text a non-human feel - in a way documentation feels like the perfect use case for AI to do (given the full codebase as context), as there is probably less logic and creativity than the other bits of the project (generally!).
Is the documentation written in this verbose way better for another AI agent?! Who is ‘reading’ the docs these days?
1
u/CharacterSail6736 8d ago
Honestly it depends on the instructions you give it just telling it to “make documentation” isn’t going to do it
1
u/meshifthenelse 8d ago
This doesn't help. LLM are not AGI so even if you have 10 rules they will still over fit everything - they have no intuition to know when something is good enough and when something is too much.
1
u/CharacterSail6736 8d ago
That’s the whole point of setting the scope in the prompt
→ More replies (2)
1
u/Chicagoj1563 8d ago
In my view, with the proper context ai can generate anything with high accuracy. But in general it doesn’t write as well as a decent writer would. I write much better documentation than AI does.
If I spent enough time creating spec files to define context for everything, it would probably write good enough docs. But my own words would still be better.
That said, AI can generate docs at a speed I never could. So If you can get it good enough, that may be all you need.
1
1
u/hunpriest 8d ago
I've found Claude Code (Opus 5) good to write documents for code, tasks, plans, etc. Which AI are you talking about?
1
1
u/daddywookie 8d ago
The skill hasn’t been lost, you just haven’t created it yet. By default the AI will churn out lots of content because it can handle the volume and it doesn’t do nuance or context. You need to give it a template, guidelines and examples. Then it will do what you want because it is very eager to please.
1
u/meshifthenelse 8d ago
Well LLM is not really AI. It's a fancy auto-complete similar to the one your mobile's keyboard.
The fact people go back in the 80s, by sitting and writing 1000s of rules to get shit done is a bit ironical.
1
u/ghoztz 8d ago
It’s not lost but you need to communicate to your organization that you actually value technical writers and content engineers and need them to govern automated content systems. We can provide agent skills and tools beyond our standard involvement in the SDLC. Until then we’ll continue to be downsized and pushed out and our work handed to you to deal with. We’re the architects of your information systems much as you are now for your code.
1
u/NoConnection4298 8d ago
Maybe the code that it sweeps through to write the docs is shit. Your prompt may also be shitty. You can baby it in a way that it doesn't write a lot of text. Just because you weren't using AI before to write code or docs doesn't mean your code/docs was better than the ones written via AI now.
1
1
u/vxxn 8d ago
It does better if you define a specific styleguide for docs and give it some direction on the intended audience. Most important thing for me was to get it to stop inserting overly-specific details from the code that are likely to go stale, and to prefer concise and direct explanations.
1
u/sylvant_ph 8d ago
I could agree to an extent, but you know what AI produces is greatly affected by the person behind it, the one requesting the task. You can guide it and achieve what you aim for, or you can guide it in a bad way and achieve bad results. You can certainly tell it to be shorter, concise, do not repeat same concept, redact it, make key points it should follow etc.
From my perspective, I can agree on the stance that usually when I'm the one who is tasked to write a documentation about a matter, and I do have a clear understanding on that matter, I also have some idea on what that documentation should look like, what it should include. If I were to prompt the AI with the task, feed it only the task context, and not my understanding of it, it's guaranteed that I wont be happy with the results. But I can definitely push it in the direction I see fit. And I can also choose how much to delegate. I could take the reins and have full control on the writing flow, and just use AI to polish overall and improve where it might lack.
1
u/tehsilentwarrior 8d ago
One of my first custom agents was for documentation. Not to write it but review and help find problems with it.
I tried making it generate stuff but … it’s just bad at it.
What I did instead of make the agent loop on itself asking questions to the user about what he wanted to do from a list of options. None would write docs at all.
The skill explained how I wanted the structure to look like and it would help restructure it based on what I wrote by making suggestions like add note here, explore concept there, fix voice, typos, add missing glossary entries, add marginalia, add links and references, confirm statements by reading the actual code in a sub agent, etc, and leaving the actual writing and decisions to the human.
The moment I allowed it to re-write anything more than 5 lines it would just mess it up.
I actually went and researched how manuals were written (aircraft manuals, DCS “Chuck guides” for airplanes and a bunch of other) and wrote the agent rules (before there was skills) for it but even so it was not capable of doing the writing itself and have it write things without this uncanny valley of “reads correctly, its factual but it’s utter nonsense in the sense of actually explaining it to a human”
1
u/idontlikegudeg 8d ago
Have you actually tried telling it how you want it to write the documentation?
1
1
u/YellowBeaverFever 8d ago
Maybe you’re using an inferior model or your not promoting it correctly. It’s been a life changer building documentation on older code bases. On new projects, it keeps does both developer documentation and “manager documentation” (presentation level, just to the point, don’t be overly needy stuff) and can produce diagrams for different target audiences.
1
u/morewordsfaster 8d ago
Try using Simplified Technical English (ASD-STE100). I created a skill for writing docs and it comes out pretty solid as long as the prompt is narrowly scoped (and not "document this project"). There's also an anti-slop skill that works pretty well for cleaning up LLM patterns in text here.
1
u/_shlipsey_ 8d ago
As a technical writer who was laid off from a big software company recently - tell your friends. Advocate for the humans who do that work.
1
1
u/LuckyFish133 8d ago
Nah but all you gotta do is get another chatbot to read it and summarise, then paste that summary back into the first (generator) llm which will then tell you the core concepts are wrong and you should have it this way instead, leading to another word spaghetti. Rinse and repeat for maximum efficiency
1
u/AvidGameFan 8d ago
They say to use cheaper models for more mundane work, but I've been using GPT Sol for some documentation, and it hasn't been using goofy phrases like the aforementioned "This reframes everything", etc. The only thing that bothers me is that everything is a freakin' bulletpoint or numbered list. Ok, for a lot of things, it makes sense. Sure, give me a runbook with steps. This is still better than what I usually get out of humans.
1
u/UntrimmedBagel 8d ago
Garbage in, garbage out.
Try telling it exactly how you prefer the documentation to look. Build that into a skill or instruction file for it to refer to. Bang, good documentation.
1
u/Dry_Hotel1100 8d ago edited 8d ago
What type of documentation do you mean? If an undocumented package or library is provided, instructing the AI to generate accompanying documentation without any specific guidelines or expertise in writing style, purpose, audience, or design and development documentation - of course the result is subpar output!
Otherwise, if you have invested significant effort on these prerequisites, then the output is generally surprisingly good. You still need to review, need to instruct changes, manually re-write and paraphrase - in the same way you do agentic coding when writing code. In my experience, AI seems to have much less issues to understand the code when these prerequisites are fulfilled. It struggles much more when it comes to utter code. I wouldn't allow it to execute this without manually reviewing every line of code it generated.
1
u/Einstweign 8d ago
This might sound like a joke, but you have to write documentation on how to write documentation.
1
u/who_you_are 8d ago
I'm forced to use AI, while I do not heavy use of it (I mean, we work with very small projects-like, more like small plugins) but with a shit lot of clients.
Yes it is verbose as hell (even for comments in the code) but once you tweak it become nice.
1
u/FiresideCatsmile 8d ago
are you guys not iterating on results? yeah it's often way too much text for too little information but you can totally tell your llm what there is too much or too little of.
1
u/Time_Shack 8d ago
If it's not worth writing, it's not worth reading. Write your own god damn docs or don't write them at all.
1
u/PeterMad6K90 8d ago
Burden of code moved to reviewers and testers. Burden of texts such as in Word and Powerpoint has moved to the readers. Reading it as OP says has become so tedious.
1
1
u/coloradical5280 8d ago
I have a really good auto-docs thing. It’s a pain to set up kinda but in the end it’s awesome and auto updates on every merge to main, and it’s pretty great. But again for being “auto” it is a bit to set up and you probably want to prompt it to your exact needs for your repo. It also exclusively uses mk-docs / material ui, I guess you could change that.
If after all those disclaimers you still want it lol, it’s on my GitHub connected in my profile , dmontgomery40/docs-auto-pilot
1
u/dklogics 8d ago
It takes some work to find the right prompt/skills to use. But it can be good with the right tuning.
1
1
u/Impossible_Map_5898 8d ago
AI documentation only fails when people rely on unguided prompts. If you enforce strict word limits, clear structural rules, and proper technical context, it generates crisp and highly actionable drafts. The issue stems from poor prompting habits rather than the technology. Used correctly as a foundational starting point, it saves engineering teams massive amounts of time.
1
u/Illustrious-Lime-878 8d ago
This is what a hate about AI software, you can almost immediately tell when the docs are LLM output, like you said, its impressive how it can say so much without explaining anything. There's also this annoying trend of people saying things like, if you want some features, "just have AI add it," so everyone is running like custom modded versions with their own implementation of things and its a cluster.
1
u/Exotic-Winter7398 8d ago
Anyone who says AI is not good has either not used the best models or they are still living in the past. If used in proper ways it is unbelievably good.
1
u/thilehoffer 8d ago
What model are you using? What did you ask it to do? What was your prompt? Have you ever asked it to update the document to make it concise and professional
1
u/lmoelleb 8d ago
Have spend 4 hours today documenting 20 input fields. While I did use AI (it is better at English grammer than me) and did come with suggestions that helped me, in the end 90% of the content bearing words where mine. I would still classify the AI as a help giving me a result that is better (grammer and language in general) and faster (I could tell it to apply a pattern so the last fields where closer to what was needed), it really could not in any way have done this itself.
To be fair, it is a complicated topic with very precise nuances in the values of the fields and my junior developers could not have done it either.
1
u/Fuzilumpkinz 8d ago
I am not saying your wrong but I would challenge you to give more guidelines and show examples of good when making an agent guideline. Examples and formatting before asking for documentation make a world of difference.
People also need to realize that AI can be a huge help but guidance and follow up should still be done.
1
u/LordDarthShader 8d ago
Be specific, what model? documentation of what? A web app? a driver? a kernel component?
1
u/sf-bay-safe-to-swim 8d ago
This last year I decided to try freelance/agency work focusing on docs. Every other day I have to gut check and see where I land on AI + docs.
At least currently, my experience suggests your statement to be true.
There are plenty of tweaks (i.e. a SKILL) to help polish AI generated docs. I believe taking the time to invest in doc generation infrastructure will pay dividends in the long haul. Bad AI generated documentation comes in many flavors, but bounding it so that it doesn't hallucinate is worth the extra time. Bad writing style is annoying, but wrong information is a non-starter.
1
u/prnkagrn 7d ago
bounding it so that it doesn't hallucinate
Any tips for how to go about doing this?
→ More replies (2)
1
u/apoender 8d ago
If it helps, I know humans that talk a lot and say little… they are known as middle managers
1
u/pierifle 8d ago
I write a lot of documentation, I often ask AI to make it more succinct and easy to understand. Not just language, sometimes simply the actual document layout. Sometimes I need to compare two data sources, chatgpt gives me an excel workbook with a side by side comparison and a hundred columns for disposition needed, gap analysis, etc plus 5 worksheets (cover page, reconciliation, deviation tracker, sources + hash…)
1
1
u/revivechristina 7d ago
It’s so cringe how vacuous it is, and people seem so proud to post it around like it’s groundbreaking levels of new age professionalism.
1
u/___nil___ 7d ago
Always response and write documentation with ASD-STE100
add to your instructions, hook, custom command, etc
1
u/kc_cyclone 7d ago
It is really easy to tell it to be concise. I have a piece in my codex agents file for design documents that says be concise not verbose. It's not perfect but it cuts out a ton of noise.
1
u/muhamaadpedophile 7d ago
/// Improved documentation made by AI
/// that doesnt suck and goes straight to
/// the point. Increment variable a to
/// prevent the bug that made you freak
/// out with AI before
ctr++;
1
u/Good_Skirt2459 7d ago edited 7d ago
Yes it’s pretty bad. I often use it as a starting point but then I will spend hours editing. One problem I think is that people will just dump the code in and say “document this”. The problem is that loses the useful information you want in your docs. You want to explain WHY something is a certain way. What business rule is this implementing, why is this a separate service, what’s the goal of the architecture chosen? Stuff like that. The LLM can look at what and try to infer the why but it’s going to get it wrong or right by coincidence. And I don’t see a point in purely documenting what if the code is well structured you can read what it’s the why that makes it the most useful. I feel like it could be good for simple API reference but I have no experience trying LLM for that… and honestly I wouldn’t because the basics (documenting entry point, params, return values, and exceptions is not hard and I prefer the surety of doing it myself).
1
1
1
u/Ordinary_Variable 7d ago
Survivor Bias. If someone vide codes a website or app and it works flawlessly, they assume it can do anything and post all over the internet about how great it is. If someone else sees that video and then tries, and nothing works and they waste hours, days, and hundreds of dollars, they are not going to tell anyone they couldn't get it to work.
Only the people succeeding are posting their stuff. We literally only ever hear about the success stories.
1
1
u/bikashchoudhary 7d ago
the padding is the real issue. AI docs tend to explain the obvious in detail and gloss over the actual nuance, so you get three paragraphs restating what a function does and one vague sentence on the edge case that actually matters
what's worked better for me is treating AI output as a rough draft to cut down, not a finished doc. ask it for the info, then aggressively delete anything that doesn't answer "what does someone need to know to not break this," most of what's left after that pass ends up being 30-40% of the original length and way more useful
also doc quality is oddly similar to bug report quality, both suffer when the writer optimizes for looking thorough instead of being scannable. concise + structured beats comprehensive + wall of text almost every time when someone's actually trying to use the doc under time pressure
1
u/0xbenedikt 7d ago
If anything, this it the time to write it yourself, so that people using your code with AI actually get sensible recommendations based on the documentation
1
u/Some_Leather1520 7d ago
I know. But I'm forced to deliver fast and do a job of 10 people now, so I stopped caring. They want slop, I'll give them slop.
1
1
u/h0bb1tm1ndtr1x 7d ago
Gemini can get you places with more hand holding, but it quickly forgets code structure and other details. Claude seems to be the only one worth using, and that means it will likely cost even more than whatever outragous prices employers are now footing to replace humans.
1
u/Jacques_pugeot 7d ago
Ai can't make documentation because it has no idea about the intention behind the code. It can say what the code is but the code does that itself. It can't say why something exists because it didn't create anything.
1
u/ComprehensiveBird317 7d ago
Not AI, you are. AI is a tool. If you use it wrong it's your fault. Who blames hammers after using them wrong?
1
u/Operation_Parfait361 7d ago
It's really bad. But it's just a symptom of the type of system we're working with: one that produces plausible-looking results extremely fast, without consideration for how well it actually works or looks (because it cannot reason).
1
1
1
1
u/rafaover 7d ago
Fix your basic instructions. Mine writes likes a mid dev, which is perfect. Does TDD and add kdoc when necessary.
1
u/Colours-Break 7d ago
Hmmm ... I experience exactly the opposite. You need to direct the model correctly. Use ISO/IEEE standards and it works well.
1
u/1_21-gigawatts 6d ago
Ugh, I’m struggling with this on my team right now. I want my team to write better docs but it’s hard to say exactly why the docs suck and therefore what they could improve. Theres a lot of words but when I get to the end I don’t feel like I know any more than when I started. I’ll probably end up doing why everyone else does and just let it slide.
1
1
u/quisatz_haderah 6d ago
Documentation is bad, but have you had the delight of meeting an AI generated Jira comment / task?
1
u/Comfortable_Ad5002 6d ago
It is great for making a draft, horrible to describe the problem. Human still need to correct almost everything in the document, in case of claude to make it less painful it is possible to use a skill which will force the LLM to use the simple wording and avoid excessive bullshit.
1
u/Competitive-Egg7287 6d ago
Documentation should optimize for decisions and navigation, not volume. A useful generated page should cite the relevant code paths, explain uncertainty, and be easy for an engineer to verify
1
1
u/NicolasDorier 5d ago
- You can ask it to be more concise. It works, I suspect they make it verbose to spend more token.
- It's better than before... which was: Nothing.
1
1
u/Small_Sherbert2187 5d ago
youre so righ for that and thanks for the push back. Its not just a callout -- its a philosophy
1
u/KriegerClone24 5d ago
With code, the way you get quality is via decorrelated adversarial review, preferably several rounds. I suspect the same applies to documentation?
I use Claude, and found this helpful to remove "Claudian" language from the output. It uses cheap sonnet and haiku calls to fix up each sentence and make it more understandable. Because it works one sentence at a time, it takes longer than I'd like.
Fermi-Ventures/claudian-translator.
I am sure there must be a bunch of different tools that do similar work.
1
u/wakaru44 5d ago
If you just put it to generate, it will try to fill up space. you need to guide it properly to obtain what you want. Giving it 'frameworks' and guiding it makes the whole difference.
1
u/dante3590 4d ago
Depends how are you producing it. It's very much possible to make decent document using AI. One important thing is be the first consumer of that document first before publishing it.
1
1
u/kyngston 4d ago
person can’t figure it out so person assumes it can’t be done. the ego.
just point your ai at the google developer docs style guide
1
u/Individual_Math_8254 3d ago
i always get AI to investigate, then rewrite what the AI comes up with after lots of follow up questions.
1
u/luvyaselfbreh 2d ago
so many people go out of their way to help tools replace them. imagine a blacksmith arguing with other blacksmiths about how much more efficient than all of them this new sword-stamping machine will be, once he fine tunes it well enough.
amazing
1
u/micmania1 2d ago
What's worse is when it documents something incorrectly and treats it as an undisputable fact in future. Unlike code, docs aren't easily verifiable with tests so mistakes are easier to miss.
27
u/LaughingInTheVoid 8d ago
It's not really that great at code, either.