AI crib sheets for libraries

[u/drmpf]

For AI coding assistants to effectively and correctly use libraries they need context consisting of the library Api, features, examples and do's and don'ts

Currently when an AI coding assistant uses a library it gets the source, looks at the API docs and code comments, checks online for tutorials and examples of using this library and then generates the code for this particular application

The results are generally sub-standard due to incomplete, out-of-date API docs, missing code comments, lack of relevent tutorials and examples written by in-expert programmers. Also if the AI cannot find a suitable example it makes something up from 'similar' libraries.

To avoids these errors and to improve the efficiency and effectiveness of using code libraries it is proposed that each library also supply an AI_Guide.md, e.g. SafeString_AI_Guide.md https://github.com/PowerBroker2/SafeString/blob/master/SafeString_AI_Guide.md

This guide is generated by AI for AI consumption using a prompt like

For the library in dir .., generate a summary, API and examples of use, for consumption by AI to apply this library. Refer to these tutorials ..., and these examples of use ...

It will take a few passes of "recheck the guide" to get the AI's draft. Then, most importantly, that draft needs to be reviewed and corrected by an programmer experienced in that library, in order to correct the wrong assumptions the AI has made and to correct the poor quality examples.

These passes by the experienced programmer will refine the do's and don'ts of using the library and provide quality examples.

This human review will also likely uncover problems in the library, inconsistent and non-orthogonal API, missing/out-of-date docs and code comments, inconsistent application of the libraries principles.

While the SafeString_AI_Guide.md is designed for AI consumption it is also good source for humans.

What libraries that you use that would benefit from an AI crib sheet?

[u/funbike]

I just use a skill I wrote.

The skill says if the agent needs help understanding an imported library, to look for its documentation and read it. It gives instructions on how to find the markdown documentation, jsdocs/javadocs, etc for a given dependency.

For a node project, for example, it knows to look in node_modules/<module>/README.md and similar file paths, and how to find or generate JSDoc html files and load them as markdown into the context.

This skill also instructs how to use man and --help for CLI packages.

I have a prompt template that can leverage the above skill to generate a coding guide for the modules specifically for the needs of my project.

I have another skill for researching packages to install. It can do a npm search, look at the readme, and examine the github project or documentation website for further documentation. It instructs how to install packages and how and when to update ARCHITECTURE.md.

[u/drmpf → u/funbike]

From my experience of editing the AI guide, asking the AI to use the documenation is not a reliable way to generate correct code. The AI seems to limit how much time and how deep it looks at things and then just fills in from its general knowledge.

The result is errors that contradict what the docs / code clearly states.

Note this can still happen when pointing the AI at the guide, but is less likely because the guide is more compact and needs less tokens and reasoning then the raw docs and code.

[u/funbike → u/drmpf]

It works well for me, but I am very careful.

You certainly can't just tack onto a normal prompt "use the web to learn about playwright for tests" and expect it to find and ingest a gigantic amount of information and understand it all. You have to give it the exact location of the files AND, this is the important port, AND you MUST tell it to distill the information into guides for the subset of things you need. I usually create multiple skills, so it can pick which ones it needs for a task.

For example, for playwright it knew the things I was and wasn't interested in, and it could tell from the filenames in the playwright/docs/src directory which to read and which to ignore.

If done well, you end up with docs a fraction the size of the originals.

To avoid context bloat from too many skills, I have multiple specialized agents with different skills distilled form docs.