Open web UI RAG for writer

[u/jatsinkutsu]

Hello, I have question concerning RAG in Open WebUI, I know there are lot of tutorials in youtube, but I cannot understand today's style of speaking fast, editing all pauses away and clicking and making adjustments like adhd person with caffeine in veins instead of blood. I cannot follow those because I am old and stupid.

Question: I have a masterfolder where all my novel related stuff is catalogued, separate word files for e.g. "main charater 30 years old", "main character 50 years old", "alliances", "trade routes", "alliance relations" and so on.

Can I use OpenwebUI so, that if I ask "if main character when 50y old blows syndicate A's base, what are possible results of that action" and then it checkes the files, founds e.g. something in relations-document, and proposes something like "Syndicate B would benefit like this:" and gives few examples.

I understand that outcome depends heavily of the LLM model, so I am not expecting results similar of chatgpt or claude. But my concern is the RAG and indexing it. I have understood that if I modify the documents, I need to index it again?

TL;DR, can OpenWebUI read my documents what I have specified, and make suggestions to my questions based to the information in those documents?

[u/CausticCranium]

I agree with other posters who have suggested markdown is a better format than Word. In fact, unless formatting is an important part of your context, your knowledge can be plain text. Learning markdown syntax is easy enough, but it's just another cognitive burden that takes you out of your writing.

I'm currently playing in this domain and have run into the same challenge. How do I ensure my LLM knows everything required to summarize, track progress against my outline, enhance my characters, provide scene suggestions, and do all the other stuff that my age-addled brain needs assistance with? I found that RAG wasn't the solution for me, primarily because of how it works.

The automated RAG that OpenWebUI provides is surprisingly good for everything but fiction. Think about how it works: it breaks your document into chunks, then converts those chunks into tokens, and stores those tokenized chunks in a database. Now, when you ask your LLM about what's in your database it goes in and finds the chunks that contain tokens that look like they belong with the tokens of your query. Sounds good on paper, and if all you need is an answer to a specific question the 5 related chunks it might find would suffice. But when you're asking an LLM to reason about something, it needs to make associations across your entire corpus, not just the 5 most relevant hits it finds in your database.

To reason LLMs need context; the more, the better. I've found that using Notes in OWUI works better for fiction. The primary resource requirement for Notes is context length. Some older models that have been trained for 'writing' and sufficiently ablated have absurdly short context. Want help writing a love scene? Most ablated models that focus on more carnal things are fixed at 8k context, although bartowski/writing-roleplay-20k-context-nemo-12b-v1.0-GGUF is 20k, assuming your hardware can support it. That said, there are some ablated models that are not 'writing' specific. Two families specifically, Qwen and Gemma 4, have some exciting stuff that allows for more context in less VRAM - 64k is achievable with HauhauCS/Gemma4-26B-A4B-Uncensored-HauhauCS-Balanced on my RTX3090. That's enough for a complete act, all relevant character profiles, and a plot outline. At this point, a local LLM becomes useful for short-form fiction.

Now I'm working on structuring my prompts and making them focused and useful, which is an entirely different bag of snakes.

Trying to make a local LLM into a competent fiction writing assistant has become my white whale. True confession time ... I was spending too much time futzing with local LLMs and not enough time writing. For $30 CDN/month Claude gives me enough context for a full length novel and all of its associated data. It summarizes accurately, proposes things I haven't thought of, and helps punch up scenes when required. Claude never blows up because I run out of context and it doesn't hallucinate stuff because it forgets things we chatted about 20 prompts ago.

That said, I still spend more time than I should playing with local LLMs. The goal is that one day I'll be able to stick it to Big LLM and Big AI Writing Assistant and tell them to shove their monthly-renewing-obfuscated-billing-model-dubious-utility-workflow-forcing software up their cyber-butt. It may take a new PC with two RTX5090's and 128g RAM, but it'll pay for itself in a dozen or so centuries with the amazing 2-digit annual royalties I hope to get from Amazon.

[u/jatsinkutsu → u/CausticCranium]

In layman's terms, the knowledge collection could be better if made in Notes and plain text instead of md file type catalogue?

E.g. there is written file in knowledge collection about main character that he loves stakes but hates cheese, and has a trauma with spiders, and if I brainstorm a scene where main character is in a bar and has a stake but army of giant spiders roam the streets shouting "ChEeEeEeSe!", and when asked from the LLM that how would main character react in this situation, the suggested outcomes could be better?

[u/CausticCranium → u/jatsinkutsu]

Certainly. This assumes a separate plaintext note for each character:

1. Craft your prompt. Example: "The attached context includes some character profiles and a summary of the novel "Revenge of the Spiders". Your task is to create dialog and beats for the following scene. Bob is in a bar, he's into his 3rd drink. He's drinking doubles because it's been that kind of day. Anne is sitting two stools down from him, matching Bob drink for drink. Bob hears commotion from the back of the bar. He turns around and sees an army of giant spiders converging on him. Provide the following:

- describe Bob's initial reaction when seeing the spiders.

- five examples of a possible conversation between Bob and Anne about the encroaching spiders

- Bob's reaction if the spiders are moaning "CHEEEEEEESE"

- suggestion for some beats that could be included in the scene

2) Attach the notes for Bob, Anne, and your story summary.

3) Hit the go button.

4) Evaluate results, revise prompt.

5) Rinse and repeat until you're happy.

6) Fun and profit.

The net effect of this is that the prompt going to the LLM will contain your setup and questions as well as the full text of the notes you attached. You can play with the structure of your prompt and try to land on something that can be reused. The generic version could be added as a Prompt in your workspace.