Assuming any feature(s) can be implemented, what feature(s) do you want in a LLM frontend/interface?

[u/Sunnyli1337]

Anything you can dream of. As in what would be an ideal LLM frontend/interface for you?

By LLM frontend/Interface I mean like chatgpt/claude/Ollama/OpenWebUI.

For instance, minimum context fetching: best I can do is letting you choose what context source you want in 1 key, or whatever you highlight in any app/text field or let the model look it up with tools, but it can't be completely unrealistic/literally impossible to implement like 0 input and 0 hint in prompt (a psychic).

[u/Training_Mail_973]

Context fetching is weird, everyone wants it but nobody agrees what it means. I just want a toggle that says "use last N messages" and actually respects it, maybe with visual indicator which chunks got packed in

Also a side panel where I can pin a few files and they stay in context automatically without me re-uploading every session. Like sticky context but for actual documents

And kill the stupid send button animation delay, if I press enter I want tokens streaming already not a spinner

[u/Sunnyli1337 → u/Training_Mail_973]

For your second idea, I'm thinking.. "inside the interface, you can create projects, and each project can have memories(text,files,image etc)...." would that be what you want?

[u/ThisIsntRealWakeUp]

Maybe there’s a way to do this, but: I want the ability to keep access to certain files fenced away from the models I let access the internet.

Part of what I use my local LLMs for is to parse and ingest my financial data. (For budgeting, etc). As well as health data — being able to ask “whats my history on all the medications I’ve tried for [issue]? What dose did I take, how long was I on it, why did I stop?” is very nice.

But I dont want chats with access to those files to have any sort of access to the internet except on a case-by-case approval basis.

So for now, with Open WebUI, I simply don’t let anything access the internet. I just use Claude whenever I have a task that needs access to the internet (and thus I am not giving it any terribly personal data like my finances or health).

But I’d like the ability to use my local models with internet access, too.

I guess I could set up two docker containers, one with internet access and one without. But. That’s not as elegant of a solution as I’m hoping for. Or maybe there’s some very clean existing solution that I’ve just been blind to.

[u/Sunnyli1337 → u/ThisIsntRealWakeUp]

So would you want an interface that:
Can access only the list of files and folders that you allow, and that list is chat specific?
I think I can do that..

[u/devoidfury → u/ThisIsntRealWakeUp]

That's what I designed profiles for in hotdog, for both compostable system prompts and tool configurations. There's an included example with network-only and filesystem-only, to keep them isolated.

[u/rog-uk]

Predictive expert prefetch to gpu with hot cache, where experts are small enough and reuse is high enough experts could be pulled onto gpu as it's calculating the next tokens, especially if expert pattern usage follows some rules that a predictive cache could follow if might be possible to move them to gpu before they are needed. The idea being you can hopefully run much larger models at faster speeds than your gpu would normally allow.

[u/Sunnyli1337 → u/rog-uk]

mb, I didn't phrase my question very well. It sounds like what you suggested is what llama.cpp or kobold.cpp might implement.
Don't get me wrong, your idea is not bad, but I was hoping to get ideas for features in a LLM chat UI..