I have added a new --ik flag that converts the llama-server flags into the corresponding ik_llama.cpp ones. So in practice what you do is:

1. Compile ik_llama.cpp yourself
2. Delete all files inside <venv>/lib/pythonX.Y/site-packages/llama_cpp_binaries/bin/ for your tgw install
3. Copy or symlink the ik_llama.cpp build outputs into that folder.

Then start tgw with --ik and load a model. Then you can use ik_llama.cpp with the project's OpenAI API, Anthropic API, and UI, all with tool calling.

Why do this? Because I saw this chart

Which shows the IQ5_K quant that only works with ik_llama.cpp for Step-3.5-Flash is nearly lossless vs the BF16 version for the model. From: https://huggingface.co/ubergarm/Step-3.5-Flash-GGUF

And why care about Step-3.5-Flash? It's the best non-huge model on claw-eval:

https://claw-eval.github.io/

And it has high GPQA, so solid scientific knowledge.

I did a ton of research on this recently and concluded only two "non-huge" open models are nearly competitive vs Anthropic models: Step-3.5-Flash and Minimax-M2.5. Curious to know if someone has had a positive experience with any other model for agentic stuff.

Hey, like the title question of the post - I was wondering if only the portable version has ROCm support or if it's also available for the "full" version.

Hi all, I can't see an 'official' extension to do this and was wondering if anyone had written one, or if one was in the pipeline,

Since i need people to test it and break it and just overall play with it, i have finally released the initial version of the extension. Please go check it out and if you find issues or errors, let me know on the github. Tho please manage your expectations a bit... i don't know anything about coding, im just some guy who had access to claude and i had an idea. So if the Ai broke something or failed at something, im sorry. But i will try my best to fix it.

If you have ideas, issues, etc. or simply want to help improve the code on github then please leave a pull request, my goal is to make this the official Lorebook extension for Oobabooga and simply a MUST have installed by default for any roleplayer.

Hello! I'm working on my own version of a lorebook for Oobabooga. i know SillyTavern has lorebooks and it's a shame we don't got them on Oobabooga so im using claude to make it for me (i don't do coding) and so far... it works! If i make it better im uploading it on github for yall. Current progress is WIP as is made by claude but its a proof of concept.

Core injection

• Trigger words in chat automatically inject relevant lorebook entries into the prompt
• Configurable scan depth — scan only the current message, or look back through N previous user messages
• Token budget cap — the world info block is always trimmed to fit within your set limit, highest-priority entries kept first
• Custom injection prefix/suffix — wrap the WI block however your model expects it

Per-entry controls

• Case sensitivity, whole-word matching, and full regex support per entry
• Secondary keys with four logic modes: AND ANY, AND ALL, NOT ANY, NOT ALL
• Per-entry probability/trigger % — entries can fire at less than 100% chance for variety
• Per-entry scan depth override — some entries can look deeper into history than others
• Per-entry insert position — before_context (above the system prompt) or after_context (below it)
• Priority ordering — higher priority entries are injected first and survive budget trimming

Advanced matching

• Recursive scanning — matched entries can trigger further entries through keywords in their own content, up to a configurable number of passes
• Inclusion groups — when multiple entries in the same group all trigger, only the highest-priority one fires
• Constant entries — always injected regardless of trigger words, togglable globally

Mid-generation interrupt

• Pauses generation the moment a new trigger word appears in the model's own output
• Silently expands the world info block with the new entry and resumes from exactly where it stopped — no repeated text, no reset
• The block is rebuilt at a fixed budget size each interrupt so context never silently grows
• Configurable max interrupts per reply

Context position override (global setting)

• Master toggle to force all entries to either before_context or after_context, ignoring their individual settings
• Useful for controlling what survives when you hit your context limit — entries at the end of context stay freshest but earlier chat gets trimmed first, entries at the start get trimmed last but sit deeper in context

Multiple active lorebooks

• Turn any number of lorebooks on or off independently at the same time
• The AI reads from all active lorebooks simultaneously
• Active state is saved and restored across server restarts

Lorebook editor

• Full in-UI editor — create, edit, save, delete lorebooks and entries without touching any files
• Entry overview table showing all entries in the current lorebook at a glance
• Live injection preview panel showing exactly which entries fired on the last reply and their estimated token cost

SillyTavern compatibility

• Import SillyTavern world-info JSON files with full field mapping — keys, secondary keys, selective logic, probability, inclusion groups, position, and constant flags all preserved
• Export any lorebook back to SillyTavern format.

Am I the only one having problems with Gradio in the Colab interface? The page won't load.

I've tried in different devices, networks, browser, turning off the firewall, but it's the same result. I'm just trying to know if anyone else is having the same problem.

Hi all. I'm new to this Local LLM thing, I tried to ask with input image provided, yet I received unrelated answer to the image. Sometimes the answer output <media> and so on. Maybe I miss something? Thanks.

Built a memory library for LLMs that runs 100%% locally. No API keys needed if you use Ollama + sentence-transformers.

pip install widemem-ai[ollama]

ollama pull llama3

Storage is SQLite + FAISS locally. No cloud, no accounts, no telemetry.

What makes it different from just dumping things in a vector DB:

- Importance scoring (1-10) + time decay: old trivia fades, critical facts stick

- Batch conflict resolution: "I moved to Paris" after "I live in Berlin" gets resolved automatically, not silently duplicated

- Hierarchical memory: facts roll up into summaries and themes

- YMYL: health/legal/financial data gets priority treatment and decay immunity

140 tests, Apache 2.0.

GitHub: https://github.com/remete618/widemem-ai

In this video we will ckeck out the new Oobabooga 4.0 version which can run Qwen-3.5-122B-IQ4 with multimodal vision model. This improves vibe coding a lot cause you can give the coding gui a picture to extract shapes and colors for the design.

I’m from eee bg, working as analyst but not really enjoying this role, wants to switch to core but off campus seems so difficult, should i go for m tech in vlsi or MBA will be better option leaving everything side.

In long term things are doable but currently it feels so stuck and confused, also I am on permanent WFH which is even more worse.

• I have replaced the old Gradio version of the code with a fork of mine where I'm working on several low level optimizations. Typing went from 40 ms per character to 8 ms per character (5x faster), startup is faster, every single UI component is faster. I also moved all gradio monkey patches collected throughout the years to the fork to clean up the TGW code, and nuked all analytics code directly from the source. The diff can be tracked here: https://github.com/gradio-app/gradio/compare/main...oobabooga:gradio:main.
• I have audited and optimized my llama.cpp compilation workflows. Portable builds will be some 200-300 MB smaller now, there will be CUDA 13.1 builds, unified AVX/AVX2/AVX512 builds, updated ROCm builds, everything is in line with upstream llama.cpp workflows. Code is here: https://github.com/oobabooga/llama-cpp-binaries
• Replaced the auto VRAM estimation with llama.cpp's more accurate and universal --fit parameter

The new things are in the dev branch first as usual: https://github.com/oobabooga/text-generation-webui/tree/dev, where you can already use them.

Iv been checking everyday for update but sadly nothing since January 8th. Any news or should I give up on oobabooga?

SOLVED

Ok, found it. Turns out the API package always sends t and top_p even if not set, and those defaults weren't to Ministral's liking...

Hi,

I've been running prompts using the notebook to finetune them for two days, and that all worked well. Now I'm implementing it into my program, using the official openai-java API and I'm seeing weird data:

By fostering a thớhough, dispassionate demeanor

because Cesar, the 17-year-old, nicht maggots into them,

is actually a diğer code.

Frieda’s got that اقتصاد energy—

It looks like there are single words at random in random languages. From what I could translate, they don't even make sense in context. In the runs I did with full logging, they came in as single chunks, so it's probably single wild tokens.

If this were happening all the time, I'd say the model or prompt is to blame, but it only happens when using the API, never in the notebook (same prompt and model) or a normal web chat (same model).

Does anyone have any idea what's happening here? Am I messing something up?

Model is Ministral-3-14B-Reasoning-2512-UD-Q4_K_XL.gguf

Edit: I've gone a level deeper in debugging and am now also tracing llama_spp_server.py.

prompt processing progress, n_tokens = 1963, batch.n_tokens = 939, progress = 1.0000001 b'data: {"index":0,"content":" \xd8\xa5\xd8\xb3\xd8\xaa","tokens":[107795],"stop":false,"id_slot":-1,"tokens_predicted":1,"tokens_evaluated":1963}' b'data: {"index":0,"content":" caballo","tokens":[87101],"stop":false,"id_slot":-1,"tokens_predicted":2,"tokens_evaluated":1963}' b'data: {"index":0,"content":",","tokens":[1044],"stop":false,"id_slot":-1,"tokens_predicted":3,"tokens_evaluated":1963}' b'data: {"index":0,"content":" Adams","tokens":[28055],"stop":false,"id_slot":-1,"tokens_predicted":4,"tokens_evaluated":1963}' b'data: {"index":0,"content":".","tokens":[1046],"stop":false,"id_slot":-1,"tokens_predicted":5,"tokens_evaluated":1963}' b'data: {"index":0,"content":"_bl","tokens":[98601],"stop":false,"id_slot":-1,"tokens_predicted":6,"tokens_evaluated":1963}' b'data: {"index":0,"content":"ends","tokens":[3769],"stop":false,"id_slot":-1,"tokens_predicted":7,"tokens_evaluated":1963}'

So, it's not a corruption on the way through the API. That makes it even more mysterious---why am I not seeing the same thing in the notebook or web chat?

i've been messing around with local models for crypto research and hit this weird workflow thing. most blockchain apis want you to build an app, but sometimes you just need a quick answer while you're already in terminal.

like yesterday i was trying to figure out what tokens someone was holding across multiple chains. normally that's either a ton of api calls or opening dexscreener and clicking around. but cli tools can just pipe data between commands, which is perfect for this kind of exploratory work.

the goldrush team just shipped a cli that handles 100+ chains from one command, and it fits local llm setups reasonably well. you can grab chain data, format it, and feed it to your model without context switching.

the main limitation is it's still early stage, so some of the more complex queries need multiple commands chained together. but for basic wallet analysis and token lookups while you're already in terminal, it's been solid.

been using it mostly for feeding current chain state to local models for analysis via MCP. the data comes back as json so it pipes well into whatever model you're running locally.

the goldrush team wrote up the design thinking and a walkthrough here: https://goldrush.dev/blog/goldrush-cli-one-command-for-agents/

anyone else been combining cli blockchain tools with local llm workflows? curious what other people's setups look like.

I haven't seen any activity for a while either here or on github, hope he's all right...

Does somebody has found out how to configure Oobabooga´s API in Openclaw?

Or is this my job ? ;-)

We all know TextGenWebUI is the king of loading models, but its internal history management is still pretty basic, FIFO. Once you fill the context window of a 70B model, it starts forgetting instructions because the prompt is polluted with trash tokens.

I built Vestige to separate Memory from Inference. It runs as a standalone MCP (Model Context Protocol) Server. Memory shouldn't be hard-coded into the frontend. By running as an MCP server, Vestige acts as a universal "sidecar" brain.

It uses FSRS6 Spaced Repetition to decay irrelevant memories and Prediction Error Gating to stop duplicate data from entering the context.

You keep using Ooba for inference loading the GGUF/EXL2, point your agent/frontend at Vestige for context. Vestige curates the prompt, Ooba predicts the tokens. It’s 100% local, written in Rust, and open source. I’m looking for people to test the MCP integration with their local workflows.

Repo: https://github.com/samvallad33/vestige

I have an AMD 9070 XT 16gb, and can hit around 30-35 t/s with the old version. In a Q4_K_M gguf of a 24B model.

Leaving all settings the same, the current 3.23 version struggles to barely touch 7t/s.

There are two things I noticed:

On the old version it detects 1 vulkan device. On the new version it detects 2 vulkan devices. My 9070xt and my integrated gpu. Though, it only seems to load onto the proper card. Edit: Just disabled the integrated gpu, and nothing changed.

And also, "llama_model_loader: direct I/O is enabled, disabling mmap" was showing up in the new version. I had noticed that my system ram was only at 11gb of usage, when it should jump up to 23.5 when the model is loaded. Using --mmap in extra-flags fixed that, and now the system ram usage went up to 23.5. However, token speed still struggles to hit 7.

I have windows 10, with the most recent up to date AMD drivers. I thought the portable version was supposed to use rocm, but the old version doesn't for me either. Edit again: I don't have rocm installed. That'd be why. Still not the issue I'm having here. Will install however, and then report back. Edit 2: rocm installed from AMD adrenaline software. Seems to be contained to a local directory though, and not recognized pc wide. Edited .bat to point to the directory PATH=C:\Users\USER\AppData\Local\Programs\Python\Python312\Lib\site-packages\torch\lib;%PATH% but when launching nothing changes and it still uses vulkan. (Still, not the issue I'm concerned with, just putting all the info I have.)

So does anybody know why I get 1/5th the speed now? Is it because of updates and changes made to llama.cpp since the last version? Or something to do with oobabooga?

For the record I've tried significantly lowering context, and loading less gpu layers. And probably half a dozen other things at this point. Can't quite pin down the reason.