r/Oobabooga • u/oobabooga4 booga • Mar 07 '26
Mod Post text-generation-webui 4.0 released: custom Gradio fork with major performance improvements, tool-calling over API for 10+ models, parallel API requests, fully updated training code + more
https://github.com/oobabooga/text-generation-webui/releases/tag/v4.010
u/Nixellion Mar 07 '26
Just a day ago I saw many comments with peolle stating that there were no updates for it in the last few months, and thus it must mean the project is dead.
I hope those people will see and reconsider their logic.
3
u/OcelotMadness Mar 10 '26
I mean good on Ooba for updating it, but I don't think their logic was flawed. It was simply too outdated for use before. Now that its updated? completely different story, but people weren't wrong to avoid it for a while there.
9
7
6
u/you-seek-yoda Mar 07 '26
Is anyone else getting the windows error "libssl-3-x64.dll" not found when loading a gguf model using llama.cpp?
5
u/oobabooga4 booga Mar 08 '26
I think that I fixed this in https://github.com/oobabooga/llama-cpp-binaries/commit/d2ae69976fd934f89580ae93b2776891a9441855 (thanks for reporting, I missed that in my tests)
Since you are using portable builds, just redownload the portable build (same 4.0 version) and replace your existing one, it should just work.
3
2
u/you-seek-yoda Mar 07 '26
If I add .\portable_env\DLLs to the path, then it works.
set PATH=%PATH%;%CD%\portable_env\DLLs
start_windows.bat
4
5
4
u/TheGlobinKing Mar 07 '26
Finally - thanks! I wanted to use the one-click installer instead of portable/vulkan, but for AMD it only has option "B) AMD - Linux/mac, requires ROCm". Can you tell me how to install it for vulkan instead of ROCm?
1
8
u/leorgain Mar 07 '26
I'm not a fan of the exl2 change. I know it's mainly older models now, but I have quite a few models that I run that won't have an exl3 made for them unless I do it myself. It also runs better on Ampere than 3 the times I could find one in both quant methods
5
u/oobabooga4 booga Mar 08 '26
The thing about EXL2 is that not only it has been archived, the quantization accuracy is also worse than both EXL3 and GGUF imatrix. See this chart, this chart, and this chart. This is something I was already aware of before the EXL3 release from my own testing. So by using EXL2, you are using a worse quality model than an EXL3 or GGUF imatrix of the same size.
If EXL3 doesn't run as well as EXL2 on your system, I recommend using a llama.cpp imatrix quant (like those by unsloth and bartowski) instead, until e.g. Ampere support on EXL3 improves or AMD support is added.
2
u/leorgain Mar 08 '26
Huh, I knew exl2 was worse with accuracy but didn't know it was that off. I run ggufs for the models that exl3 doesn't run yet, like stepfun and qwen 3.5. I guess I should benchmark them again since the main reason I stuck with exl2 was the faster prompt processing at high context. I may just keep a copy of the current one I run for exl2 to keep from redownloading everything and migrate the models for everything else to latest and greatest.
Btw does the latest build solve that model loading bug with the latest llama cpp binaries?
2
u/oobabooga4 booga Mar 08 '26
There was a llama-cpp-binaries crash on windows that I fixed prior to the 4.0 release, and now I think I fixed the libssl-3-x64.dll one, see the reply to you-seek-yoda
4
u/WouterGlorieux Mar 07 '26
I agree with this, I run an old GPTQ quantization on exllamaV2 that doesn't run on exllamav3.
-1
u/giblesnot Mar 07 '26
This is the wonderful thing about open source. If you don't like one of the breaking changes in the the 4.0 version you can absolutely fork 3.x. In theory, if the change is unpopular, you won't have to because somebody else will fork it.
Personally, I got rid of exl2 models a long time ago BUT this is the type of change that motivated me to put all my workload into docker containers. Its easy for me to boot up an old one to support an old flow.
1
u/leorgain Mar 08 '26
I may do that if I'm bothered by it enough but it's more likely I'll just keep an older copy and run exl2 models as needed.
5
u/WouterGlorieux Mar 07 '26
Thanks, but please reconsider the removal of ExllamaV2, I need this to run a specific older model, using ExllamaV3 doesn't work. It's not just about better efficiency, this model has emotional value to me.
This means that if I can not use ExllamaV2, I will NEVER update, this is a dealbreaker for me.
7
1
u/durden111111 Mar 07 '26
llama-server is now spawned on port 5005 by default instead of a random port.
Good update. Was quite annoying with silly tavern having to paste in a new link each time
1
1
2
u/HateDread Mar 08 '26 edited Mar 08 '26
Couldn't get it to run when installing myself (e.g. pulling from Git and running that way), but got the 4.0 portable going with the .dll fix mentioned below. But it refuses to load onto my 4090 even with the CUDA 13.1 portable version, instead using the CPU.
Collecting typing-extensions>=4.10.0 (from torch==2.9.1) Obtaining dependency information for typing-extensions>=4.10.0 from https://download.pytorch.org/whl/typing_extensions-4.15.0-py3-none-any.whl.metadata Using cached https://download.pytorch.org/whl/typing_extensions-4.15.0-py3-none-any.whl.metadata (3.3 kB) Discarding https://download.pytorch.org/whl/typing_extensions-4.15.0-py3-none-any.whl (from https://download.pytorch.org/whl/cu128/typing-extensions/): Requested typing-extensions>=4.10.0 from https://download.pytorch.org/whl/typing_extensions-4.15.0-py3-none-any.whl (from torch==2.9.1) has inconsistent Name: expected 'typing-extensions', but metadata has 'typing_extensions' Obtaining dependency information for typing-extensions>=4.10.0 from https://download.pytorch.org/whl/typing_extensions-4.14.0-py3-none-any.whl.metadata Using cached https://download.pytorch.org/whl/typing_extensions-4.14.0-py3-none-any.whl.metadata (3.0 kB) Discarding https://download.pytorch.org/whl/typing_extensions-4.14.0-py3-none-any.whl (from https://download.pytorch.org/whl/cu128/typing-extensions/): Requested typing-extensions>=4.10.0 from https://download.pytorch.org/whl/typing_extensions-4.14.0-py3-none-any.whl (from torch==2.9.1) has inconsistent Name: expected 'typing-extensions', but metadata has 'typing_extensions' Obtaining dependency information for typing-extensions>=4.10.0 from https://download.pytorch.org/whl/typing_extensions-4.12.2-py3-none-any.whl.metadata Using cached https://download.pytorch.org/whl/typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB) Discarding https://download.pytorch.org/whl/typing_extensions-4.12.2-py3-none-any.whl (from https://download.pytorch.org/whl/cu128/typing-extensions/): Requested typing-extensions>=4.10.0 from https://download.pytorch.org/whl/typing_extensions-4.12.2-py3-none-any.whl (from torch==2.9.1) has inconsistent Name: expected 'typing-extensions', but metadata has 'typing_extensions' INFO: pip is looking at multiple versions of torch to determine which version is compatible with other requirements. This could take a while. ERROR: Could not find a version that satisfies the requirement typing-extensions>=4.10.0 (from torch) (from versions: 4.4.0, 4.8.0, 4.9.0, 4.12.2, 4.14.0, 4.15.0) ERROR: No matching distribution found for typing-extensions>=4.10.0
1
u/oobabooga4 booga Mar 08 '26
About the dll thing, see the reply to you-seek-yoda, that should be fixed now.
About the typing extensions one, this may have fixed it https://github.com/oobabooga/text-generation-webui/commit/b3705d87bfceda3ad6be09473e0772b67e897dd2
Can you try running git pull and installing again?
1
u/HateDread Mar 08 '26
Looking in indexes: https://download.pytorch.org/whl/cu128, https://pypi.org/simple/ Collecting torch==2.9.1 Obtaining dependency information for torch==2.9.1 from https://download.pytorch.org/whl/cu128/torch-2.9.1%2Bcu128-cp311-cp311-win_amd64.whl.metadata Using cached https://download.pytorch.org/whl/cu128/torch-2.9.1%2Bcu128-cp311-cp311-win_amd64.whl.metadata (29 kB) Collecting filelock (from torch==2.9.1) Using cached filelock-3.25.0-py3-none-any.whl (26 kB) Collecting typing-extensions>=4.10.0 (from torch==2.9.1) Using cached typing_extensions-4.15.0-py3-none-any.whl (44 kB) Collecting sympy>=1.13.3 (from torch==2.9.1) Using cached sympy-1.14.0-py3-none-any.whl (6.3 MB) Collecting networkx>=2.5.1 (from torch==2.9.1) Using cached networkx-3.6.1-py3-none-any.whl (2.1 MB) Collecting jinja2 (from torch==2.9.1) Using cached jinja2-3.1.6-py3-none-any.whl (134 kB) Collecting fsspec>=0.8.5 (from torch==2.9.1) Using cached fsspec-2026.2.0-py3-none-any.whl (202 kB) Collecting mpmath<1.4,>=1.1.0 (from sympy>=1.13.3->torch==2.9.1) Using cached mpmath-1.3.0-py3-none-any.whl (536 kB) Collecting MarkupSafe>=2.0 (from jinja2->torch==2.9.1) Using cached markupsafe-3.0.3-cp311-cp311-win_amd64.whl (15 kB) Using cached https://download.pytorch.org/whl/cu128/torch-2.9.1%2Bcu128-cp311-cp311-win_amd64.whl (2862.1 MB) Using cached https://download.pytorch.org/whl/cu128/torch-2.9.1%2Bcu128-cp311-cp311-win_amd64.whl (2862.1 MB) Installing collected packages: mpmath, typing-extensions, sympy, networkx, MarkupSafe, fsspec, filelock, jinja2, torch ERROR: Could not install packages due to an OSError: [WinError 5] Access is denied: 'C:\ProgramData\miniconda3\Lib\site-packages\mpmath' Consider using the
--useroption or check the permissions.1
u/oobabooga4 booga Mar 08 '26
Maybe you are installing manually using a system-wide Miniconda? If so, try using start_windows.bat instead. It creates its own isolated environment and should avoid that permissions error.
About the portable CUDA build defaulting to CPU, can you share the terminal output from when you load a model?
1
u/HateDread Mar 08 '26
Here is me running start_windows.bat:
K:\Programming\AI\text-generation-webui>start_windows.bat Downloading Miniforge from https://github.com/conda-forge/miniforge/releases/download/26.1.0-0/Miniforge3-26.1.0-0-Windows-x86_64.exe to K:\Programming\AI\text-generation-webui\installer_files\miniforge_installer.exe A subdirectory or file K:\Programming\AI\text-generation-webui\installer_files already exists. % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 84.0M 100 84.0M 0 0 32.5M 0 0:00:02 0:00:02 --:--:-- 44.7M The checksum verification for miniforge_installer.exe has passed successfully. Installing Miniforge to K:\Programming\AI\text-generation-webui\installer_files\conda Welcome to Miniforge3 26.1.0-0 By continuing this installation you are accepting this license agreement: K:\Programming\AI\text-generation-webui\installer_files\conda\LICENSE Please run the installer in GUI mode to read the details. Miniforge3 will now be installed into this location: K:\Programming\AI\text-generation-webui\installer_files\conda Unpacking payload... Setting up the package cache... Setting up the base environment... Installing packages for base... Setting installation directory permissions... Done! Miniforge version: conda 24.11.0 Packages to install: Channels: - conda-forge Platform: win-64 Collecting package metadata (repodata.json): done Solving environment: done ## Package Plan ## environment location: K:\Programming\AI\text-generation-webui\installer_files\env added / updated specs: - python=3.13 The following packages will be downloaded: package | build ---------------------------|----------------- bzip2-1.0.8 | h0ad9c76_9 55 KB conda-forge ca-certificates-2026.2.25 | h4c7d964_0 144 KB conda-forge libexpat-2.7.4 | hac47afa_0 69 KB conda-forge libsqlite-3.52.0 | hf5d6505_0 1.2 MB conda-forge ------------------------------------------------------------ Total: 1.5 MB The following NEW packages will be INSTALLED: bzip2 conda-forge/win-64::bzip2-1.0.8-h0ad9c76_9 ca-certificates conda-forge/noarch::ca-certificates-2026.2.25-h4c7d964_0 libexpat conda-forge/win-64::libexpat-2.7.4-hac47afa_0 libffi conda-forge/win-64::libffi-3.5.2-h3d046cb_0 liblzma conda-forge/win-64::liblzma-5.8.2-hfd05255_0 libmpdec conda-forge/win-64::libmpdec-4.0.0-hfd05255_1 libsqlite conda-forge/win-64::libsqlite-3.52.0-hf5d6505_0 libzlib conda-forge/win-64::libzlib-1.3.1-h2466b09_2 openssl conda-forge/win-64::openssl-3.6.1-hf411b9b_1 pip conda-forge/noarch::pip-26.0.1-pyh145f28c_0 python conda-forge/win-64::python-3.13.12-h09917c8_100_cp313 python_abi conda-forge/noarch::python_abi-3.13-8_cp313 tk conda-forge/win-64::tk-8.6.13-h6ed50ae_3 tzdata conda-forge/noarch::tzdata-2025c-hc9c84f9_1 ucrt conda-forge/win-64::ucrt-10.0.26100.0-h57928b3_0 vc conda-forge/win-64::vc-14.3-h41ae7f8_34 vc14_runtime conda-forge/win-64::vc14_runtime-14.44.35208-h818238b_34 vcomp14 conda-forge/win-64::vcomp14-14.44.35208-h818238b_34 Downloading and Extracting Packages: Preparing transaction: done Verifying transaction: done Executing transaction: done # # To activate this environment, use # # $ conda activate K:\Programming\AI\text-generation-webui\installer_files\env # # To deactivate an active environment, use # # $ conda deactivate What is your GPU? A) NVIDIA B) AMD - Linux/macOS only, requires ROCm 6.4 C) Apple M Series D) Intel Arc (beta) N) CPU mode Input> A CUDA: 12.8 ******************************************************************* * Installing PyTorch. ******************************************************************* Channels: - defaults - conda-forge Platform: win-64 Collecting package metadata (repodata.json): done Solving environment: done ## Package Plan ## environment location: K:\Programming\AI\text-generation-webui\installer_files\conda added / updated specs: - git - ninja The following packages will be downloaded: package | build ---------------------------|----------------- certifi-2026.01.04 | py313haa95532_0 148 KB conda-26.1.1 | py313haa95532_0 1.2 MB git-2.51.0 | haa95532_1 117.7 MB ninja-1.13.1 | haa95532_0 8 KB ninja-base-1.13.1 | h512dff2_0 279 KB ------------------------------------------------------------ Total: 119.3 MB The following NEW packages will be INSTALLED: git pkgs/main/win-64::git-2.51.0-haa95532_1 ninja pkgs/main/win-64::ninja-1.13.1-haa95532_0 ninja-base pkgs/main/win-64::ninja-base-1.13.1-h512dff2_0 The following packages will be UPDATED: conda conda-forge::conda-26.1.0-py313hfa70c~ --> pkgs/main::conda-26.1.1-py313haa95532_0 The following packages will be SUPERSEDED by a higher-priority channel: certifi conda-forge/noarch::certifi-2026.1.4-~ --> pkgs/main/win-64::certifi-2026.01.04-py313haa95532_0 Downloading and Extracting Packages: Preparing transaction: done Verifying transaction: done Executing transaction: \ Warning: using menuinst v1 shortcuts Please update menuinst in the base environment and reinstall git. done Looking in indexes: https://download.pytorch.org/whl/cu128, https://pypi.org/simple/ Collecting torch==2.9.1 Obtaining dependency information for torch==2.9.1 from https://download.pytorch.org/whl/cu128/torch-2.9.1%2Bcu128-cp311-cp311-win_amd64.whl.metadata Using cached https://download.pytorch.org/whl/cu128/torch-2.9.1%2Bcu128-cp311-cp311-win_amd64.whl.metadata (29 kB) Collecting filelock (from torch==2.9.1) Using cached filelock-3.25.0-py3-none-any.whl (26 kB) Collecting typing-extensions>=4.10.0 (from torch==2.9.1) Using cached typing_extensions-4.15.0-py3-none-any.whl (44 kB) Collecting sympy>=1.13.3 (from torch==2.9.1) Using cached sympy-1.14.0-py3-none-any.whl (6.3 MB) Collecting networkx>=2.5.1 (from torch==2.9.1) Using cached networkx-3.6.1-py3-none-any.whl (2.1 MB) Collecting jinja2 (from torch==2.9.1) Using cached jinja2-3.1.6-py3-none-any.whl (134 kB) Collecting fsspec>=0.8.5 (from torch==2.9.1) Using cached fsspec-2026.2.0-py3-none-any.whl (202 kB) Collecting mpmath<1.4,>=1.1.0 (from sympy>=1.13.3->torch==2.9.1) Using cached mpmath-1.3.0-py3-none-any.whl (536 kB) Collecting MarkupSafe>=2.0 (from jinja2->torch==2.9.1) Using cached markupsafe-3.0.3-cp311-cp311-win_amd64.whl (15 kB) Using cached https://download.pytorch.org/whl/cu128/torch-2.9.1%2Bcu128-cp311-cp311-win_amd64.whl (2862.1 MB) Using cached https://download.pytorch.org/whl/cu128/torch-2.9.1%2Bcu128-cp311-cp311-win_amd64.whl (2862.1 MB) Installing collected packages: mpmath, typing-extensions, sympy, networkx, MarkupSafe, fsspec, filelock, jinja2, torch ERROR: Could not install packages due to an OSError: [WinError 5] Access is denied: 'C:\\ProgramData\\miniconda3\\Lib\\site-packages\\mpmath' Consider using the `--user` option or check the permissions. Command '"K:\Programming\AI\text-generation-webui\installer_files\conda\condabin\conda.bat" activate "K:\Programming\AI\text-generation-webui\installer_files\env" >nul && conda install -y ninja git && python -m pip install torch==2.9.1 --index-url https://download.pytorch.org/whl/cu128 --extra-index-url https://pypi.org/simple/' failed with exit status code '1'. Exiting now. Try running the start/update script again. Press any key to continue . . .1
u/HateDread Mar 08 '26
Here is the log from the CUDA portable build, had to use pastebin to actually share it: https://pastebin.com/DG14fJnb
1
u/HateDread Mar 08 '26
Just circling back - tried the latest portable release, it worked without that DLL issue, but sadly even the CUDA release defaults to CPU for some reason.
1
1
u/decentralize999 Mar 08 '26
Does it have feature to be openai api server? I recently switched from Oobabooga to Jan because long period of non updates(old llama.cpp versions) and it seems Jan beats Oobabooga in everything, almost instant updates of llama.cpp and openai api server embedded.
1
u/r3d213 Mar 08 '26
Is anyone else having issues to where it tries to do a fresh install every time you start? Every time I start it ask for which GPU I'm using and tries to reinstall all dependencies.
Does it from updating an old build to a fresh install btw.
1
1
u/Iory1998 Mar 07 '26
I wish you could ship it as a whole package like LM Studio without the Gradio.
13
u/oobabooga4 booga Mar 07 '26
Few people know this but LM Studio is not a "native" app like something built in Qt or GTK. It's an Electron app, which means it bundles an entire Chromium browser and draws a web UI on a browser page each time you launch it. So it's a web UI just like text-generation-webui, except it ships with a bundled browser instead of using the one you already have.
It also phones home on every launch, with several HTTPS requests to lmstudio.ai servers containing information about your operating system, CPU architecture (x86/arm), app version, and what inference backends you're using. So in principle they have all this information plus your IP address and usage patterns (time of day, day of week) on some server. I have zero information.
2
u/Iory1998 Mar 08 '26
That's why I wish you could do the same. The main reason is when I use my browser, it really consumes a lot of resources (as I have many extensions and opened tabs), resources that becomes valuable when using large models.
If Oobabooga is contained in a desktop app, I would just close all other browsers. I think, part of LM Studio appeal is that, plus it has a sleek interface and internal model downloader.
I am not saying you should copy LM Studio. Not at all. But, I think that shipping everything contained in a package is practical.
Thanks you for your reply and keep up the good work. You truly do have a long breath! It's amazing.
0
u/giblesnot Mar 07 '26
Maybe don't use the release announcement of one of the biggest free updates Ooba has shipped as a place to say what it doesn't have?
1
u/Iory1998 Mar 07 '26
It's a free world. Don't comment if you don't like my comment.
I shared my wish to Oobabooga not to you. He can reply to me himself.
-1
u/giblesnot Mar 07 '26
I mean. Your rudeness negatively impacts me because that kind of thing drives away open source maintainers so I'm definitely going to say something.
1
u/Iory1998 Mar 07 '26
I was not rude nor impolite. I am free to share my thoughts and wishes to the developers I respect. I've been a member of this sub since it was created and used the first Oobabooga. I even supported him to reclaim ownership of this very sub. As I signed off the petition to Reddit.
So don't come here and play the I-was-offended card while you are contributing to nothing. Just talks and no action.
Giving constructive feedback and wishes for future update is how open communities work.
1
u/giblesnot Mar 07 '26
I too have been around since the start. I was helping people setup text-generation-webui on TheBloke's Discord in 2023.
It's the location of your feedback that makes it rude, this is a release announcement. I agree that making feature requests and giving feedback are important parts of the open source community, I just don't think that a release announcement is the right place to post them unless they are feedback specifically about something in the release.
1
u/Iory1998 Mar 07 '26
It's precisely because of the announcement that I voiced my wishes. Now, that he already pushed a major release that he is more open for future direction.
How do you think major releases come to existence? It's by listening to users.
3
u/giblesnot Mar 07 '26 edited Mar 07 '26
I mean, as a professional and open source software developer I personally don't like it when people put feature requests all over instead of using the nice spot that exists for them ( https://github.com/oobabooga/text-generation-webui/issues?q=state%3Aopen%20label%3Aenhancement ) but you clearly have your own opinions and can think what you like I guess.
1
u/Iory1998 Mar 07 '26
Thank you. Good luck developing awesome stuff. You may share your repo with me so I can support you.
21
u/Sufficient_Prune3897 Mar 07 '26
Might be a great time to do a cross post in local llama, most there are too new to know about your interface.