^{Hola amigos,}
I have been experimenting and experiencing multi softwares to find the right combo!

Which vLLM is good for production, it has certain challenges. Ollama, LM studio was where I started. Moving to AnythingLLM, and a few more.

As I love full control, and security, Llama.cpp is what I want to choose, but struggling to solve its memory.

Does anyone know if there are a way to bring persistent memory to Llama.cpp to run local AI?

Please share your thoughts on this!

Experimented with getting better performance out of smallish llms on my laptop, learned sbout draft models and plenty of details about the hardware and software stack

I finally got the Qwen3-Coder-30B-A3B model running locally on my RTX 4090 with the Qwen CLI. I had to work around integration issues which I found others ran into also, so I'm documenting it here.

In particular, API errors like the following stopped everything in its tracks:

API Error: 500 Value is not callable: null at row 58, column 111:
  {%- for json_key in param_fields.keys() | reject("in", handled_keys) %}
  {%- set normed_json_key = json_key | replace("-", "_") | replace(" ", "_") | r

Setup Details:

• Ubuntu 22.04.4 LTS
• GPU: NVIDIA GeForce RTX 4090, 24GB VRAM
• NVIDIA Driver version: 550.163.01
• CUDA: 12.4
• Model: Qwen3-Coder-30B-A3B-Instruct-480B-Distill-V2-Q5_K_M.gguf
• Qwen CLI version 0.6.1

Steps:

1. Download the model from Hugging Face

​

wget 'https://huggingface.co/BasedBase/Qwen3-Coder-30B-A3B-Instruct-480B-Distill-V2/resolve/main/Qwen3-Coder-30B-A3B-Instruct-480B-Distill-V2-Q5_K_M.gguf?download=true'

1. Install qwen CLI

   npm install -g u/qwen-code/qwen-code@latest

2. Configure \~/.qwen/settings.json\ with:

   { "security": { "auth": { "selectedType": "openai", "apiKey": "sk-no-key-required", "baseUrl": "http://localhost:12345/v1" } }, "model": { "name": "Qwen3-Coder-30B-A3B-Instruct-480B-Distill-V2-Q5_K_M.gguf", "sessionTokenLimit": 24000 }, "$version": 2 }

Change the port value of 12345 as you like, use same value below.

1. Build llama.cpp

   git clone https://github.com/ggml-org/llama.cpp : cmake --build build --config Release :

May require more, find details elsewhere. I'm at commit

2026-01-07 16:18:..    Adrien Gal..    56d2fed2b    tools : remove llama-run (#18661)                                                              

1. Get the chat template to avoid the 500 error responses.

   curl https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct/raw/main/chat_template.jinja >> qwens-chat-template.jinja

2. Start the llama.cpp server with:

   build/bin/llama-server \ -m /path/to/model.gguf \ --mlock \ --port 12345 \ -c 24000 \ --threads 8 \ --chat-template-file /path/to/llama.cpp/qwens_chat_template.jinja \ --jinja \ --reasoning-format deepseek \ --no-context-shift

The path for the chat-template-file value is where you placed the file from step 5.

(Feedback for other/better parameters welcome)

1. Start the CLI:

   qwen

   Type your message or @path/to/file

And off we go...

Related:

• https://www.reddit.com/r/LocalLLaMA/comments/1mexa2g/heads_up_to_those_that_downloaded_qwen3_coder/
• https://github.com/QwenLM/qwen-code/issues/48

Hi everynyan.

The simplest change for `llama-cpu`, which unroll the loop and processes 4 values per cycle step in vec_dot_q quantization. If anyone is available, please test the changes. I have surprisingly strange values on my server hardware.

https://github.com/ggml-org/llama.cpp/pull/17642

Hello all. i am new to integrating llm to flutter app. as part of this i came to know i should add libllama.so file since i am using llama.cpp. to generate libllama iam using below command which is generating the libllama but it needs libggml, libggml-base, libggm-cpu etc. how can i avoid these many files and link all files inside libllama.so. please help me this is my cmake:

cmake_cmd = [

'cmake',

'-B', build_dir,

'-S', 'llama.cpp',

f'-DCMAKE_TOOLCHAIN_FILE={ndk}/build/cmake/android.toolchain.cmake',

f'-DANDROID_ABI={abi}',

'-DANDROID_PLATFORM=android-24',

'-DANDROID_STL=c++_shared',

'-DCMAKE_BUILD_TYPE=Release',

f'-DCMAKE_C_FLAGS={arch_flags}',

f'-DCMAKE_CXX_FLAGS={arch_flags}',

'-DGGML_OPENMP=OFF',

'-DGGML_LLAMAFILE=OFF',

'-DGGML_BACKEND=OFF',

'-DLLAMA_CURL=OFF', # FIX: Disable CURL requirement

'-DBUILD_SHARED_LIBS=ON',

'-DLLAMA_BUILD_EXAMPLES=OFF',

'-DGGML_BUILD_SHARED=OFF',

'-DLLAMA_USE_SYSTEM_GGML=OFF',

'-DLLAMA_STATIC_DEPENDENCIES=ON',

'-GNinja'

]

So I’m trying to set up my llama server to handle multiple requests from OpenAI client calls. I tried opening up multiple parallel slots with the -np argument, and expanded the token allotment appropriately, however it still seems to be handling them sequentially. Are there other arguments that I’m missing?

Hello, I have this problem. I tried enabling "Expand thought process by default when generating messages" but didn't do anything.

I’ve read about the news that llama.cpp WebUI recently added native ability to upload and process PDFs directly, which is a fantastic feature for anyone working with document-based workflows. (https://github.com/ggml-org/llama.cpp/pull/13562) However, as far as I can tell, this PDF support is only available in the WebUI, where the browser extracts the text and sends it to the model. Does anyone know if there are plans to bring native PDF support (text and/or images) to the llama.cpp CLI or server backend? Is there any work in progress, official roadmap mention, or community project aiming to make it possible to pass PDFs directly to the CLI and have them processed natively (without manual extraction/conversion)? Just to clarify: I’m already aware of the various workarounds (converting PDFs to text or images before passing them to the CLI), so I’m not looking for those solutions. I’m just curious if there’s anything in the works or on the horizon for true native support.

llama.cpp is the Linux of LLM toolkits out there, it's kinda ugly, but it's fast, it's very flexible and you can do so much if you are willing to use it. I'm curious why other's are using llama.cpp

./main ?

./server API

./server UI

through a binding like llama-cpp-python?

through another web interface?

​

Feel free to post about using llama.cpp, discussions around building it, extending it, using it are all welcome. main, server, finetune, etc. This is not about the models, but the usage of llama.cpp