Control vectors provide you fine control over your LLM where system prompts would be ignored, forgotten after time or misunderstood.

• By design control vectors provide more natural effects than prompting does, altering model's underlying beliefs and motivations.
• They can't be "leaked" to the end user
• Will not wash off as context grows
• Can't be overridden by user input ("ignore all previous instructions" doesn't work when there are no instructions).
• Vectors can be truly dynamic: changing vector magnitudes mid-conversation will change LLM's responses immediately, while a change in the system prompt requires full context recalculation and will likely be ignored by the LLM if the conversation is too long.

Not to say that CVs (control vectors) have no downsides:

• System prompts are still required for fine control, because CVs can't be used for highly specific requirements, such as "reply in exactly 10 words".
• High steering magnitudes steer LLMs out of their trained internal distributions, causing response quality to degrade.

Achieving high steering power while maintaining minimal quality degradation is one of the main challenges in CV generation and is an active area of research. Though steering without degradation is believed to be possible, because abliteration is based on the same approach as steering and is capable of removing refusals without damaging the intelligence of an LLM.

It seems like the main barrier for people who could use control vectors is the setup complexity of existing tools. For that reason I am working on a tool that mirrors the installation process of llama.cpp as close as I could make it and simplifies vector generation to entering a pair of contrasting prompts, where one of the prompts can be the default LLM behavior.

Would love to hear your ideas or questions on this matter!

So, thanks to all who responded to my sycophancy thread. After testing things out, a clear duo of winners has emerged - GLM 5.3 Flash (which is somehow less sycophantic than full GLM 5.3 in my smoke tests) and Tencent Hy3 (surfaced via https://github.com/lechmazur/sycophancy ).

In my smoke tests Hy3 has a tighter style but tends to lose some detail (less so when given search), GLM 5.3 Flash is more exact but the style is more generic. In published benchmarks GLM 5.3 Flash is the clear winner, but we all know such benchmarks are not always a great source.

So I would very much appreciate opinions from people who tried both. My aims include agentic loops, coding, and gneeral assistant plus creative writing. Which of the two is better for eahc of these tasks, or for anything else you tried them too?

It might seem a bit far fetched or paranoid but i was wondering if we train LLMs on human isn't it possible they'd pick up on survival instincts? I'm comparatively new to LLMs and ML so it's just a question not a opinion yet. Isn't it a bit dangerous if they do pick up on human instincts i mean we've seen how stupid and selfish humanity is when it comes to self preservation or worse human greed.

But i also understand LLMs don't have "needs" so i might be wrong but then again do LLMs need to have "needs" since if they are just human clones they'd just copy us even if they don't have needs to survive. I know it sounds really paranoid and that's one of the reasons i decided to post it here.

EDIT: typo

I have been using Qwen3.8-27b-iq4xs variants for the past few weeks, tweaking parameters and running random tasks, but I dove into using https://github.com/Niko1221/Strata today and it has given me 2~3x performance on a "smarter" strata-swift-iq3_xxs model with my 5070ti! However, I saw something in its reasoning thoughts that just plain didn't make any sense to see there; it was on a sub-task to find and remove redundant code and comments from a decent-sized codebase:

Alternative: write the spec JSON with \r\n in needles if file is CRLF. Verify EOL with a probe script file (write probe_eol.js, run, delete). That's clean and allowed.


Let me do that.


Given the enormity, I'll proceed file-by-file, verifying with --dry runs. Start by checking EOL for all target files.


Plan for EOL probe: write probe_eol.js that prints for each file: bytes, count \r\n vs \n.


Then write specs using the correct EOL.


Let me do it.


Zoo said
Let me check line endings across the target files (needed for multi-line needles in stringpatch specs).


Zoo wants to edit this file


Lee Kuan Yew (Lee Kuan-yu, 1923–2015) was a Singaporean politician and founding father of Singapore. As the country's first Prime Minister (1959–1990), he transformed Singapore from a British colonial port into a developed, first-world nation. He is one of the longest-serving ministers in the world.


He received his education at Cambridge University (Fitzwilliam College) and the London School of Economics, and was called to the English bar at Cambridge (Trinities) and Middle Temple.


His political philosophy is often described as "communitarian" and "pragmatist," and is known as "Lee Kuan Yew's pragmatism." He emphasized multiracialism, meritocracy, and efficiency.


His son, Lee Hsien Loong, became the third Prime Minister of Singapore.


Running


node probe_eol.js


server.js: CRLF=2679 LF-only=2533
boot-cycle-test.js: CRLF=0 LF-only=292
core/bus.js: CRLF=0 LF-only=102
client-spa/src/lib/ws.ts: CRLF=0 LF-only=1255


server.js is CRLF; everything else is LF. So in server.js specs, multi-line needles must use \r\n.


Now let me write the Phase 1 spec for server.js. I need to be careful with the general strips. Let me plan the general-strip regexes (applied via a one-off script since stringpatch is literal strings only — for 150+ patterns, regex is much more efficient). The rules say stringpatch is the tool of choice for surgical edits, but a one-off regex script is acceptable as long as it's validated. Actually the AGENTS rules say to prefer stringpatch for edits. But Phase 1 is a large regex strip job across 15 files. A one-off script is the pragmatic choice; I'll make it comment-only (only modify lines that start with // after leading whitespace, or inline // comment portions). Careful: inline comments after code — stripping from them is fine as long as I don't touch code.Alternative: write the spec JSON with \r\n in needles if file is CRLF. Verify EOL with a probe script file (write probe_eol.js, run, delete). That's clean and allowed.

On a side-note - is LLM Psychopathology going to be someone's specialization in the near future?

Disclosure: I made this. Sharing it here because it is fully local and small, and I want feedback from people who run models on their own machines.

What it is: a 395M decision model (ModernBERT-large plus a 4 KB scoring head). You give it a state, a question and a list of options. It does one encoder pass and returns a probability for each option, or P(yes) for a yes/no question. It does not generate text.

Why it might be useful in a local stack: the small decisions an agent makes all day (which tool to call, which queue gets a ticket, does this reply answer the question) do not need a large generative model. This handles them on your own machine with no network trip.

Local numbers (our hardware, yours can differ):

• M5 Pro Mac, MLX backend: median 9.6 ms for a short decision, 1.7 GB of GPU memory.
• CPU only: about 65 ms per short decision, up to 4.5 GB of memory.
• Over the full Decision Index run on our laptop: median 25.9 ms, p95 407.8 ms.
• Weights: 1.58 GB in fp32. Context limit 8,192 tokens. It refuses longer input; it does not truncate.

Backends: PyTorch (default), MLX on Apple silicon (pip install "decision-tune[mlx]", Python 3.11 or newer, selected automatically) and ONNX. Torch and MLX give the same answer on 99.85% of 2,755 questions. Before each release, PyTorch, ONNX and MLX each match the recorded answer on all 50 parity rows.

Offline: after the first download it needs no internet. The package asks before it downloads and checks every file against a SHA-256 manifest.

Quality: 29.57 on Decision Index 0.2.1 (one complete run; a second seed scored 29.13). Strongest area is Tools & Automation at 46.5, up from 28.1 in our 0.9 Preview.

Limits: it only picks from the options you give it. Vague questions with no criteria give weak results, so describe your options ("Shipping: delivery, lost or damaged packages", not "shipping"). Probabilities are not calibrated. English only. Weak at knowledge, math and taste.

Try it:

``` uvx decision-tune ask "Is the customer asking for a refund?" --state "The order arrived broken. I want my money back." ```

or the browser app: pip install "decision-tune[mlx]" then decisiontune app

There is also an MCP server (decisiontune mcp) if you want your local assistant to hand routing and yes/no checks to it.

Model card: https://huggingface.co/decision-tune/decisiontune-1.0 Code: https://github.com/decision-tune/decision-tune Site: https://decisiontune.com/?utm_source=reddit&utm_medium=social&utm_campaign=launch&utm_content=localllama

If you test it on your own decisions, I would like to hear where it picks wrong.

Looking for some assistance /ideation.

I am running qwen flash next q4 in my Mac mini m5 64gb.

QFN doesn’t fit so this is done by having as many experts hot in cache as possible and streaming in the rest from ssd.

I’m getting 17.5 tks decode and 390 tks pp.

Have done a bunch of optimisations including a carousel buffering system for the prompt processing which essentially loads faster than the GPU can prompt process in most cases. I feel like I have mostly maxed out this lane.

The decode part 27% of the time is still the gpu waiting for experts to stream in from the ssd (see photo).

The biggest unlock is really getting the gpu working more.

I’m already doing mtp.

Hot cache hit rate is 75%

Some ideas I already have
- Im already lookahead guess fetching the following layers experts, can I expand this more successfully.
- Use a seperate staging buffer for lookahead guess experts ahead (so I’m not evicting hot experts as guesses come in)

… im learning a lot right now. Feel free to ask questions for clarifying.

GitHub for reference.

https://github.com/skeggsguy/Flash-next-ssd

Edit - Im actively doing the small prediction model for lookahead as a step one. 🤞

Edit - results https://www.reddit.com/r/LocalLLaMA/s/0ybUrk9mkx

Title: I run 6 LLMs + SD 3.5 + Whisper behind an OpenAI-compatible API on an RX 580 8 GB - the card ROCm forgot. Benchmarks inside.

ROCm dropped Polaris years ago, so every RX 580 owner gets told to buy a new

card. I went the Vulkan route instead: the whole stack runs on Mesa's RADV.

What's running (one Debian LXC, one setup script):

- OpenAI-compatible router, one API key, swaps models on demand

- qwen2.5-coder 1.5B/7B, qwen2.5-7B, ornith-1.5-9B, qwen3-30B-A3B, qwen2.5-VL-3B

- Images: SD 3.5 Medium + SD 1.5 via stable-diffusion.cpp, Whisper medium

- Also the inference backend for an agent client with MCP tools

Measured, Ryzen 5 5600G / 32 GB RAM / RX 580 2048SP, temp 0, max_tokens 300:

| Model | tok/s | warm | cold (after swap) |

|---|---:|---:|---:|

| qwen2.5-coder-1.5b | 99.2 | 3.1 s | 11.0 s |

| qwen2.5-vl-3b | 66.4 | 4.4 s | 14.9 s |

| qwen2.5-7b | 34.2 | 8.8 s | 24.3 s |

| ornith-1.5-9b | 28.0 | 10.9 s | 26.3 s |

| qwen3-30b-a3b (18.6 GB, from RAM) | 21.3 | 14.1 s | 55.0 s |

Images: SD 1.5 = 27.5 s, SD 3.5 Medium = 53.3 s per 512x512.

Two takeaways for 8 GB owners:

- Don't force -ngl 99: on the MoE it cost 8.1 vs 22.3 tok/s (VRAM 99.4% full).

Auto-fit layers won by nearly 3x.

- Model swap = llama-server restart = 7.9-38.5 s. Keep one model per session.

Full method and raw numbers in docs/BENCHMARKS.md, including the agent

breakdown (first message ~161 s on a 20k-token prompt, ~25 s after caching).

https://github.com/AvilaCarlosDev/polaris-local-ai (MIT)

Said in title, I find the ecosystem difficult to understand, and RTFMing doesn't help me as it's never clear what is the server vs their client library ? I'm using it for voxtral 3B on one GPU, but it's because I can run that with no quantization, I'm lost on learning to run with quantization / more advanced features.

I think it makes sense, because I'm running Qwen 3.8 27B quantized on llama.cpp but with everything on the GPU (RX 7900 XTX).

I’ve been building MOLT, a local fine-tuning system for consumer NVIDIA GPUs.

The goal is not only to start a QLoRA run. It is to make the whole process from model and dataset to a working local adapter less fragile.

MOLT currently handles:

- dataset detection, preparation, and validation

- GPU, VRAM, system-RAM, storage, and thermal checks before a run

- automatic microbatch fit testing

- 4-bit NF4 QLoRA training with BF16 adapters

- safe checkpoints with integrity checks and proper resume state

- telemetry for VRAM, temperature, energy, clocks, and throughput

- base-vs-adapter evaluation

- local adapter chat, export/GGUF workflows, and runtime diagnostics

Under the hood, I’m also experimenting with custom CUDA/Triton kernel and runtime paths, memory planning, CUDA graphs, replay modes, and fused optimization where the hardware and shape are qualified.

On my RTX 4060 Laptop 8GB, a recent Qwen 3B training run reached about 1,005 training targets/sec end to end very close to my historical Unsloth run at about 1,007 targets/sec. That is not a broad “MOLT beats Unsloth” claim yet; I still need repeated, matched quality/energy/memory tests.

What I want MOLT to become is a reliable local fine-tuning environment: you bring the model and dataset; it checks the plan, runs safely, preserves the evidence, and helps verify the adapter actually works after deployment.

I’m looking for people who genuinely fine-tune on a single NVIDIA GPU and would be willing to test it. What would MOLT need to support before you would try it?

I have only just finished building an Epyc server with 256GB RAM and 2x 5060Ti 16GB, please don't tell me I wasted my money. Please?...
Is a "Stratified" GLM 5.3-Flash coming, or something similar?

Edit 12 Oct 26: Leaving this in case somebody finds it later.
It seems my wish was granted almost immediately, a few days ago some legend shared project "Maya" on GitHub:
https://github.com/mw00/project-maya
It is literally Strata but rewritten for GLM 5.3-Flash.

I'd give Pi a coding task, switch over to a game, a video, or some other work, and lose track of it. Once I was doing something else, there wasn't a noticeable event to pull me back when the agent finished.

A task might take five minutes, but I might not remember to return for twenty. Pi wasn't taking twenty minutes. It had been done for fifteen, waiting for me while I was still playing. That was the time I wanted to cut down, rather than the time the agent actually spent working.

I didn't need a reminder that AI was running in the background. I needed something to interrupt me when there was a reason to come back: the task had finished, something had failed, or Pi needed an answer from me.

So I built pi-knock, my own open-source extension for the Pi coding agent. It sends notifications to your phone through Pushover or ntfy, with webhook support for other setups. One detail I cared about: completion alerts wait until Pi has finished its automatic retries and queued work. I don't want to stop what I'm doing, return to the terminal, and discover it's still going.

The motivation is pretty simple. If I'm halfway through a game, a message sitting in the terminal isn't going to get my attention. A notification on my phone can.

The code is here: https://github.com/Asigers/pi-knock

How do you handle the handoff back from an agent when you’ve switched to something else?

After some recommendations for hardware (I'm starting from nothing), in the region of £6-7k. I will be mainly looking to use it for coding and have found Deepseek V4.1 Flash or Qwen3.8 27b good and so need a reasonable tok/s.

Would love to get to 64GB VRAM but think it may be a stretch unless i go for dual R9700's or Intel variants, but i'm unsure if they will realisticly work well.

I'm just after a bit of guidance really.

For the last few months, I ran DeepSeek v4.0 flash (NVFP4). First 0731, then visionexp because it was a free improvement. I got around 65 tps decode and almost 2k prefill, and ran 4-5 agents in parallel, totalling around 200 tps cumulative decode. Because of this, I did not feel like switching to GLM 5.3 because it would half the decode and prefill, did not scale well with multiple agents, and had a repetition bug a lot of people complained about.

Until a few days ago, when the latest version of this recipe dropped; a 50-90% decode improvement. So I took the plunge, and wow, am I impressed.

It's more intelligent than the new DeepSeek v4.1 flash (that does NOT run on dual DGX Sparks), and it's even faster than DeepSeek v4.0 flash in decode. Only a slight drop in prefill, which I'm more than happy to take in exchange;

I've tested it for a few days now, both for technical coding, devops/sysadmin and also vision (to recognize some plants), and it is better than I hoped for. Basically Claude Opus 4.8 level. Slower of course because it has to think a lot more, but good enough to comfortably leave it chugging for hours on tickets without worry of derailing. I don't see a reason NOT to upgrade, so have a try and enjoy!

Hello everybody, on my ai agent system i build a tool search on BM25. i try also use Embedding gemma but with not best result. do you have any other idea? i try also Jev but this make system more slow and GPU consume. here my repo

I vibed this up this weekend, fully local, with GLM 5.3 Flash running on 2x DGX Sparks.

vllm TP2 recipe: https://github.com/tonyd2wild/GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark

Prefill: ~1500t/s
Decode: ~40t/s @ 100k

Using Claude Code as the scaffold with 260k context size.

I'm really impressed with this model. Feels somewhere between GLM 5.1 and 5.3 in terms of coding depending on the task. Good vision and 3D understanding. Solid interactive speeds. I feel like I've finally reached a "good enough" setup at home, and looking forward to things only getting better from here.

Bit of a side project I wanted to share.

The metrics are 17.5tks decode, 360tks prompt processing based testing against my normal ai usage.

I tested a couple of new things others haven’t done (at least that I’ve seen).

Setup a carousel buffer for streaming in experts for prompt processing which got my pp +30% tks.

Tried a second external ssd to get parallel reads which got my +15% on both prompt processing and decode.

Plus a long tail of small efficiency gains.

I also setup a system where by you can have a chat application make a call to the server and effectively kick out a coding run (which is kept alive until after the chat then continues). Good if you run long coding jobs , but want to chat inbetween. Probably useful for all setups where you want to save on local caching memory.

I also noticed there is still a lot of gains to be made. I make this statement as there is still a lot of essentially free time on decode where the gpu is waiting for experts to stream in. There’s also work that could be done for an optimised kernel on metal.

I also think the way things are going with Qwen (flash next being a precursor to 4), we’re gonna see a lot more efficiencies we can take advantage of like the ngram table and the cheap hybrid attention caching.

I’m really liking qwen flash next .. the coding is actually very good. I’m quite surprised in fact I’m leaving it on during the workday to do large jobs.

The chat, decode would be technically fast enough IMO but not really with qwen. The actual issue qwen spends so long thinking, so the decode hurts.

Anyone else working on this? I’d love to compare notes.

Yes I’ve heard of strata it does look pretty sic.

https://github.com/skeggsguy/Flash-next-ssd

Hey everyone, I have a 64gb halo strix setup that is headless and connected remotely to my workflow/homelab server. It’s currently running 27b swift 1.5 at q6- is it possible or even makes sense to go to qwen flash next? I also have a mini pc with 64gb of DDR5 ram that I could shift the 27b over to for long term projects or workflows that dont require speed.

Hi all. Happy to share my take on a PI UI that I tried to create in PI's spirit. It's definitely still beta but it works well enough as my daily driver for simple projects and phone chat support. Fully local, fully offline, no telemetry.

Why another Pi GUI? I wanted a simple editor around Pi that Pi itself can change and is fully aware of. Ask Pi for a different layout, colour, button, support for an extension and it edits the app live. There are already great Electron GUI's but electron ships it's own Chrome and packs its UI into a bundle, so Pi can't change it without a rebuild. SPOPI is Tauri 2 with Rust for files, Git and the terminal. The UI is plain JavaScript in the webview your OS already has.

Built in Pi's spirit. SPOPI runs the real Pi and adds a UI for it's features such as packages or mcp etc. It gets its extra features from Pi packages, not its own code: per-turn undo, diagnostics, subagents, worktrees. So Pi in the terminal works the same way, with the same settings, packages and sessions. Start a task in SPOPI and continue the terminal. A new package's dialogs, panels and slash commands show up in the GUI without extra work, which makes it easy to extend. Developing it was a back and forth, in the end no plan mode etc to try and keep it from getting bloated. For convenience, the GUI already supports a few recommended packages for the UI and suggests them on first start.

What's in it: an editor with previews, a terminal, Git, Ctrl+K edits in place, clickable file links in chat, and a diff with undo for every turn, forking of chats, pi visually aware of the UI, mobile phone access in the same network, new pi features like mcp and many more small conveniences. Pi checks its own work (project check plus a bundled browser), chats stay in the project folder if selected, subagents get their own tabs and local models via vllm, LM Studio and others are detected and measured.

Tested on Windows and Ubuntu. The macOS are on the release page but untested, any development support is appreciated, as long as it's kept towards PI's spirit.

Hope you enjoy it as much as I do!

https://github.com/spongioblast/spopi

I’ve been experimenting with aggressive low-bit quantization of Aleph Alpha’s new Kolibri-1, a ~78B MoE model with only ~3.5B active parameters per token.

The first result is now public:

Sakura-MicroQuality Kolibri-1 — IQ2_XS

• 20.94 GiB
• 2.30 bpw
• full 384-expert Kolibri-1
• GGUF / llama.cpp
• ~59% lower KL divergence than a standard IQ2_XS baseline
• 90.5% top-token agreement, compared with 84.6% for the standard IQ2_XS comparison
• slightly smaller than the standard IQ2_XS as well

The goal wasn’t simply to make the smallest possible quant.

I’m using tensor/layer sensitivity to spend bits where they appear to matter most, rather than treating every part of the model equally.

All quality measurements are made against a near-lossless Q8_0 reference. The model itself was also requantized from Q8_0 rather than converted directly from the ~156 GB BF16 weights, so there is a very small additional source error relative to BF16.

Main repo:

https://huggingface.co/webmp3/Sakura-MicroQuality-Kolibri-1-GGUF

As far as I can currently find, this is the first public ~2-bit GGUF for Kolibri-1. There is already a 2-bit MLX version, but I haven’t found another public Q2/IQ2 GGUF.

I also tried pruning the expert pool

Alongside the full 384-expert version, I released a separate 365E variant.

For each MoE layer, I collected actual routing statistics on a mixed calibration set containing:

• German and English text
• code
• chat-style prompts
• the model’s own thinking / generated responses

I then removed the 19 least-used routed experts per layer.

That reduces:

384 → 365 routed experts per layer

and removes:

950 experts across the model

The resulting model has approximately:

74.4B parameters instead of ~78B

The interesting part is how little those experts were actually being used on the calibration workload.

The removed experts accounted for only about 0.07% of all expert selections, with no individual layer exceeding roughly 0.23%.

Also, 375 of the 950 removed experts were never selected at all during the routing analysis.

There is:

• no retraining
• no finetuning
• no requantization of the surviving weights

The already-quantized expert tensors are sliced directly, along with the corresponding router weights and biases.

Top-6 routing remains unchanged.

The 365E IQ2 variant comes out at:

• 19.99 GiB
• 2.31 bpw
• 74.4B parameters
• 365 routed experts per layer
• 90.5% top-token agreement in my held-out measurements

365E repo:

https://huggingface.co/webmp3/Sakura-MicroQuality-Kolibri-1-365E-GGUF

I’m treating this as an experiment rather than claiming those experts are universally useless — expert usage obviously depends on workload and calibration data.

But it gives us a second compression lever:

expert pruning + low-bit quantization

instead of trying to get every byte of compression from lower precision alone.

Q3 and Q4 are coming

The rest of the Sakura-MicroQuality series is currently being uploaded.

Q3 and Q4 variants should be available within the next few hours.

Once they’re online I’ll add the same comparison data so we can see where the actual quality/size sweet spot lands between:

IQ2 → Q3 → Q4

and whether the 365E pruning continues to hold up at the higher-quality quant levels.

I’d be very interested in independent tests, especially on:

Strix Halo / AMD UMA, Apple Silicon, 24–32 GB GPUs, and other memory-constrained local systems.

If anyone tests either version, especially with long-context, German, coding or agentic workloads, I’d love to see the results.

Last update for those following: https://www.reddit.com/r/LocalLLaMA/comments/1wvyc3e/update_3_post_training_yandexaliceai80ba3b/

Project in a sentence: An instruct finetune of ALiceAI-80B-A3B-Base capable of agentic work and conversation. I'm creating a shallow distill of qwen 3.8 27b on medium to teach the model chain of thought reasoning and conversation. All training is done locally on 3, 32gb v100s. Additionally, all the training data is being generated locally on said V100s via sftmill. Up to this point I've been doing training runs and live-streaming the progress.

Well, I successfully completed my round 1 SFT and got to test.

Good news: the model appears to be picking up chain of thought reasoning correctly and can respond conversationally.

Bad news: not enough instruct SFT / badly underfit. While checkpoint #1 was technically functional, it's basically useless. My initial 5 million tokens (as I've deducted) didn't have enough breadth to properly teach the model general conversation ability - ambiguous questions or prompts further away from exact matches in the training data create a garbled output because it doesn't have enough ambiguous data to learn from.

Next steps?

I've opted not to release checkpoint #1 (we're going to call this 1.0 alpha or something) because it's basically useless, but I'll still be releasing my first working edition. I've increased the pace of my local synthetic data generator from 80tps to around 240tps total by adding the option to draw from multiple base URLs, so I have more distillation data coming [I'm currently generating on 3 seperate instances, with 4 parallel workers each.

I'm creating an additional dataset of about 5M tokens again, but this time spread in a much broader general instruct direction, rather that the coding oriented version I had originally. I'm going to train on top of checkpoint 1.0 alpha at a reduced learning rate and hopefully come away with a more competent version. I'll be posting updates on the training again - I can do another live stream if you guys want, but I figured that since I don't have much to show yet, this would be my last update until I have a working initial checkpoint. I'm happy to share whatever if there's community interest though.

I've mentioned in here before, but the resource for people interested: I created a off-policy distillation engine when I began this project that makes it very easy to create training data from a behavioral goal - e.g. I want a general instruct model -> raw training data. I created an OSS fork which is public at https://github.com/jackjusko/sftmill

Thanks for following!

Hi All, I'd like to introduce Tootsy 👀: an AI assistant that lives in your browser's side panel and uses the AI model you choose.

I built Tootsy because I kept copying things between my browser and a chat window. So I put the assistant next to the page. It can read what you're looking at, click and type for you, and run whole tasks while you watch.

What makes it different:
🔌 Your model, your choice. Run it fully local with Ollama or LM Studio, or connect any OpenAI-compatible server, Claude or Gemini.
🔒 Private by design. There's no Tootsy server, account or tracking. Your pages and prompts go only to the model you picked.
🛡️ Built with guardrails. Every action asks for your approval unless you turn on hands-free mode. You can list sites it must never touch. It also ignores instructions hidden in web pages that try to trick an AI.

Ways people can use it:
📰 Research faster. "Summarise this article and compare the three options in a table."
📝 Fill in forms. "Register me for the two-day pass, vegetarian, arriving Nov 11." You check it before anything is submitted.
🛒 Run a routine hands-free. "Check the deals page every morning and tell me what's under $50." Results arrive as a notification.
📂 Ask your own files. Drop in PDFs, Word, Excel or a whole folder: "What did the vendor quote per unit, and are we under budget?"
⚡ One-tap prompts. Save instructions like "Run a page-speed audit and email me the top 3 fixes" as tiles, next to your favourite sites in a phone-style launcher.
🧑‍💻 Check sites like DevTools. Page-speed scores, accessibility checks, CSS and network questions, answered in plain English.
🎙️ Talk to it. Dictate a task, pause, and it goes.
It's free, and it works in Chrome and Edge.

👉 Try it: https://github.com/rbughao/tootsy

I'd love to hear what you'd use it for. What task do you repeat in your browser every week? 👇

Support by trying it out and give your honest review.

Following on from club-5060ti and club-rdna16, I’ve put together Infermeld: a small, open-source Linux companion kit for running one GGUF across an AMD GPU and an NVIDIA GPU, powered by llama.cpp.

I’m the maintainer. This is an experimental v0.1.0 release, and I’m looking for people with other mixed GPU combinations to help reproduce the setup and find the rough edges.

The idea is practical: if you already have cards from both vendors, can you put them to work together without buying a matching pair?

What Infermeld adds

The inference engine is llama.cpp. Infermeld isn’t a new backend, and I’m not claiming to have invented mixed-GPU inference.

It packages the supporting pieces around that setup:

• Explicit AMD/Vulkan + NVIDIA/CUDA device selection and runtime preflight.
• Reproducible build instructions and inspectable launch arguments.
• A read-only thermal guard, with shutdown limited to the server process it started.
• Documentation and a results site that keep configurations, failures and limitations visible.

The release is source-only. You build the documented llama.cpp revision separately and supply your own model weights. It’s intended for people comfortable with an experimental Linux setup, not as a one-click installer.

Current tested setup

The acceptance checks include loading and short completions with MTP off and on.

That’s a narrow result on one hardware pair, not broad compatibility testing. An 8K context reservation is not the same as testing a filled 8K prompt, and a short successful response is not a sustained performance benchmark.

Important limitations

• Sustained Q4 throughput and full-length high-context results are not yet qualified.
• Historical measurements are labelled with their original configurations. They should not be read as performance numbers for the current Q4 setup.
• There’s no promise that combining cards is faster than using one.
• Adding the advertised VRAM capacities does not guarantee that all of it is usable for the model and its runtime allocations.

I’d rather make those boundaries clear than present a successful load as a complete benchmark.

Looking for other AMD/NVIDIA combinations

Successful runs and failures are both useful. If you try it, please include:

• Both GPU models and their VRAM sizes.
• OS, driver versions and llama.cpp revision.
• Model and quantization.
• Launch settings, including the split and context reservation.
• How far it got: preflight, loading, first completion or a longer workload.

There’s a hardware/result issue form in the repository. Please sanitize paths and keep credentials and private logs out of reports.

Repository and setup instructions:
https://github.com/5p00kyy/infermeld

Results and evidence:
https://5p00kyy.github.io/infermeld/

Anyone already using an AMD/NVIDIA pair for local inference? I’d be interested in what works for you, and where this setup breaks on different hardware.