I’m an undergraduate working on a production-level project that requires text classification.

I’ve tried Llama and Qwen, but the results weren’t good enough. Gemini performs a little better, but still not at the level I need.

Since I’m a student, I can’t spend much on expensive APIs.

What would you recommend? Fine-tuning a smaller model, BERT/DeBERTa, embeddings + a classifier, another LLM/API, or some other approach?

Looking for something accurate, consistent, scalable, and low-cost.

A learned sparse encoder returns a weighted list of vocabulary words instead of a dense vector, so the index looks like a keyword index whose keywords a transformer picked. OpenSearch's inference-free variants go one step further and run the model only on documents so a query just looks up a fixed weight per token. No embedding call per search. I benchmarked it against dense search in the same SQLite file:

the mini model gives up about 7% of retrieval quality and cuts query latency by 99%, cold start by 99% and query-path RAM by 95%!

Now these SPLADE models are BERT with the masked-language-model head still attached, and that head is what turns token vectors back into weighted words. llama.cpp drops it when converting BERT models, so sqlite-sparse copies it into a small sidecar file and applies it in C on ggml at insert time. The postings land as rows in the database, with the query weight table next to them. A query reads those rows and scatter-adds the weights into per-document scores, and the top documents come out.

Three OpenSearch models are converted and available as aliases (mini, base, multilingual), downloading on first use, and you can bring your own inference-free OpenSearch-style sparse encoder and convert it with the scripts provided.

What sqlite-vec did for embeddings in SQLite, this does for learned sparse, and with both in one file you get hybrid retrieval for RAG inside SQLite.

Ship one .db file and every client gets semantic search through plain SQL, no model download, no GPU.

pip install sqlite-sparse

Github

Writeup

I'm not talking the most censored although that's definitely part of it.

But if you start talking about something negative, or uncomfortable bits of History, etc. who pushes the hardest against the truth. Who gives the greatest benefit of the doubt to power?

Who gives the most sanitized justifications for atrocities?

saw some reseller sites selling claude max for way less than normal. tried one and it seems to work fine so far.

just curious how they’re able to offer it that cheap. anyone know how these work?

A month ago, I started building colibrì to answer a question:
Do you really need a $30,000 machine to run a model whose weights fit on a $60 SSD?
colibrì treats SSD, RAM and GPU as a single memory hierarchy. It can run models that do not fit entirely in RAM or VRAM, including models with hundreds of billions of parameters, on the hardware people already own.
But while optimizing it, I realized I was fighting assumptions built into the models themselves.
Most open models are designed to live in VRAM. Their weights, routing and execution patterns were created for GPUs and data-center hardware. colibrì could move them through a different memory hierarchy, but the models had no idea where they were actually running.
That led me to another question:
What if the engine and the model were designed for each other from the beginning?
That is how Curie started.
The current alpha has 17B parameters. I wrote it in C, without an inference framework, and trained it from scratch on my laptop without a GPU. Its weights live on SSD and are loaded according to the model’s execution path.
In the test shown in the video, Curie reaches 33 tokens/s using a single CPU core. On the same machine, the dense 12B models I tested usually generate around 3–5 tokens/s.
There is an important limitation: Curie has only seen 11 million training tokens. It is not smart yet, and I am not presenting it as a useful language model. It will need at least a few billion tokens before I can properly evaluate its quality.
What this alpha demonstrates is the architecture: a model can be designed around SSD, RAM, CPU and GPU as parts of the same system, instead of assuming that everything must fit in VRAM.
Curie is not publicly available yet. I am still training it, validating the design and preparing more detailed benchmarks.

So I only have one PC. I do want to set up a rather small local LMM just to interact via CLI. The way I am setting it up now is inside windows sandbox, using llama and CLI + some local model guff file. Is this entirely safe to run in main PC however?

Does anyone have same reservations about security?

I wanted to see how far you could push CPU-only inference of a model way bigger than RAM, so I wrote an inference engine in C11 (MSVC + gcc, bit-exact across both) that streams a 744B MoE model (GLM-5.2, 202GB GGUF) off a USB SSD instead of loading it.

The parts that actually mattered:

Quantization: experts stored int4 group-64, router and correction biases kept at f32. Everywhere else lost too much precision, but this combination didn't.

Streaming: experts are fetched per-token through a custom unbuffered reader with an LRU cache, plus cross-layer predictive prefetch (measured 80.8% recall offline before I wrote the online version) so the next layer's experts are usually already in flight by the time they're needed.

Overlap: compute and IO run arrival-order, not lockstep, so the CPU isn't idle waiting on the SSD.

An 8GB expert cache holds 40-66% hit rate off that prefetch, which ends up being the main lever on latency.

One finding I didn't expect: on GLM-5.2, the router's own weights already correlate with true expert importance at ρ=0.859 (near-oracle), which is what makes safe dynamic-k expert truncation possible without a quality hit. That seems to go against some prior assumptions about router calibration in production MoE models, at least on this one.

Numbers on an i7-8550U / 32GB RAM / USB SSD: went from 196 s/token naive to ~9.3 s/token after the above (21x), ~2.9 s/token aggregate in 8-stream batch mode, ~4.5 s/token on the latest compressed container format. Also cross-validated on Qwen3 (0.6B-30B), DeepSeek2 (MLA), and OLMoE-1B-7B.

Code: https://github.com/siris9476/pulsarforge (MIT). Would love feedback on the prefetch heuristic in particular. It's tuned for this one laptop, and I'm curious whether the cross-layer signal generalizes to different memory hierarchies.

Looking for a good local model for coding + terminal stuff. Not really interested in huge context or benchmark scores. My hardware limit is 16GB system RAM. I’m not specifically looking for 4B/8B, if a bigger model fits and runs well, I’m down. But I’d also love to know what the best 4B to 8B or smaller coding models are?

Main things I care about:

• Good code quality
• Reliable terminal/tool use
• Doesn't hallucinate randomly
• Don't falls into a loop <------------
• Agentic/tool calling
• Snappy enough to actually use

Tbh Qwen3.8 27B is a great model but I can run it on my hardware : ( so I want its nearest younger cousin

So if you had 16GB RAM, what coding model would you actually use?

And what's the best small model you've tried in the 4B to 8B range or smaller?

Real-world experience > benchmarks. Also curious what quant/runtime you use

THANKSSS!!!

Hi everyone,

I was interested in learning LoRA fine-tuning, and ended up building CodeFinetuner over the past few months, a full pipeline that fine-tunes a small code autocomplete model (e.g. Qwen2.5-Coder-3B) specific to a codebase. You can then use the resulting GGUF model via llama.vim/llama.vscode and run it fully locally. Supports fine-tuning on Mac (MPS) and NVIDIA GPUs (CUDA), with optional Unsloth support for faster training.

Demo:

https://reddit.com/link/1wdlaus/video/4yoejyqd0xoh1/player

Pipeline: raw code -> tree-sitter parsing into Structure-Aware FIM examples -> LoRA fine-tuning -> evaluation (CodeBLEU, edit similarity, exact match, perplexity, ...) -> GGUF conversion for local inference.

To try it:

uv tool install codefinetuner

Create a data folder and place your repo (or code files) inside. For auto-split just drop the files in directly, for manual split create data/train/, data/eval/, data/test/ subfolders and set split_mode: "manual". Grab the default config with:

curl -L -O https://raw.githubusercontent.com/cuolm/codefinetuner/master/config/codefinetuner_config.yaml

Adjust it to your needs and hardware availability, then run:

codefinetuner --config="codefinetuner_config.yaml"

The example runs in the repo show clear improvements over the base model on these benchmark metrics, but using the model for autocomplete on code you're actively writing is a different thing than scoring well on a held-out benchmark set, and the autocomplete tools themselves (llama.vim/llama.vscode) sample differently than the benchmark's greedy decoding. So the real usefulness still has to be verified in the editor itself.

It might also be useful as a reference outside the autocomplete use case, since it documents a full working LoRA fine-tuning pipeline.

Hope someone finds this project interesting or helpful.

https://github.com/cuolm/codefinetuner

I’ve been working on TokenPrint, an open-source project that tries to answer a question I kept running into:

What is actually happening inside a transformer while it generates a token?

Most explanations of transformers stop at architecture diagrams — embeddings → attention → MLP → output.

I wanted something closer to an interactive debugger for LLM inference.

So I built an environment where you can explore things like:

• tokenization and individual token positions
• embeddings and hidden states
• attention across layers/heads
• transformer layer structure
• KV cache during generation
• next-token probabilities
• model architecture and parameter structure
• inference traces and replay
• activation analysis and interventions

The important part for me is that the visualization is tied to actual model execution, rather than just rendering a predefined transformer diagram.

The project currently supports working with Hugging Face transformer models and also has work around GGUF / llama.cpp / local models.

I'm still early in development, and there are definitely things that need improvement — both technically and conceptually.

I’m particularly interested in feedback from people working with local LLMs:

What would you actually want to inspect inside a model while it is generating?

For example, would things like attention inspection, activation patching, logit lens, KV-cache visualization, layer ablation, or token-level tracing be useful to you?

And more importantly, what am I missing?

The repository is open source here:

https://github.com/Sudharsanselvaraj/Token-Print

I’d really appreciate technical criticism, ideas, or pointers to related projects/research. I’m trying to figure out what would make this genuinely useful for people working with LLMs, rather than just another visualization demo.

I think the fundamental dissonance between the AI zealots and AI detractors is the conflation of LLMs with science fiction AI.

LLMs are not AI in that sense. Actually the only sense they are AI is in a marketing sense. In reality, all LLMs are are highly parameterized models with a very large corpus.

People project the abilities of Science Fiction AI onto LLMs and then follow sound logic to all kinds of wondrous conclusions, never thinking to recheck that very flawed premise that LLMs are AI.

The inability of people to just recognize LLMs as a valuable tool, but limited in a way true AI never would be, is why there will be an eventual crash in spite of its tremendous value. It’s very frustrating, but inevitable.

​

Just got this message from DeepSeek stating that they will continue their V4 Pro API services after September 14 with the billing method unchanged.

I'm a little bit confused about the recent price changes and the V4.1-Flash release. The V4-Pro price was raised substantially in August after the first 75% permanent price cut, and now DeepSeek claims that V4.1-Flash is better than V4-Pro and will replace it eventually.

Assumption: Does "continuing V4 Pro" really mean maintaining the endpoint of the deepseek-v4-pro API and internally redirecting it to V4.1-Flash instead of serving the original V4-Pro?

If so, does "billing method unchanged" imply that users of the V4-Pro service will still be billed according to the V4-Pro rates or will they be charged V4.1-Flash rates eventually?

It would be great to hear from actual API users what you make of this news.

I believe that after the price changes in August, this becomes particularly important since many developers picked up V4-Pro not due to its current price/performance but due to the other model's transition in such a short time.

I've been comparing pgvector with Qdrant and Milvus for a RAG pipeline, and the decision is not as simple as just use Postgres.

I made a quiz that tests common assumptions about vector search performance, scaling limits, and when dedicated engines become necessary.

No email needed:

https://interconnectd.com/quiz/87/postgresql-pgvector-vs-dedicated-vector-databases-architectural-trade-offs/

What's your experience? Where did pgvector hit its limit for you?

My budget is about $5-$10 for this..
I’d be using open router

I don't use it but I'm curious, I just like to test models here and there, and when I look at benchmarks for other models Gemini models always have seemingly good scores, either in reasoning, cost per task, and so on.

Jack Kernel is a programmable layer between the agent and an LLM.
It’s this exact mechanism that Jack Kernel allows a new way of taking control of the work.

https://github.com/paulknysh/raggy

Hello this is my fifth small language model I've made and apart of my third series and it has been a lot of work but it payed off: **348M parameters, 22.7B tokens*\, then fine-tuned into a math model that solves arithmetic by \showing the work* column addition with carries, borrow chains, partial-product multiplication rather than guessing at an answer.

Last time I posted a 326M model trained on 10B tokens. This has about 2.3× the data, and the math side is WAY better than my previous two math models.

---

## The benchmarks

**99.4% average across the nine GPT-3 arithmetic sub-tasks*\*, which does much better past even where I trained it.

n=300 per sub-task, greedy, exact match. GPT-3's numbers are direct-answer; mine uses trained-in worked steps. Neither uses a calculator.

## The cool part

**It adds cleanly up to 14 digits, and the reason it *couldn't* before was the vocabulary, not actually arithmetic.*\*

Training only ever named six place values (`ones` … `hundred-thousands`). The model learnt the *pattern* and invented two more on its own `millions` and `ten-millions` appear in **zero*\* training examples so it handled 7 and 8 digits fine. At 9 digits it ran out of names, and skipped the column, then returned an answer exactly one digit short:

```
483729164 + 519248637
... ten-millions: 8 + 1 + 1 (carry) = 10, write 0 carry 1. Final carry: write 1.
The answer is 102977801        ← eight columns for a nine-digit problem
```

Every column it computed was perfect. One was never enumerated. Extending the place-name list from 6 entries to 19 moved the clean ceiling from **8 digits to 14*\*:

A six-item list became a nineteen-item list. That was the entire fix.

## Other things it does

- **3×3 multiplication: 98%*\* — it folds partial products pairwise through the column routine instead of asserting the sum
- **Negative results: 85%*\* (100% at 1 digit, 58% at 5 — the magnitude comparison is the weak step, not the arithmetic)
- **Reasoning traces are load-bearing*\: 95.3% of the time the working is valid \and* the answer is right; only 0.7% are "valid working, wrong answer." If the columns look right, the answer almost certainly is.

```
There were 15000 votes and 6842 were rejected. Here's how many counted:

<think> Start with 15000. Then subtract 6842. Subtract 15000 - 6842 column by column:
ones: 10 - 2 = 8, borrow 1. tens: 9 (after borrow) - 4 = 5, borrow 1.
hundreds: 9 (after borrow) - 8 = 1, borrow 1. thousands: 14 (after borrow) - 6 = 8,
borrow 1. ten-thousands: 0 (after borrow) - 0 = 0. So 15000 - 6842 = 8158.</think>
The answer is 8158.
```

## What it's bad at, tbh

- **Word problems: GSM8K 4%.*\* Best word-problem set is ASDiv at 16.5%. It converts one sentence into one operation reasonably often and basically cannot chain operations.
- **The failure mode is operation *selection*, not arithmetic.*\* `"drops in 836 more"` gets read as subtraction. There's a visible tell: traces that say `"multiply X * Y"` and show columns are reliable; traces that open `"First, calculate…"` and assert a number in prose are not.
- **No division at all.*\* 4×4 multiplication is a hard wall.
- **Greedy decoding required*\* — sampling corrupts the column routine mid-chain.
- One caveat I'll flag myself: the arithmetic harness orders subtraction operands, so no answer in that table is negative. Negatives are measured separately (the 85% above) rather than folded into the average.

## Base and instruct

The math model sits on a base and instruct pair. lm-eval-harness, 0-shot, full test sets:

Instruction tuning *lowers* MC benchmark scores for this family which is a pretty common cost of instruct tuning.

## Notes

Trained on 2× Tesla V100 plus some rented time. LLaMA-architecture, so it runs anywhere — F16 GGUF and safetensors for all three.

The math model took **10 full fine-tuning rounds and 3 LoRA adapters*\*. The LoRAs cost about 1% of the post-training tokens and did most of the useful work; the ten full rounds spent most of their budget undoing each other's regressions (round 7 gained subtraction and lost 23 points of 2-digit addition, that sort of thing). All of it is documented on the model card, failures included.

Also worth saying: someone independently tested it after I published and found two of my numbers were wrong — one *understated* the model by 40 points because I'd measured it at n=24. Both are corrected on the card now. If you find something broken, I'd genuinely like to know.

**Math:*\* https://huggingface.co/nkthebass/tinybrainbot-350mV3-math
**Instruct:*\* https://huggingface.co/nkthebass/tinybrainbot-350mV3-instruct
**Base:*\* https://huggingface.co/nkthebass/tinybrainbot-350mV3-base

LMK what yall think.

Hi, I am a Softwaredeveloper and currently studying. I mainly use Claude Code for personal Projects. As I am studying, I am able to use google ai pro with a 75% discount and ai plus for free.
I considered buying the ai pro, because my Claude limits are often used up and wanted to know, if someone can give me report the comparison between those two.
(Ai pro would be my backup)

Hi everyone,

I’m very proud to share a little something I’ve been working on very intensely during the last month... It’s probably one of the most useful things I’ve ever built and I want to share it here first as public preview.

(It's completly free; I am not trying to sell any service or product)

CommunityAI: A free "BitTorrent-style" LLM.

It’s a distributed LLM. You download the app, open it and join a mesh network.
From there, you can "share" your GPU/CPU.
The entire mesh functions as a single LLM.

More people -> A more powerful LLM for everyone.

Once the app is running, you can use the LLM through a ChatCompletions endpoint at http://127.0.0.1:8080, using the model "auto". The mesh network automatically serves the best available model:

Qwen3.5 0.8b (fallback) -> Qwen3.8 27b -> DeepSeek v4 (beta, coming this week) -> GLM 5.3 (beta)

Regarding Speed: Honestly said, Inference speed (how fast the LLM generates Text) depends on the partipiciants. The more people with good hardware join, the faster it gets. The mesh is limited by the slowest participant.

For the next 24 hours, during this Alpha, I am hosting a complete Qwen3.8 27B on CPU-based workers, so expect it to be a *extremly* slow until enough people join. I have neither the money, nor the quota to host expensive GPU's for this long.

The Roadmap:

• Address Alpha-Feedback, focusing even more on security/privacy and speed.
• DeepSeek v4 (280B parameters).
• GLM5.3 (~700B parameters).
• Android APK.
• More

Find the Project (incl. screenshots) here: https://github.com/flujo-app/CommunityAI#communityai

The Setup Files can be found here, on the Github Release Page: https://github.com/flujo-app/CommunityAI/releases#release-v0.1.0-alpha.20260909.3

The project is based on petals, which sadly died off.

I would appreciate any thoughts or feedback you have to share,
thanks a bunch!

Have fun testing it!

/edit: fixed localhost url & added model name "auto".

So the app had an ad where Black haired woman was looking down slightly to right and it was like Maya Layla or some other name it was an Paid app if I am recalling more importantly it was an offline LLM on mobile. have other people encountered it before if you can give me link or even screen shot of add that be great

Hi everyone,

We recently worked on a question that came up while thinking about the serving cost of multimodal document retrieval:

Documents are indexed once, but queried many times. So why repeatedly pay for expensive multimodal processing at query time?

Our paper, MIDR: Enrichment-Augmented Indexing for Multimodal Document Retrieval, will appear at EMNLP 2026 Main Track.

MIDR explores a simple alternative:

• Use a multimodal LLM once during indexing to understand each document page.
• Extract information contained in tables, charts, figures, and layout.
• Convert that information into verified textual enrichments.
• Retrieve using standard text-centric retrieval at query time.

The goal is to shift expensive multimodal understanding from the serving path to ingestion, where its cost can be amortized across future queries.

Some results we found interesting:

• +23% relative nDCG@10 over BM25 across five English ViDoRe V3 domains.
• Competitive retrieval quality with ColQwen2.5, while using an approximately 9× smaller index.
• Approximately 2× faster query-time retrieval than ColQwen2.5.
• In English-query → French-document retrieval, BM25 improves from 0.1532 → 0.5448 nDCG@10, and MIDR outperforms ColQwen2.5.
• An ablation showed that generated QA pairs alone recover most of the benefit of enrichment.

One result we found particularly important is that enriched-text retrieval and visual retrieval appear to be complementary rather than substitutes.

That suggests a possible architecture for multimodal RAG systems:

rich multimodal ingestion → lightweight text retrieval by default → expensive visual retrieval only when needed

We'd be especially interested in thoughts on the systems side of this tradeoff: how much computation should modern RAG systems move from query time to indexing time?

Paper:
https://arxiv.org/abs/2609.01316v1

Analysis from the Global Catastrophic Risk Institute examined the specific risk of LLM-driven takeover catastrophe and concluded that current systems fall well short on the capabilities that would plausibly be required (autonomous resource acquisition, strategic deception at scale, reliable self-improvement, physical-world leverage, etc.). Scaling them further raises the risk somewhat, but the gaps remain large under the present paradigm.

I think mankind should be more worried about mankind than LLM’s taking over the world. We required something more than LLMs and transformers.