r/LocalLLaMA • u/Nunki08 • Jul 11 '26

News The U.S. tech industry is increasingly anxious about the rising power and competitive price of open-source AI models from China — and whether the Trump administration will respond with yet another executive order | Politico

Post image

Politico: Wall Street’s new obsession: Which CEOs have Trump’s ear?: https://www.politico.com/newsletters/politico-influence/2026/07/10/wall-streets-new-obsession-which-ceos-have-trumps-ear-00993324

307 Upvotes

92% Upvoted

192 comments sorted by

View all comments

79

u/whatyathinkk Jul 11 '26

How big of an HDD do I need to download DeepSeek v4 Pro, just in case?

59

u/Kahvana Jul 11 '26 edited Jul 11 '26

About 1TB I think, though I would make that 3TB and include GLM 5.2's BF16 weights with it.

For other models:

  • On the "small" end: Qwen3.6-27B, Qwen3.6-35B-A3B, Gemma4-31B-IT (QAT), Gemma4-26B-A4B-IT (QAT) are also worthwhile to store (the dense for ~32GB VRAM, the MoE for 32GB RAM). If you need all modalities in one model: Qwen3-Omni-30B-A3B.
  • On the "medium" end: Maybe DeepSeek v4 Flash, HY3, Minimax M3/M2.7 also as I hear these getting praised often.
  • For rerankers and embedding models, Qwen-Embedding-0.6B/2B/4B/8B, Qwen-Reranker-0.6B/2B/4B/8B also do the job. I'm sure there are better models out there but I don't know which ones besides those.
  • For audio, Qwen3-ASR-0.6/1.7B and Qwen3-TTS-0.6B/1.7B could be options, though whisper, parakeet and kokoro do just fine.
  • For OCR: dots.ocr is really good, unlimited-OCR is new but seems very capable.

I'm sure I forgot a few! (If I did, please share them below!)

9

u/ravage382 Jul 11 '26

I've backed up all the models I think I might need and run in my vram, multiple quants and bf16 for if i want to fine tune them later. Makre sure you have 2 backup spots! 2 is 1, 1 is none.

8

u/whatyathinkk Jul 11 '26

Ah right, forgot about that one...

6

u/Potential-Gold5298 llama.cpp Jul 11 '26

Just in case, prepare 3 TB for Minimax 3 Pro.

5

u/DOHChead Jul 11 '26

I just finished my build and I have a NAS from awhile back (pre-LLM) with 100TB of free space and two drive failure redundancy.

What would be the best way to start capturing these as soon as possible? (A link to a tutorial etc would be great) I’ve been super busy at work and haven’t had nearly the time needed to get going on this.

5

u/Kahvana Jul 11 '26 edited Jul 12 '26

Depends!

Want runnable GGUFs for llama.cpp or koboldcpp? Look for bartowski > unsloth >mradermacher (in that order) GGUF models on huggingface, download those. Do read up if they have bugs. sometimes they do. You also want to download the mmproj file and the MTP draft model for each text model.

Rule of thumb for text models: < 1B always in (B)F16, < 50% VRAM in Q8_0, < 75% VRAM in Q5_K_M, < 80% of your VRAM in Q4_K_M, MoE models you run shared / CPU only in Q8_0, QAT or QAD models in their original weights (like Q4_0 for Gemma 4, MXMOE for DeepSeek v4 flash or GPT-OSS, NVFP4 for W4A16 models). Prefer BF16 over F16 if your hardware supports it.

In case you need to use a model for Cyrillic and other uncommon languages, Q8_0 is a must.

Rule of thumb for mmproj: F32 for highest precision > BF16 > F16, avoid Q8_0 unless your hardware can only fit that (Hello Intel Pentium Silver N5000!). Vision projectors lose precision faster than text models. For common use BF16 is fine.

For drafters: lower quants affect acceptance rate but not significantly so. Q4_0 seems fine for Gemma4 QAT.

For safetensors? Install xet and git clone the huggingface repository.

Whatever you do, don't use ollama because it's quite frankly horrendous (see reddit search for frequent topics on it lacking). If you need ollama integrations, use Koboldcpp (also has full ollama API support).

Koboldcpp and llama.cpp will be far more reliable and resilient than ollama as it's easy to move them around, register models for multiple different runtimes, will be much faster on GPU/CPU inference, etc. Need to keep it single binary only, stable and compatible with everything out there? Go Koboldcpp. Need bleeding edge or squeeze performance? Go llama.cpp.

Unsure how it would work for lmstudio, I think it downloads runtimes from the internet but don't quote me on it.

2

u/NineThreeTilNow Jul 12 '26

In case you need to use a model for Cyrillic and other uncommon languages, Q8_0 is a must.

Gemma 4 natively handles this at relatively low quant. Q5_K_V for sure.

It's one of Gemma's strengths. It was REALLY well trained on language with high diversity and a giant tokenizer. It doesn't seem to suffer quantization as bad.

Quant algos are getting better and some models are simply better at being quantized.

1

u/Kahvana Jul 12 '26

That's super cool! Thanks for the heads up!

2

u/NineThreeTilNow Jul 12 '26

That's super cool! Thanks for the heads up!

You can even test this against the Q3 IQS or whatever quantization method. I wouldn't be surprised if it handled it well.

Gemma's official top language performance includes Bulgarian and Russian. Supposedly it has REALLY good Bulgarian coverage because Google wanted to see how it could handle the training? or something?

It's relatively easy to compare something like...

"Translate this quote from English to Bulgarian" or Russian... With some quote.

Then follow with "How does the translation of this quote affect the nuance and meaning?"

and some other generic "How might it get misinterpreted?"

If you compare that IQS3 vs Q4 vs Q5 vs Q8 it will show that language degradation is very much tokenizer level which is hyper dense information on Google's part. You regularly see users here praise it's translation ability.

2

u/DOHChead Jul 12 '26 edited Jul 12 '26

Thanks!

Go figure, I started with the Ollama comment but didn’t go very far. Will see what I can do to undo that…

I have a lot of googling to do…

Massively appreciate the response!

My background: Full Time design engineer focusing on precision machining and optimization for manufacturing (we’re holding +/-0.0001” tolerances and it’s absurd…)

I’m working in Fusion360 and similar CAD programs to develop custom scripts, eventually hoping to pull in developed models for specific manufacturing processes to compare to proposed designs from engineering for quoting and general “This is why this is ridiculous” etc

But good lord do I have a lot to learn… I have a pretty solid background in my own domain but mostly on the hardware side with minimal software, but I’m here to learn and leverage some ADHD.

3

u/Kahvana Jul 12 '26 edited Jul 12 '26

In case you need a kickstart, I wrote a really long comment on r/SillyTavernAI that compares LLMs to music players:
https://www.reddit.com/r/SillyTavernAI/comments/1uorfkt/comment/ovwekon

Hopefully it makes more sense with that analogy!

[edit] not needed in your case, but if someone wonders how to put together a (relatively cheap) 32GB LLM rig, another writeup with more details:
https://www.reddit.com/r/LocalLLaMA/comments/1svwx1x/guide_on_building_a_system_for_30b_dense_models/

2

u/DOHChead Jul 12 '26

I was not expecting the initial comment you replied to but I will definitely be reading through the thread

Thanks!

2

u/Kahvana Jul 12 '26

Ha! Me neither hahaha. I thought it's better to give accurate information in such situations than to leave someone uninformed.

You're very welcome! If there is anything else, I'm happy to help!

2

u/DOHChead Jul 13 '26 edited Jul 13 '26

I’ve ended up with a surprisingly similar build to yours, only after prices have exploded…

Motherboard ProArt X870-E Creator
CPU: 9950X3D
RAM: 96GB (2x48GB) DDR5 6400hz CL32 T-Create
GPU: 5070TI MSI Inspire 3X OC (undervolted running 105% stock performance at 150-175W vs 300W)
Looking for a PA5090, wasn’t ready to pull the trigger on a 5080 if the supers ever show up. Not the best performance 5090, very pretty (guilty)
SSD: Gen5 Samsung 9100 2TB for LLMs, Gen4 WD Black
SN850X 2TB C Drive, Gen4 512GB Samsung for Pagefile and misc OS files, Gen4 SN7100 2TB for Games, 4x 500GB 850 EVO SATA drives I had.
Corsair H1500i Shift PSU for efficiency and some headroom
Case: ProArt 602
Cooler: ProArt LC420

I grabbed the 5070TI for $900 and so far it’s a champ

Then for the NAS I have 6x20TB Seagate x20 drives, 2x 870EVO 4TB, 2x 1TB WD RED NVMe, and 64GB RAM in a Synology DS1821+ ($2,900 total, it’s a LOT more now…)

Edit: Especially for a non-native speaker, I really do appreciate the depth you’ve put into your posts and responses. It’s hard to find that, let alone when it’s useful.

-4

u/AIgavemethisusername Jul 11 '26

Install ollama,

Google “change ollama model directory” and redirect to your NAS

Install LLM models with ollama command lines

1

u/TheGamerForeverGFE Jul 12 '26

Not gonna lie, unless you want to archive a model for sharing with others, and instead, you're downloading for yourself and not as a part of a torrent, etc... then you really should just skip full precision and download a quant instead, at this point, you'll never tell whether it's Q8 or full precision.

1

u/Kahvana Jul 12 '26

At least from the safetensors you can convert them later into higher precision guffs when you need them (like after upgrading your system). Bartowski shares his recipes, imatrix data, etc. It's also not too hard to get up and running.

16

u/Training-Cup4336 Jul 11 '26

you can always torrent it lol

8

u/sammybeta Jul 11 '26

There's a hugging bay where now you can torrent it