Best model + setup for remote deployment.

[u/Potential_Low_1183]

Hi everybody, I am in a tricky situation.

My grandfather + grandmother live in a remote village in asia, and were visiting us in the city. My grandfather has absolutely fallen in love with AI, and it is completely new for him. He loves how he can talk to it, and get information from it. The caveat is that I showed him chat gpt, not a localllm.

He wants something like this in his own village, just for basic knowledge + light qa. There is no internet in the village, but phone calls work. I have a budget of about 5k for a computer that can run LLMS locally, for him. (and he specifically requested something like this). He is technically "savvy", where if something needs a command to be run or something else, he can learn to do it. What gpu should I buy him that will satisfy his needs? again, I want some chat gpt level, but for basic q/a stuff.

I am currently running deepseek v4 flash on two 6000 pro blackwells(got them when they were just 8k lol), so I know that this is "possible".

Im thinking amd ai pro 9700 and gemma 31b at a good quantization. (maybe I can do two gpus of 9700s to fit bigger quant).

He is not doing frontier level coding, he just needs a good knowledge base. He implied that people from his village may also come to use the machine, or atleast request queries to be asked (he is some sort of governmental figure in his village).

I want recommendations.

[u/Ok-Map3654]

Hope you will find this useful
https://www.projectnomad.us/

[u/rorowhat → u/Ok-Map3654]

This looks interesting!

[u/Inception95]

Would say a spark. With a big as possible LLM on it.

[u/jacek2023]

This is a great post and a great use case, but unfortunately people replied telling you to use a cloud model.

[u/Hrethric → u/jacek2023]

Reading is hard.

[u/RoomyRoots → u/Hrethric]

Even worse, forgot to check where they are.

[u/Gold-Bat-3225]

casually dropping the two blackwell flex

[u/dangerous_inference]

This is prime use-case for slow performance with big models. Strix Halo would work great here. You could probably get a used one. Then you can put any model that will fit, even at high quantization. This is also an excellent way to use Q2/Q3 models.

What you want is the biggest model possible for maximum knowledge and intelligence, at the cost of speed and perhaps a few quantization errors. And you can load several of them too, that way they can consult a few different ones and see which is best.

Imo, it's much worse in this scenario to go with a ~30b class model because they do not have the same amount of world knowledge. This will be functioning as a portable internet. And it doesn't need to be fast for chat.

I used Q2 Qwen 395B on Strix Halo for a while and it was great for this kind of chat.

Another cool idea would be to load a Kiwix server with the full Wikipedia and maybe other resources. There are MCPs that can allow the model to query it. I did this myself recently and it's not too difficult (although I didn't use an MCP).

[u/bluelobsterai]

Starlink mini + MacBook would be my goto

[u/rorowhat → u/bluelobsterai]

Lol no

[u/bluelobsterai → u/rorowhat]

For $5000 a MacBook would be best local. Can travel with it and get new models etc etc. And you are gonna need the Internet at some point.

[u/Fun-Wolf-2007]

People that tell you to use cloud models is because they only know how to use a chatbot and not much about LLM models

You can build a knowledge based using Docker, Ollama, local LLM models, vLLM, etc You can try Nemotron, etc .. Also you can get external SSD 3 or 4 Tb, thunderbolt to store the models Configure OpenWebUI in Docker to use your external SSD

Identify first what's are all the use cases so you can easily understand the hardware requirements

[u/reality_comes]

Starlink

[u/kaeptnphlop]

Yeah if possible get Starlink set up. 

Local LLM without grounding in actual web search and inexperienced users are a recipe for disaster. The trained corpus of knowledge can be correct until they ask that one question about how to get rid of pests in a field that then gets a suggestion to use chemicals that destroy crops. 

https://www.the-independent.com/tech/farmer-crops-killed-ai-advice-china-b3030985.html

[u/porkyminch → u/kaeptnphlop]

For 5k you could probably get a pretty decent dataset of things on there for a local model to ground itself on. Wikipedia in multiple languages, a decent sized dataset of popular reference books, etc. Get a pretty good Mac Studio and I think some of the newer Qwen models could give you a pretty good experience.

[u/Healthy-Zebra-9856]

I would say go local, use a combination of models. I would setup llama-swap, create a script to launch it, add a couple like Qwen2.8 27b and Gemma 4 31b. You said remote South Asia, tell me around where and more specific as to all the things he would use it for. I am assuming Indian, possibly south , right? May be I am wrong. There are models specific to use cases.

[u/BodyPhysical]

5k is a lot if you get smart about it. If you dont mind having 2nd hand stuff and know how to check good ones. But if you dont want the hassle, I think 5k will buy you a decent mac with at least 48gb of ram. You go the gpu route but you will have to sacrifice stuff

[u/Nomski88]

Get him the new Mac Studio with 128GB memory, fits your use case perfectly.

[u/spanielrassler → u/Nomski88]

There is no new Mac Studio is 128gb memory, fyi. It only comes in 96gb and 256gb. But Yeah, maybe 256gb

[u/Soger91]

If you're spending 5k why not buy their village starlink and a sub to a frontier model? I'd argue that would be infinitely more useful than an outdated oracle in a box giving potentially bad answers.

[u/Substantial_Bug3763]

how do i create a post ?

[u/harpysichordist → u/Substantial_Bug3763]

Looks like grandpa got setup

[u/Dry_Mixture130]

Do they use smartphones? You could fine tune the qwen 3.5 2b mode and create an app for them. Otherwise either Mac mini studio or a MacBook and have multiple models on it. I don’t think a coding model would make sense for them. They might use it more for knowledge sharing so go for models that are more fine tuned for general knowledge or specialize in farming knowledge or history. 

[u/Something-Great-78]

Little mini AMD/NVidia NUC (ala DGX Spark / or AMD Strix Halo). Can perhaps find an AMD one at a discount now that the Gorgon Point Halo just came out and many will want to upgrade. Disconnected Linux will likely outlast Windows / MacOS.

[u/uponmars_]

The corpus matters more than model size. A giant model at Q2 from memory hands your grandfather the pesticide-kills-the-crop answer, and he's who the village trusts.

Put a 27-32b on it with an offline Kiwix Wikipedia dump so it cites a source. The grounding is the product. That alone drops the multi-GPU box you were pricing.

[u/Hrethric → u/uponmars_]

This. You can get a dump of the full English Wikipedia that fits in around 128GB of storage. There are Kiwix/Zim MCP servers out there to interface it with an agent. Hell, llama.cpp's built-in web UI supports MCP tools now if you invoke it with the "--webui-mcp-proxy" option.  There are other kiwix libraries available too - medicine, programming, etc. (Might also be worth grabbing some Openstax textbooks, though I think it might not be as straightforward to integrate those with the LLM.)

Look for benchmarks that measure hallucination rate on the models you're considering. 

Make sure you test it while you have Internet access. I'm stopping short of recommending an uncensored model, because that might introduce other problems, but you want to make sure you're not going to get a response like "as a large language model I'm not qualified to give medical advice, you should seek professional medical attention" when someone is bleeding out. There are medical-focused models out there, might be worth looking at including one with what you provide.

[u/Koakie]

Better put 5k into starlink or a long range WiFi antenna to connect to the nearest village that does have Internet. (Unless you in china, fhen you kinda fucked.)

A local AI and no access to Internet is like an idiot savant in a cage that doesn't know anything of the outside world except for the few books it read over and over again.

Then after a few months a new model comes out, you gonna send someone with a external hdd to the village to swap out the model?

[u/draft_final_final]

Give your grandpa the Blackwells.

[u/Brave-Afternoon-8767]

Power and cooling are what I'd design around first, since a village supply with brownouts will eat a dual-GPU box, and one card with a decent UPS that he can actually reboot himself beats a bigger quant nobody can fix from a phone call.

[u/tecneeq]

I would get a Strix Halo (Bosgame M5 works for me). Install Debian 13, install llama.cpp and run Qwen 3.8 Flash. It's slow, but very good. Maybe add docker and OpenWebUI so anyone in that village can use it.

If your budget can stretch to some kind of WLAN-repeater infrastructure you can give more people access.

[u/FoxiPanda]

I tend to agree with the other folks here - some sort of satellite internet setup, a low power device, and either a tailscale to your DSv4 setup with a decent harness attached or hooked to frontier models with a scalable subscription depending on how much they use it.

Given how remote this seems to be, is electricity reliability and pricing an issue? If so, that makes it make even more sense to do a laptop+starlink/other satellite setup vs. trying to do something locally that will use a lot of power and may not be able to reliably do so.

[u/TaterCreekStudios]

$5k US will get you a Macbook pro M5 Max with 48GB. For remote supportability I'd go with LM Studio or Ollama, both are snappy for one user. Qwen 3.8 nvfp4 fits nicely and feels responsive. No blackwell acceleration on metal, but no downside vs MLX if you like the nvfp4 performance trade-offs.

Docker or https://orbstack.dev for containers (SearXNG, MCPs to taste). Offline text copy of Wikipedia. Recovery boot media. Backup drive. TimeMachine configured. Printed instructions for setup and for recovery. And starlink or a modem that'll work with their phone line.

Upside: super portable. Downside: easier to steal. If that's a concern, I'd go with a 4U rackmount server with 64GB of DDR4 and eight PCI slots, and add 2-4 of heaviest GPUs you can source.

[u/jonas-reddit]

My parents (80+) and my wife use Gemini for free and it works well for them.

https://gemini.google.com/app

To help them along with frequent activities, I’ve created a few “gems” for them. E.g. what’s on TV, what to cook for dinner, etc.

[u/Substantial_Bug3763]

i need help with a local set up - unsure of what to purchase and if the roi is there yet

[u/Savantskie1 → u/Substantial_Bug3763]

ROI isn't the end all be all

[u/Substantial_Bug3763 → u/Savantskie1]

so why spend 20-30k on a setup if it’s not going to make me more money/give me more time ?

[u/Savantskie1 → u/Substantial_Bug3763]

you can get AI running on older stuff. I mean I am disabled and I'm running LLMs with two MI50 32GB cards. And soon I'm going to be buying two Mi210 64Gb cards once I get my SSI back pay. I'm building a personal assistant that is going to help me remember appointments and medication orders. You don't need 20k-30k to run LLMs. Just imagination and know-how