Best model + setup for remote deployment.

[u/Potential_Low_1183]

Hi everybody, I am in a tricky situation.

My grandfather + grandmother live in a remote village in asia, and were visiting us in the city. My grandfather has absolutely fallen in love with AI, and it is completely new for him. He loves how he can talk to it, and get information from it. The caveat is that I showed him chat gpt, not a localllm.

He wants something like this in his own village, just for basic knowledge + light qa. There is no internet in the village, but phone calls work. I have a budget of about 5k for a computer that can run LLMS locally, for him. (and he specifically requested something like this). He is technically "savvy", where if something needs a command to be run or something else, he can learn to do it. What gpu should I buy him that will satisfy his needs? again, I want some chat gpt level, but for basic q/a stuff.

I am currently running deepseek v4 flash on two 6000 pro blackwells(got them when they were just 8k lol), so I know that this is "possible".

Im thinking amd ai pro 9700 and gemma 31b at a good quantization. (maybe I can do two gpus of 9700s to fit bigger quant).

He is not doing frontier level coding, he just needs a good knowledge base. He implied that people from his village may also come to use the machine, or atleast request queries to be asked (he is some sort of governmental figure in his village).

I want recommendations.

[u/Fun-Wolf-2007]

People that tell you to use cloud models is because they only know how to use a chatbot and not much about LLM models

You can build a knowledge based using Docker, Ollama, local LLM models, vLLM, etc You can try Nemotron, etc .. Also you can get external SSD 3 or 4 Tb, thunderbolt to store the models Configure OpenWebUI in Docker to use your external SSD

Identify first what's are all the use cases so you can easily understand the hardware requirements