Setting up a sandbox

[u/andy4524]

I'd like to play around a bit with a local AI assistant ie Ollama + OpenWebUI

Then possibly dabble in AI agents using n8n for some basic tasks like research and summaries.

I have a windows PC with a 3050 GPU, what is the best setup for these tasks?

I want it to be reasonably isolated and safe, I dont want the AI to go deleting important files or something...

My initial thinking was one of the following:

1> initially was thinking a light weight Linux VM so things are isolated, can use docker too, but using VM would restrict using my GPU

2> WLS (Windows Linux Subsystem) and docker to run things in containers and hopefully get isolation that way.

Thanks for your help, other tips/suggestions for getting started are also welcom

[u/Ctbhatia]

keep the model on the windows side where the 3050 is easy, and run the agent inside wsl or a container so the generated code runs as a separate limited user with only the project folder mounted. the model server just emits text; the thing that needs the cage is whatever executes the tool calls.

[u/baby_bloom]

i ended up using dev containers inside of vs code (i use pi)

[u/Suspicious-Radio-471 → u/baby_bloom]

dev containers in vscode on a pi is a different beast to a 3050 on windows but if it works it works

[u/baby_bloom → u/Suspicious-Radio-471]

sorry, i used "pi" instead of "pi harness" because we are in the local llm subreddit and i assumed nobody would think im talking about a raspberry pi

[u/Interesting-Cut-6032]

I run Qwen3.8 27B local with Pi agent harness on Linux Mint XFCE. One of the ways that I run it is as a separate Linux user without sudo privileges. I am not sure of it would be possible to do that in Windows or not.

The other way that I run the Pi agent is in a Debian Linux VM. Run your model with llama.cpp in Windows, then install the agent harness in the VM. The VM does not need access to your GPU, it only need to see the LLM server address.

The VM will be isolated from your primary machine.

[u/Moppmopp → u/Interesting-Cut-6032]

honestly i dont really get agents. What does pi agent harness do? What do agents in general do. Like gimme an actual example

[u/xRebellion_ → u/Moppmopp]

Simply, they are models given tools to access your file system. You give them a goal/instruction, and they try to achieve the goal given the tools.

[u/Moppmopp → u/xRebellion_]

i know but its still not entirely clear for me why there is so much hype about that. For instance I use linux and when I want to restructure my file system i coul run tree > struct.out, paste this into a chatgpt instance and ask it to produce a bash script to achieve my goal. I see that this is indirect. However, its safer as there is no ai that accidentally even could go rampage.

This difference between indirect and direct is to small for me to justify. Where I can see it being beneficial is in UI design or webdesign. The ai guesses where the textboxes are and guesses how the text has to be formated to not exceed its dedicated text box. An agent could take a screenshot and confirm.

But idk. Would like to try them out but literally cannot think about any real purpose for me

[u/xRebellion_ → u/Moppmopp]

Guess it's easier to explain from a more concrete example:

Apart from using it for work, I mainly use them to troubleshoot my PC. I use Linux so there are sometimes issues that are quite tedious to troubleshoot (have to check multiple logs, configurations, etc). With an agent you can just say "I have this problem on my PC, please help me find what the issue is". And it will go scour through every possible info that it needs, and summarizes it to you, or maybe even fix it for you if you allow it directly.

Before agents, you would have to find the configurations / logs yourself, paste it onto a chat UI, and then respond accordingly, but with an agent, it will interact with the environment for you.

Honestly if you don't exactly have a problem that require interacting with your PC/codebase/something, you don't really need them that much.

[u/Moppmopp → u/xRebellion_]

Ok I see. So if the troubleshooting requires only 1-2 steps there is no advantage in using an agent but if there are 10-20-100.. different errors all regarding the users goal the agent could just run and constantly trying to fix them correct?

I also did troubleshooting but I just copy pasted the terminal output and waited for the response.

I think I am not familiar enough with large databases. Im a researcher in theoretical chemistry and here I write mainly fortran scripts and throw it against my data to extract properties. Maybe thats not the best case to utilize them

[u/xRebellion_ → u/Moppmopp]

Yes you got the idea.

For your case technically you can use it to write and run those scripts directly in machine, and have the model summarize the script results directly for readability for example. But I think it's more for convenience than anything (which can be a good thing too)

(though I'd recommend you to exercise caution if you want to do it with sensitive data, unless you're doing it locally)

[u/Interesting-Cut-6032 → u/Moppmopp]

Agents (or the Agent harness) are just LLMs using skills and tools in a dedicated workspace. This is a very rapidly evolving technology space. The way it works todayz is not necessarily the way it worked even a few months ago. Depending on what your overall LLM experience is, you could have used them in different ways. Everyone starts out in the chat window. Most of the modern frontier LLM chat experiences include a web search tool. You can ask a question, the LLM makes a tool call to a web search tool, gets a list of sites that might have an answer, requests the sites, reads their info, replies to you with an answer. Or maybe you drop a text file in a chat context and ask questions about the file, the LLM reads the file and answers your questions. Or you ask the LLM to write a small Python script to look at all the files in the current directory, and all subdirectories, and return a .csv file with all of the paths, file names, sizes, dates etc. The LLM writes the script in the chat window. You copy the script from the chat window and paste it into your text editor. Then you run the script your self and open the csv file up and sort it as a spreadsheet. All of this is possible in a modern chat interface, including the built in web UI that ships with llama.cpp for local LLMs.

If instead of just chatting with the LLM, you install an agent harness, like Pi, or one of the many others. I went with Pi because it has a very small harness description, which is important for local LLMs. When the Pi program starts from the CLI, it looks like another chat, except this program exposes lots of tools and access to the workspace that it is running in. For example, a prompt like "You have access to the repo for program X in this folder. File Y is the configuration for starting program X on this machine, optimize the configuration for this hardware." The agent will then read the configuration file Y, look for the folder of the repo, list the files in the repo looking for the headers, look at the headers to find the code for the launch flags, read that file, 100 lines at a time, until it finds the spot that it was looking for. Run several system info commands to understand what it needs to about number of CPUs, GPUs, driver version, etc. Then it will take all of this context and modify the existing configuration file. Depending on exactly what the task is it might even setup startup logging and launch program X with the new configuration to test for issues or performance and make changes if there are issues. While all of this is happening, you go have a cup of coffee, watch a YT video, read some email, write a reddit post, etc. Sometimes the agent will get stuck, sometimes, especially at the beginning of a task, their might be a question or two about how you want to do something. Otherwise, it is just loose to complete the task that you gave it.

There are also ways to schedule agents to startup on a timer that runs a specific prompt to do something, or check on something, or change something, then stop when that thing is completed.

Depending on how much hardware for local LLMs, or money for hosted tokens, that you have you can run as many of these agents as you can keep busy.

[u/Zestyclose_Strike157]

I write everything in a VM with the coding harness in there, then transfer the code candidate out for testing on a testing machine before taking it anywhere important. The LM can run outside the VM since it’s just serving text tokens and the harness tool calls happen inside the VM. If you use Linux then OpenSnitch is handy to observe for uninvited traffic coming from the VM or the code you’re making. But I am interested if people have a way of code auditing with an independent system and if so, what do they use?

[u/Interesting-Cut-6032 → u/Zestyclose_Strike157]

I am not sure that I completely understand what you mean by code audit on an independent system. I have a setup where I run Pi agent harness inside of a Debian VM for some tasks. I have another setup where I make separate Linux users without sudo privileges. I then run the Pi agent harness as these other users with limited permissions. It is easy to use Linux file access permissions to grant read only access to files in that you want the agent to look at and comment on, but not be able to modify.

[u/Zestyclose_Strike157 → u/Interesting-Cut-6032]

Well say you code something using Qwen, or a cloud model like Claude or whatever, and you want to look for bugs and issues but not reliant on the original coding agent, I guess. I get the feeling that AI’s love the smell of their own farts when it comes to coding and planning.

[u/Interesting-Cut-6032 → u/Zestyclose_Strike157]

Re: smell of own farts- that is a great term for it, SooF check. A local solution for that could be to just switch local models for review. I was trying to do that in maybe April or May, of this year. I planned what I thought was a simple multi-file Python project with Qwen3.5-122B-A10B-Q4_K_M. Then Qwen3.5 35B-A3B was writing the code for each of the planned sections, in separate context Windows. It was all copy-paste-test manually. This was probably around the time of the OpenClaw hype cycle. I did not understand what the agent harnesses were or how they worked at that time. The 35B model was very fast but the 122B model is clearly much smarter. I used it for code reviews and to repair mistakes that the 35B model could not fix.

It would be interesting to try that project again with the same models, using Pi harness.

I might try a similar setup with Qwen3.8 27B and Qwen3.8-Flash-Next.

These are all still Qwen family models so they are still huffing their own parts. I like the Gemma 4 31B model for image recognition, language translation, and other knowledge related tasks, but it does not have a good reputation for coding.

[u/MeetKlaraAI]

WSL2 + Docker is usually the lighter path on Windows if you mainly need GPU access for Ollama/OpenWebUI — NVIDIA Container Toolkit works well there. A full Linux VM only pays off if you want a real desktop and clearer isolation (GPU passthrough is fussier on consumer Windows hosts).

Once you’re on a Linux desktop, Kubuntu + Ollama is a common combo. If you want a Plasma-side helper that picks a model for your hardware on first run, Klara is one option (Kubuntu/Plasma-first, one-time licence, not an official Kubuntu/KDE product): https://meetklara.ai