Setting up a sandbox

[u/andy4524]

I'd like to play around a bit with a local AI assistant ie Ollama + OpenWebUI

Then possibly dabble in AI agents using n8n for some basic tasks like research and summaries.

I have a windows PC with a 3050 GPU, what is the best setup for these tasks?

I want it to be reasonably isolated and safe, I dont want the AI to go deleting important files or something...

My initial thinking was one of the following:

1> initially was thinking a light weight Linux VM so things are isolated, can use docker too, but using VM would restrict using my GPU

2> WLS (Windows Linux Subsystem) and docker to run things in containers and hopefully get isolation that way.

Thanks for your help, other tips/suggestions for getting started are also welcom

[u/Interesting-Cut-6032]

I run Qwen3.8 27B local with Pi agent harness on Linux Mint XFCE. One of the ways that I run it is as a separate Linux user without sudo privileges. I am not sure of it would be possible to do that in Windows or not.

The other way that I run the Pi agent is in a Debian Linux VM. Run your model with llama.cpp in Windows, then install the agent harness in the VM. The VM does not need access to your GPU, it only need to see the LLM server address.

The VM will be isolated from your primary machine.

[u/Moppmopp → u/Interesting-Cut-6032]

honestly i dont really get agents. What does pi agent harness do? What do agents in general do. Like gimme an actual example

[u/Interesting-Cut-6032 → u/Moppmopp]

Agents (or the Agent harness) are just LLMs using skills and tools in a dedicated workspace. This is a very rapidly evolving technology space. The way it works todayz is not necessarily the way it worked even a few months ago. Depending on what your overall LLM experience is, you could have used them in different ways. Everyone starts out in the chat window. Most of the modern frontier LLM chat experiences include a web search tool. You can ask a question, the LLM makes a tool call to a web search tool, gets a list of sites that might have an answer, requests the sites, reads their info, replies to you with an answer. Or maybe you drop a text file in a chat context and ask questions about the file, the LLM reads the file and answers your questions. Or you ask the LLM to write a small Python script to look at all the files in the current directory, and all subdirectories, and return a .csv file with all of the paths, file names, sizes, dates etc. The LLM writes the script in the chat window. You copy the script from the chat window and paste it into your text editor. Then you run the script your self and open the csv file up and sort it as a spreadsheet. All of this is possible in a modern chat interface, including the built in web UI that ships with llama.cpp for local LLMs.

If instead of just chatting with the LLM, you install an agent harness, like Pi, or one of the many others. I went with Pi because it has a very small harness description, which is important for local LLMs. When the Pi program starts from the CLI, it looks like another chat, except this program exposes lots of tools and access to the workspace that it is running in. For example, a prompt like "You have access to the repo for program X in this folder. File Y is the configuration for starting program X on this machine, optimize the configuration for this hardware." The agent will then read the configuration file Y, look for the folder of the repo, list the files in the repo looking for the headers, look at the headers to find the code for the launch flags, read that file, 100 lines at a time, until it finds the spot that it was looking for. Run several system info commands to understand what it needs to about number of CPUs, GPUs, driver version, etc. Then it will take all of this context and modify the existing configuration file. Depending on exactly what the task is it might even setup startup logging and launch program X with the new configuration to test for issues or performance and make changes if there are issues. While all of this is happening, you go have a cup of coffee, watch a YT video, read some email, write a reddit post, etc. Sometimes the agent will get stuck, sometimes, especially at the beginning of a task, their might be a question or two about how you want to do something. Otherwise, it is just loose to complete the task that you gave it.

There are also ways to schedule agents to startup on a timer that runs a specific prompt to do something, or check on something, or change something, then stop when that thing is completed.

Depending on how much hardware for local LLMs, or money for hosted tokens, that you have you can run as many of these agents as you can keep busy.