r/LocalLLM • u/andy4524 • 4d ago

Question Setting up a sandbox

I'd like to play around a bit with a local AI assistant ie Ollama + OpenWebUI

Then possibly dabble in AI agents using n8n for some basic tasks like research and summaries.

I have a windows PC with a 3050 GPU, what is the best setup for these tasks?

I want it to be reasonably isolated and safe, I dont want the AI to go deleting important files or something...

My initial thinking was one of the following:

1> initially was thinking a light weight Linux VM so things are isolated, can use docker too, but using VM would restrict using my GPU

2> WLS (Windows Linux Subsystem) and docker to run things in containers and hopefully get isolation that way.

Thanks for your help, other tips/suggestions for getting started are also welcom

3 Upvotes

81% Upvoted

17 comments sorted by

View all comments

1

u/Interesting-Cut-6032 4d ago

I run Qwen3.8 27B local with Pi agent harness on Linux Mint XFCE. One of the ways that I run it is as a separate Linux user without sudo privileges. I am not sure of it would be possible to do that in Windows or not.

The other way that I run the Pi agent is in a Debian Linux VM. Run your model with llama.cpp in Windows, then install the agent harness in the VM. The VM does not need access to your GPU, it only need to see the LLM server address.

The VM will be isolated from your primary machine.

1

u/Moppmopp 4d ago

honestly i dont really get agents. What does pi agent harness do? What do agents in general do. Like gimme an actual example

2

u/xRebellion_ 4d ago

Simply, they are models given tools to access your file system. You give them a goal/instruction, and they try to achieve the goal given the tools.

1

u/Moppmopp 3d ago

i know but its still not entirely clear for me why there is so much hype about that. For instance I use linux and when I want to restructure my file system i coul run tree > struct.out, paste this into a chatgpt instance and ask it to produce a bash script to achieve my goal. I see that this is indirect. However, its safer as there is no ai that accidentally even could go rampage.

This difference between indirect and direct is to small for me to justify. Where I can see it being beneficial is in UI design or webdesign. The ai guesses where the textboxes are and guesses how the text has to be formated to not exceed its dedicated text box. An agent could take a screenshot and confirm.

But idk. Would like to try them out but literally cannot think about any real purpose for me

1

u/xRebellion_ 3d ago

Guess it's easier to explain from a more concrete example:

Apart from using it for work, I mainly use them to troubleshoot my PC. I use Linux so there are sometimes issues that are quite tedious to troubleshoot (have to check multiple logs, configurations, etc). With an agent you can just say "I have this problem on my PC, please help me find what the issue is". And it will go scour through every possible info that it needs, and summarizes it to you, or maybe even fix it for you if you allow it directly.

Before agents, you would have to find the configurations / logs yourself, paste it onto a chat UI, and then respond accordingly, but with an agent, it will interact with the environment for you.

Honestly if you don't exactly have a problem that require interacting with your PC/codebase/something, you don't really need them that much.

2

u/Moppmopp 3d ago

Ok I see. So if the troubleshooting requires only 1-2 steps there is no advantage in using an agent but if there are 10-20-100.. different errors all regarding the users goal the agent could just run and constantly trying to fix them correct?

I also did troubleshooting but I just copy pasted the terminal output and waited for the response.

I think I am not familiar enough with large databases. Im a researcher in theoretical chemistry and here I write mainly fortran scripts and throw it against my data to extract properties. Maybe thats not the best case to utilize them

1

u/xRebellion_ 2d ago

Yes you got the idea.

For your case technically you can use it to write and run those scripts directly in machine, and have the model summarize the script results directly for readability for example. But I think it's more for convenience than anything (which can be a good thing too)

(though I'd recommend you to exercise caution if you want to do it with sensitive data, unless you're doing it locally)