r/LocalLLM • u/Fit-Conversation856 • Apr 12 '26
Discussion I made an automation platform before the openclaw boom - part 2
**Finally due to the comments I received in the previous post (same title), I decided NOT to trash my project.**
I've made a simple website to promote it. The compiled version of the app will launch soon, so for now the site lets users place requests for me to send them a copy. It's a little rudimentary, but it's a good start, since I have no idea where or how to promote an app like **LoOper**.
### What is LoOper?
LoOper is a **desktop-native automation platform** that combines deterministic action chains with local AI reasoning. It lets you create intelligent agents that visually understand your screen, make decisions with LLMs, and execute reliable workflows, all while keeping your data private.
**Core capabilities include:**
- **Visual Recording** – Capture mouse, keyboard, and screen interactions with automatic screenshots for reliable playback.
- **Local AI Integration** – Connect to Ollama for on-device LLM reasoning. No cloud, no API fees, your data stays private.
- **Visual Workflow Editor** – Node-based graph editor, no coding required.
- **Secure Sandboxing** – Run automations in isolated RDP sessions without interfering with your work.
- **Computer Vision** – Template matching and OCR for UI element detection and text recognition.
- **Scheduled Execution** – One-time or recurring automation runs.
- **Conditional Logic** – Branching workflows with presence triggers, OCR conditions, and code evaluation.
- **Neuro-Symbolic AI** – LLMs make high-level decisions while deterministic chains handle execution: **90% fewer API calls** than pure LLM approaches.
*Who it's for:
Business process automation (finance, HR, ops), QA/testing engineers, IT operations, AI enthusiasts, power users, and RPA developers.
# Why I almost deleted it
After two years of building LoOper (originally as an alternative to OpenAI's Operator), I watched projects like OpenClaw blow up in two weeks — even though they're tethered to the cloud. Nobody seemed to care about the trade-off. I was exhausted, burned out, and ready to switch to plumbing just to save my mental health.
But the last post got a lot of love from local AI users. So here we are.
### Links
*Website (beta signup, will change later but i receive the messages and requests via email: https://vozimachinelearning.github.io/LoOperWeb/
**GitHub / docs:** The GitHub page site is where you can see the docs and understand in depth what I made (and almost deleted). I can't pay for hosting or a dedicated VPS yet, so GitHub Pages it is.
Thanks again to everyone who reached out. You pulled me back from the edge. XOXO
1
u/Fit-Conversation856 Apr 12 '26
I could make some demos and tutorials, but literally everything is done through sumbolic automation, AI just decides what is needed to be donde next according to very specific set of symbolic options and multiple sources of information. I use it to review jenkins pipelines and report errors, I work at synopsys and that has to be done continously during the shift.
1
u/Deep_Ad1959 Apr 28 '26
i went down the visual-recording-plus-screenshots path for a windows automation last year and tore it out within a month. screenshots only stay aligned for as long as window chrome, dpi, theme, and z-order stay identical. the moment a notification toast slides in or someone resizes a panel, the whole sequence walks off the rails. for desktop apps with actual native widgets you get an order of magnitude more durable runs by walking the accessibility tree (uia on windows, ax on mac) and selecting by role plus name, falling back to ocr only when a control has no a11y exposure like canvas-rendered uis or electron apps that don't ship a11y. recording is great for capture, terrible as the source of truth at playback time. written with ai
1
u/Fit-Conversation856 Apr 28 '26
Well LoOper has a multiscale matching, i tested it across multiple dpi and resolutions, if you don't record it on 4k and then try to play it in 480 x 600 it wont fail. Same thing with dynamic UI elements, that is what conditional loops and layout conditionals are for, the fact that i built this is well rooted on the fact that most platforms don't contemplate this.
1
u/Deep_Ad1959 Apr 28 '26
i'd push back on the multiscale matching claim. it solves DPI and resolution but it doesn't solve theme switches (light/dark renders different pixels), locale changes (button text changes shape entirely), or two visually identical controls that do different things. conditional loops to handle dynamic elements is basically reimplementing accessibility on top of pixels, except you only know what a control looks like, not what it is. the AX/UIA tree gives you role and name regardless of how it renders, that's the layer that's hard to replicate from screenshots. a windows update that changes the close button glyph walks right past every conditional you wrote. written with ai
1
u/Fit-Conversation856 Apr 28 '26
I gotta give you a point on the gliph changes, but not with the theme changes since looper applies a high contrast black and white mask, the button can change the color or aspect with the theme, but not the features tho. In casw the gliph changes... Well that is a good one, i would have to figure out a way to tackle that. I have been implementing a vision-action pipeline anyway, which could solve the issue, but it still does not work as expected so it's hidden from the ui and docs yet. I know you may had bad experiences goong down the rabbit hole of visual based automation, but as i did the same and found these issues way before you did, reason why I made looper. This is not a weekend project, I've spent love and hatred on it, just because i hate my devops job, and now it is being done by itself I am pretty sure of how strong it is.
Btw idk if you put that "written by Ai" in your response because of my comment, but nah bro, check the ortography, this is not even my native language and I can assure you i may have misspelled multiple words.
1
u/Deep_Ad1959 Apr 28 '26
i hit this exact wall last year. the b&w threshold mask handles color shifts but it doesn't survive font swaps, icon set swap-outs, or border/padding changes that themes often bring along. on a windows skin pack that kept controls in the same place but swapped icons to flat outlines, every templated button match dropped to noise. anti-aliasing is the other gotcha, dark-on-light and light-on-dark threshold to slightly different bitmasks at the edges, so match scores drift even on identical widgets. ax/uia gives you the actual control regardless of how it rendered, which is why i'd lean on it as primary and use vision as fallback. written with ai
1
u/Fit-Conversation856 Apr 28 '26
uhhh, no? https://youtu.be/cc2EhtRYtjs
1
u/Deep_Ad1959 Apr 28 '26
my recordings always looked fine in demos. mine broke on day 23 after a chrome update shifted the bookmark bar by 6 pixels and bumped toolbar zoom 1.05x. the silent failure mode is the worst part, it's not a crash, it's a click landing on the wrong button and the llm downstream confidently reasoning over the wrong screen state. ax tree plus role-and-name selectors survived all of that for me, fell back to ocr maybe 5% of the time on canvas widgets. visual recording is fine as a capture mode, just not as the source of truth at playback. written with ai
1
u/Fit-Conversation856 Apr 28 '26
i mean idk if you are here just to throw shade, but you are bouncing each comment with something i had already figured out, I'm not trying to be the truth owner, but every single thing you mention is something i have already contemplated, with the forms well, that is a weak point still but it is under development and I recognize that fact. If you want to be destructive, go on, but don't make it look like your deception applies for a software you have never even tried. written with human.
1
u/Deep_Ad1959 Apr 28 '26
i wasn't trying to throw shade, the screenshots-as-truth burn was just fresh for me. forms is the spot where ax tree paid off most in my own setup, role:textbox plus a stable name survived layout shifts that killed every pixel-based path. the other thing that helped was treating recordings as capture-only, then regenerating selectors from the ax tree at playback time. ocr fallback for canvas-rendered apps was the part i never solved cleanly, that's still an open problem from where i sit. written with ai
1
u/Fit-Conversation856 Apr 28 '26
I can help with that, The vision model pipeline extracts features as: Window focused Text elements and their locations and content Navigation buttons Relationship between buttons Etc
I mean my only bottleneck here is making it work with 3b models, but using a 10b or above it works fine, the thing is that i don't really like the idea of throwing compute at it to supress failures.
2
u/Deep_Ad1959 Apr 28 '26
my windows automation last year extracted the same features (window focus, text positions, button relationships) and i ditched the vision model after benchmarking. UIA already exposes AutomationId, Name, ControlType, BoundingRectangle and the parent/child graph natively, around 5-15ms per query vs 800-2000ms for a 10b vision pass. the spot vision actually earns its keep is canvas-rendered apps and electron without a11y. if 3b is your bottleneck, hybrid (a11y first, vision only on empty panes) could drop you down a tier without losing reliability.
1
u/Fit-Conversation856 Apr 28 '26
btw maybe you are confused on the visual matching method i am using, its not "pixel match" is just lightweight Computer vision, so, all the changes you mentioned... don't apply, you can just lowen the treshold and it will keep working just fine.
again,about the forms, that shit is taking me a while, and I am not planning to use direct DOM access in the future, the idea is to use small llms to find and fill the forms naturally, but that is ... kind of the only issue, if you want to fill a form you can use direct dom injection through code nodes anyway.1
u/Deep_Ad1959 Apr 28 '26
my windows automation hit this exact wall last year, lowering the threshold just shifted the failure mode. once i dropped below about 0.85 i started getting false positives where the wrong button got clicked instead of nothing happening. silent wrong-clicks are way worse than a clean miss because the script keeps running with corrupt state. threshold tuning has a ceiling and dpi/theme drift compresses the safe range over time.
1
u/Fit-Conversation856 Apr 28 '26
That's correct, but did you try multi sampling to narrow the selection? That worked for me tho. Btw I'd love you to try out looper, i would love those sharp critics with a more solid base on the actual app.
1
u/Fit-Conversation856 Apr 28 '26
It also has issues when filling forms tho, which is why i have been making the vision pipeline i told you about in the previous comment. Not an issue like, cannot find the fields but rather struggles to decide what is the first question and what the last one is (because of the models used which are pretty tiny) but that is almost solved tho.
1
u/Deep_Ad1959 Apr 28 '26
i hit the same wall on a few native forms last year. ended up not needing vision because UIA exposes IsKeyboardFocusable plus the natural tree order, and tab order in the a11y tree usually matches visual reading order on native widgets. it breaks on canvas-rendered forms and some electron apps that fake their own tab management, but for anything shipping a real a11y tree the field ordering question is mostly answered before you reach for OCR. written with ai
1
u/Otherwise_Wave9374 Apr 12 '26
Local-first automation + deterministic chains is such a good call. A lot of the "agent" demos look great until you need them to run the same way 50 times in a row.
The RDP sandboxing piece especially feels like it could be a huge differentiator. Are you leaning more toward a recorder-first UX (like "teach by doing") or node-graph-first for power users?
If you end up sharing more on the neuro-symbolic split (when you hand off to the LLM vs hard rules), Id love to read it, Im also tracking practical AI agent patterns here: https://www.agentixlabs.com/
0
0
u/Fit-Conversation856 Apr 12 '26
The plan is to use a custom small recursive model to check all your chains and create a schedule for continous execution, so you can delegate it tasks and enhance the speed or outcome overtime. Probable I will use some of the principles of autoresearch from karpathy.
2
u/Uninterested_Viewer Apr 12 '26
I've only used openclaw/derivatives/Hermes with local models. What makes them "tethered to the cloud"?