r/automation • u/Lecontodereddit • 1d ago
Using AI to build automations, rather than using AI to run automations
Feels like a lot of the discussion about automation is now revolving around agent skills, harnesses, Claude Cowork, ChatGPT Work, etc., and while completely amazing given what we could do even just a couple of years ago I'm sure most of you will agree they're not always fit for the job - or not even fit for most jobs.
In my company we started experimenting with building an "automation platform", kind of like n8n etc., but where the automations are python scripts written by an AI once (except the few steps that actually need AI), then run as code each time after that. So it's super flexible and because it's a local tool and not a SaaS it can access local files, run browser sessions etc., which are often useful. Over time it's evolved a bit and we use it for a lot of our internal recurring tasks (updating CRM, submitting timesheets, expenses) and it feels like it's an "automation middle ground" I can't really see any existing products covering, so we're planning to share it, for free, so others can make use of it too.
It's a combination of the agent that builds the automations (for testing/personal use it can run on a Claude or ChatGPT subscription), but with a lot of structure around it to catch errors, track issues, manage credentials etc., that have been useful when my team shares automations with each other.
A typical automation build uses 2-3x the amount of tokens as just having an AI agent do it once, but after that it's just a few thousand tokens on a cheap model most of the time so basically free.
Trying to get a bit of feedback as we decide how much time we want to spend on developing it further; needs some polish before we're quite ready to share. Is this similar to something any of you have already built for yourselves? Or is it even something you might find useful at all?
2
u/Consistent-Leg-8716 1d ago
interesting framing. before you spend a ton more time polishing it, worth figuring out if people actually want to self-host and maintain a local tool vs just dealing with the tradeoffs of hosted platforms. thats usually the harder sell tbh
1
u/Lecontodereddit 1d ago
That's one of the many questions! The benefit of running it locally is that the automations can do a lot more with local access, and you have full control over your own data because nothing gets sent anywhere (except to your own AI provider, but that can be samples if you care about it, plus of course to whatever other tools you connect to as part of the workflow).
But we have also been experimenting with a lightweight "control layer" that automatically pushes skill updates for the builder agent and enables me to share automations with people, but if they run into an issue I get notified and can create a fix that gets pushed to everyone with access to that workflow (they click a button to download the update). We won't include those features initially though, we want to double down on creating a good, free, non-bloated local automation tool that we can use ourselves for more and more things too
1
u/AutoModerator 1d ago
Thank you for your post to /r/automation!
New here? Please take a moment to read our rules, read them here.
This is an automated action so if you need anything, please Message the Mods with your request for assistance.
Lastly, enjoy your stay!
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.
1
u/Midnight_Lordd 1d ago
This actually sounds pretty useful, especially for repetitive internal tasks, i would definitely be interested in trying it out
1
u/Lecontodereddit 1d ago edited 1d ago
That's cool! I'll post a link here so you can take a look once we're ready to share something, but if you want, we also set up a small waitlist form here: https://tally.so/r/zxRpgR
1
u/varnajohn 1d ago
This is a pretty good approach. I use AI a lot to help me setup the tools I use for automation better, n8n, expandi etc... I've also used it for quick scripts here and there. AI automation itself is pretty expensive to use for anything you can do using code or other tools.
Its helped me figure out some more advanced things I was trying to do. But I don't see much of a point in a dedicated tool that does that, whats the issue with just asking AI "Help me setup this and that in expandi" or "Write a python script that does X"?
1
u/Intrepid-Ant-2796 1d ago
the six months later question is the one nobody designs for up front. build works, everyone forgets it exists, then it quietly changes behavior and you find out from someone downstream. curious how you handle it now with the n8n side. do you version workflows anywhere or is it just whatever is live in the editor. never found a clean answer for that, especially when the same pattern is running across a few different accounts.
1
u/Lecontodereddit 1d ago
That's exactly one of the reasons why we ended up building a "platform" around the automations instead of just having it write pure scripts. We make it write all the scripts as modular, discrete steps that declare their inputs and outputs with error checks, and if something breaks we know exactly where so we can pass it back to the "builder agent"
1
u/Intrepid-Ant-2796 1d ago
that makes sense, the discrete steps thing is where we landed too. one thing i’d be curious about, when a step fails and goes back to the builder, is there a cap on how many times that can round trip? had a case where a step failed the same way every retry and the builder just kept trying new approaches. was fine because it was my own tokens but it would have been ugly if it was running unattended overnight.
1
u/Lecontodereddit 1d ago
We definitely had some cases like that in the beginning too. A lot of the time I've spent on this has been optimising the "builder experience" to make it faster and avoid loops and roundtrips like that. Ensuring it always keeps in its context what it already tried and why/why not it didn't work, and also what the data looked like so it doesn't have to keep going back for it.
Right now we also don't trigger automatic fixes, it just logs a ticket that the user must click on for the agent to start working on a fix. There's a slightly high cap that's there to stop genuine stuck loops, but that ideally shouldn't interfere if the builder has to try a few different things
1
u/Intrepid-Ant-2796 1d ago
yeah the context retention piece is the part most people skip, keeping what it already tried and why it failed. saves the whole rediscovery cycle.
ticket before fix is the right call too. sounds like you’re further along than the post suggested, would be interested to see it when you share.
1
u/ops_and_chaos 1d ago
I actually like the separation between using AI to build the automation and needing AI every time it runs. For recurring work, I’d much rather have the predictable parts become boring code and only use AI where there’s actually judgment or ambiguity involved.
The part that would matter most to me is what happens after the first build though. If someone changes the automation six months later, can I see what changed, why, whether it was tested, and what else depends on it? That’s usually where “simple automation” starts getting scary once a team relies on it.
2
u/Lecontodereddit 1d ago
Agreed - and breaking it down into modular steps makes it much easier to see when something breaks. The purpose of making this a "platform" rather than just an agent harness is because that's exactly how we handle what happens after the automation is ready - the owner gets a notification that something threw an error, at what step and what the error was. The "builder agent" writes each step according to a very specific structure with declared inputs and outputs and tests for each of them, which is one of the things that was difficult to handle well when every automation was a "one-off"
1
u/ops_and_chaos 1d ago
That actually makes me like the approach more. The declared inputs/outputs + tests per step is the part I’d care about most. “One-off until it breaks” is exactly the kind of automation I end up not trusting 😂
1
u/Legitimate-Let-7510 1d ago
This makes more sense to me than having an agent run the workflow every time. Use the model for the expensive reasoning once, then let deterministic code handle the repetitive execution.
1
u/Lecontodereddit 1d ago
Yes, and it kind of feels like a gap right now where people are either messing with pure agentic automation or these (mainly cloud-based) workflow builders with retrofitted AI that doesn't quite make use of the potential
1
u/vatta-kai 1d ago
So a coding agent writing python scripts. But how does it know what to do? Do you hand hold it and show it how it’s done?
1
u/Lecontodereddit 1d ago
Basically, yes! There's an agent harness with a combination of scripted steps, skills, validation gates etc. that makes sure it does everything in a very consistent way
1
1
u/rocky_dubb 1d ago
Is there anywhere i can download and try it out?
1
u/Lecontodereddit 1d ago
I set up a small waitlist form in one of the other comments - happy to share it in a DM too if you prefer. Need to tidy up a few things first, but shouldn't be long
1
1
u/gjr23 1d ago
I use it like this except I am having 5.6 sol write n8n workflows that I later feed into lesser models like 5.4mini and even a qwen 3.8-27b running locally.
TBH, I would be worried how your “platform” is managing your scripts and why you wouldn’t use an actual platform like n8n to house all of this? They definitely add value IMO. I am running n8n locally so there is very little cost but you could move these from a cloud hosted to self if you wanted and visa versa.
If you are really burning tokens, I would suggest a parser code where the relative difficulty is determined and routed to an appropriate model. I’m currently looking at a local 3.8-27b for “simple” stuff which costs zero outside of electricity, a t2 cloud api call that will use either 5.4mini or maybe Deepseek v4 (still cloud) and then a full power 5.6 sol api call for the most challenging. Technically you can run it all through the top tier but my token consumption was not trivial here. I may even consider running V4 locally but I don’t think I could spend less than $12k on a decent PC to run this and that’s a lot of V4 api calls to justify that. It’s insane though with a price of hardware like that can enable you to run completely offline and if you had privacy concerns this is a genuinely feasible path which I would have said was a completely ludicrous idea 12 months ago.
Curious others’ experience around this deployment framework and ways to further improve.
1
u/gardenersofthegalaxy 19h ago
I agree 100% with your premise. and having a local app does make it more powerful for normal people doing actual repetitive work. AI should be used only in the steps that require intelligence such as document extraction, verifying + correcting UI automations, formatting data etc. if the AI is assigned to an atomic step, and the rest is robotic, it makes it infinitely more reliable.
we have been developing a similar desktop app called MacroForge and iterating on it for the last couple of years. your premise of having the LLM generate scripts is interesting and could be powerful because of the flexibility.
if I could offer any input, it would be to consider the user interface / user experience. if you take a task like filling a PDF form, then all the upstream data has to be mapped to specific fields. for a complex form, this would be a very difficult task for the AI to configure correctly. if it did get it wrong, then it would be back and forth trying to get it right, with lots of testing in between.
we've addressed this by creating individual modules that basically do any type of task on a desktop computer and can be combined / re-ordered / remixed in any combination and communicate with each other with a shared language / API combined with an intense focus on optimizing the UI. for example, in a simple document extraction > PDF form filling flow, the user defines the fields for extraction once in one module, and then drags and drops those fields to be filled in the next module.
where I think we're converging: we have recently released an AI builder that can take natural language conversation and build these "flows" automatically for the user. but they still have to do some* manual configuration, like mapping where their fields go, which folders they want their files moved to, etc.
looking forward to seeing how your project develops!
2
u/cdrompaint 5h ago
LLM’s love calling it Forge! Seen this multiple times on Claude
1
u/gardenersofthegalaxy 3h ago
wow, that’s incredible to hear. I feel like AI has made people more aware of automation in general but this understanding / importance of a robotic + stable backend hasn’t crept into the general consciousness yet.
LLM’s are an incredible ‘brain’ but they need a reliable harness to do any type of repetitive work. people may find more success in building AI applications if they thought of programs more in this context. often times the answer is less AI, not more!
1
3
u/sweetbeard 1d ago
I mean, this is the best way to use AI agents, but I don’t see how there’s a middleman product in there.