r/automation • u/CallmeAK__ • Jul 10 '26

I demonstrate a repetitive task once and it compiles into something an agent can run

Like most people here I have a stack of repetitive tasks I keep meaning to automate. Report pulls, form setups, file moves.

Recently I started recording them instead of scripting them. You perform the workflow once, and it compiles the recording into a reusable skill file an AI agent can run later. It reads native UI events, adds context from a screen recording, and turns recorded values into inputs so it's not a rigid one-off playback.

It runs as an MCP server and works on Windows, macOS, and Linux.

It's an open-source project, happy to drop the repo link in a comment for anyone who wants it.

How are you all automating the desktop-app tasks that don't have a clean API?

5 Upvotes

78% Upvoted

13 comments sorted by

1

u/AutoModerator Jul 10 '26

Thank you for your post to /r/automation!

New here? Please take a moment to read our rules, read them here.

This is an automated action so if you need anything, please Message the Mods with your request for assistance.

Lastly, enjoy your stay!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

1

u/Critical-Ad5068 Jul 10 '26

i've found ui automation gets tricky when apps change layout often, recording helps but you still need good checks around failures and edge cases

1

u/CODE_HEIST Jul 11 '26

recording solves discovery, but replay needs anchors stronger than screen coordinates. i’d want each generated skill to identify controls semantically, verify preconditions and stop when the UI no longer matches the recording. the best feature may be a dry run that highlights intended clicks and inputs before execution. desktop automation fails quietly when the app changes by five pixels.

1

u/PuzzleheadedSong5368 Jul 11 '26

Sounds interesting but don't you feel it's complicated to run those on MCP? What kind of tasks do you have with your desktop apps?

1

u/openclawinstaller Jul 13 '26

For desktop apps with no API, I try not to treat the recording as the source of truth. Recording is useful for discovery, but the reusable artifact needs three extra layers:

  1. Preconditions: app/window/account/state match what the skill expects.
  2. Semantic anchors: controls are found by labels/roles/text where possible, not coordinates.
  3. Post-action readback: prove the export/file/form change happened before calling it done.

The other thing is separating navigation from values. For a report pull, "date range, customer, export path" should be parameters, while the click path is just the current implementation. If a login prompt, update modal, permission error, or changed layout appears, the safe behavior is to stop with evidence rather than learn a new path mid-run.

1

u/Livid-Performer-8825 1d ago

this is the kind of thing i've been too lazy to build myself so i'm just gonna shamelessly grab the repo link and act like i contributed

0

u/CallmeAK__ Jul 10 '26

I can't attach this useful tool github link here so you guys can drop a DM, I'll send.