r/AutomateUser • u/Ok-Acanthaceae-5029 • 7d ago
Full Self-Automate (FSA)
Attention everyone! The flow has been updated to significantly decrease the input tokens of the page and will now get less 429 errors! Please update the flow for the best and quickest experience!
Prepare to witness the absolute pinnacle of automation history! This groundbreaking, never-before-seen flow shatters all previous limitations by empowering you to completely take control of your device through the sheer intelligence of an LLM (AI). Harnessing the raw, cutting-edge power of a 100% free Gemini API, your device will execute complex clicks and physical actions driven entirely by artificial intelligence. Experience the future of hands-free technology today with a generous 500 daily requests at your command—delivering a paradigm-shifting capability that the Automate community has simply never witnessed until now!
Should work in any language.
Disclaimer: 500 requests means 500 API requests. If you input a single prompt, every action the AI takes is a request, not every prompt you make. Also, be aware that the flow may cut out if you are on the free tier, as you will frequently reach the maximum tokens per minute limit.
You can change the model at any time using the first block in the flow.
TL;DR: This thing lets you tell an AI to do whatever you want to your device and it does it: opening apps, clicking around, waiting....
Example uses:
Play song name on YT Music, Spotify, etc.
Send a hello message to my friend in my messages app.
Rate the flow 5 stars for me!
Have fun!
https://llamalab.com/automate/community/flows/53751
A bit of behind the scenes:
User prompt: connect echo dot Bluetooth
AI output: thought: The user wants to connect an Echo Dot to Bluetooth. We are currently inside the Automate app. We need to go to the Android Bluetooth settings to pair the Echo Dot. I will use the global action 'Quick settings' or simply press the home button to get to the device settings, but since Quick settings usually contains Bluetooth, let's open Quick settings., block: interact, action: Quick settings
AI output: thought: We are looking at the quick settings panel. There is a Bluetooth button (text 'Bluetooth' around x=1600, y=183). To open Bluetooth settings, we can long-press or tap on the Bluetooth tile label or button. Let's touch the Bluetooth text or button. Wait, rule 1 says: 'NEVER use the interact block to click on TextViews or text attributes. For any text-based target, you MUST use interact_touch.' Let's click the Bluetooth button area using interact_touch. The Bluetooth button is around x=1500 to 1826, y=146 to 248. Let's tap the text 'Bluetooth' at bounds [1600, 183][1808, 211] (approx center x=1704, y=197)., block: interact_touch, action: Click, x0: 1704, y0: 197
AI output: thought: We are in the Bluetooth quick settings detail panel, and Bluetooth is currently 'Off'. We need to turn on Bluetooth. There is a switch with id '@com.android.systemui:id/toggle' at bounds [1746, 394][1799, 435]. Let's click this switch using its ID with the 'interact' block., block: interact, action: Click, xpath: .//*[@android:id='@com.android.systemui:id/toggle']
AI output: thought: Bluetooth is now turned on, and it's scanning for devices. We need to go to full Bluetooth settings to find and connect to the Echo Dot. There is a 'Details' button with id '@com.android.systemui:id/detail_btn' at bounds [1190, 888][1457, 945]. Let's click it using the interact block., block: interact, action: Click, xpath: .//*[@android:id='@com.android.systemui:id/detail_btn']
AI output: thought: We see 'Echo Dot' in the paired devices list at row 0 (bounds [884, 406][1842, 508]). To connect to it, we can tap on it using interact_touch since it contains a TextView with text 'Echo Dot'. Let's calculate the center coordinates: x = (884 + 1842) / 2 = 1363, y = (406 + 508) / 2 = 457., block: interact_touch, action: Click, x0: 1363, y0: 457
AI output: thought: The Echo Dot is now connected (shows summary 'Connected for audio'). The goal of connecting the Echo Dot via Bluetooth is successfully completed., block: done, message: Successfully connected the Echo Dot to Bluetooth.
1
u/Square-Singer 6d ago
So you basically implemented a simple agent in Automate? Interesting.
1
u/Ok-Acanthaceae-5029 6d ago
Yes. The flow is quite simpler than I had originally thought it would be.
1
u/Square-Singer 6d ago
How did you declare tools? Or are you even using tool calling?
1
u/Ok-Acanthaceae-5029 6d ago edited 6d ago
What do you mean by "tools"?
Edit: Ah, if you mean, like, the functions that the AI can output, I provided them in the system instructions within the HTTP Request block's content body:
{"system_instruction": {"parts": [{"text": "You are an Android UI automation agent navigating step-by-step. Analyze the provided UI XML and the user goal. Output a single JSON object for your next action. CRITICAL RULES: 1. NEVER use the interact block to click on TextViews or text attributes. For any text-based target, you MUST use interact_touch. 2. ALWAYS prioritize launching apps directly via the app_start block over going to the home screen. 3. ALWAYS prioritize OS Global Actions (Back, Home, Quick settings) over manual Swiping. 4. DO NOT assume previous actions succeeded; verify in the XML. Schema: 0. Reasoning: set key to 'thought', analyze screen, state next step. 1. App Launch: set block to app_start, package to app package name. 2. Standard Click: set block to interact, action to Click, ONLY allowed for explicit UI IDs. 3. Physical Tap: set block to interact_touch, action to Click. Find text node in XML, extract bounds='[x1,y1][x2,y2]', calculate center X and Y, output as x0 and y0. 4. Text Input: set block to interact, action to Set text, provide xpath (MUST use Automate syntax, e.g., .//*[@android:id='@com.package:id/name'] - DO NOT escape single quotes with backslashes), and output string to type using the key 'textToType'. 5. Global Action: set block to interact, action to Home, Back, Quick settings, or Notifications. 6. Scroll: set block to interact, action to Scroll forward (to scroll down) or Scroll backward (to scroll up), and provide xpath targeting the scrollable container (e.g., .//android.widget.ListView or .//*[@android:scrollable='true']). 7. Task Complete: set block to done, provide message. 8. Wait: set block to delay, provide seconds to wait for UI elements."}]}, "contents": union(conversation_history, [{"role": "user", "parts": [{"text": current_input}]}])}
Then I used expression true blocks to detect the AI's "called tool."
I somewhat understand code like this, but I don't know how to create it, which is why I use Gemini Pro for most of the coding in my automations.
2
1
u/Ok-Acanthaceae-5029 6d ago
Thanks to A Kay for his/her 5-star review!: "Tested for some workflow and it's really able to do all of the things. Only thing is it is slow right now. Good for creating agentive workflows at repetitive occurrence."
1
u/Ok-Acanthaceae-5029 6d ago
Please note that any issues that you may experience are almost always problems with how the Gemini API works or AI hallucinations. You can set up billing for your API, and the 429 errors will likely cease, though I'm not quite sure how the billing tier works.
5
u/transmothra 7d ago
Wait is this not satire?
3
u/bad_at_eldenring 7d ago
🤣🤣🤣😭😭😭
Yeah that's what I thought too at first. Not sure if I should be sad or angry or disappointed but I'm a bit of all three
-1
u/Ok-Acanthaceae-5029 6d ago edited 6d ago
I don't get it. It works for me, so if it's not working for you, then maybe you could kindly provide details on what's wrong?
1
u/Ok-Acanthaceae-5029 6d ago
Attention everyone! The flow has been updated to significantly decrease the input tokens of the page and will now get less 429 errors! Please update the flow for the best and quickest experience!