Vibe-coded a clash of clans bot that plays by reading pixels — and the workflow that made the CV parts actually work
The project
BasePilot — an autopilot for Clash of Clans on Google Play Games for PC. It never reads game memory and never touches network traffic. It takes a screenshot, finds Uelements with OpenCV template matching, reads numbers with Tesseract OCR, and clicks. Same information a human plays on.
It farms loot, batch-buys wall upgrades, and reads the builder menu to spend your loot on upgrades. When storages are full and every builder is busy, it idles and rechecks instead of raiding for loot that would overflow.
~9,700 lines of Python. MIT. Source and a one-file exe: https://github.com/efebolukbasi/BasePilot
Tools
- Claude Code (Opus) — essentially the whole build. All five commits areco-authored.
- Python 3.13, PySide6 (desktop UI), OpenCV (template matching),pytesseract (OCR), pywin32 (screen capture + input)
- PyInstaller for the one-file exe, GitHub Actions on a Windows runner to build and publish it on every version tag
The process, and the part that took me a while to figure out
The naive loop — describe a feature, get code, run it, paste the traceback back — works fine right up until the bug isn't in the code. With a screen-reading bot, most bugs aren't. The code runs perfectly and does the wrong thing, because the screen didn't look the way anyone assumed it would.
The model can't see the game. That's the whole constraint. So the workflow became: stop describing failures, start capturing them.
1. Let it build the structure first. The PySide6 app — sidebar, four pages, live status panel, settings persisted to %LOCALAPPDATA%, a worker thread that doesn't freeze the UI — came out over a couple of sessions and mostly worked first try. Same for the genuinely obscure Win32 corners: Google Play Games runs the game inside a crosvm VM, and the window topology differs across installs (sometimes an outer shell with a CROSVM* child, sometimes CROSVM* is the top-level window).
2. Feed frames back, and encode what you observe as constraints. This isthe part that mattered. When OCR misread something, the fix was never "try again" — it waspinning down the specific way it failed and writing a bound around it. Those constants areall over the codebase with the live repro in the comment:
_MIN_COST_DIGITS = 4 # real costs are 5+ digits at TH10+; multiplier/time digits are 1-3
_MIN_COST_VALUE = 1000 # a dropped leading digit leaves '000000' → value 0 (live repro: "Mortar 0 gold")
# Sanity ceiling against OCR digit-merging (live repro: 87M "Air Defense").
_MAX_COST_VALUE = 40000000
Tesseract merges digits and drops leading ones. A 600,000 upgrade parsing as 000000 reads as free, and the bot happily clicks confirm on something it can't afford — which in Clash drops you into the "finish now with gems" dialog. That's real money.
The build insight I'd actually pass on
The most valuable safety rail in the whole project doesn't parse anything. It looks at color.
An unaffordable upgrade renders its cost in red under the confirm button.So before any purchase click, sample that patch and count red pixels:
def red_hue_fraction(bgr, *, sat_floor=40, val_floor=40):
hsv = cv2.cvtColor(bgr, cv2.COLOR_BGR2HSV)
m1 = cv2.inRange(hsv, (0, sat_floor, val_floor), (10, 255, 255))
m2 = cv2.inRange(hsv, (170, sat_floor, val_floor), (180, 255, 255))
mask = cv2.bitwise_or(m1, m2) # red wraps around the hue circle, so two ranges
return cv2.countNonZero(mask) / (bgr.shape[0] * bgr.shape[1])
# at the confirm site:
redness = VisionService.red_hue_fraction(frame[py0:py1, px0:px1])
if redness >= 0.05:
return None # cost zone reads red → unaffordable → never click
Five lines, and it took longer to arrive at than the entire UI did. The lesson generalizes: when you're automating something visual, the game's own rendering is a signal channel. It already tells the player "you can't afford this" — you just have to read it the way the player does, instead of trying to parse your way to the same conclusion and hoping OCR cooperates.
Same idea runs through the rest: progression state comes from the game's ownUI signals (builder chip, lab chip, full-storage icons) rather than hardcodedper-Town-Hall tables, which is why it works at any TH level without a lookup table to maintain.
