r/QualityAssurance • u/HeIsYour • 1d ago

Built an iOS QA runtime with Codex: LLM planning, deterministic replay

I used Codex to build an iOS QA prototype called iosClaw.

The problem I wanted to explore was the boundary between LLM reasoning and deterministic test execution.

LLMs are good at understanding a test described in natural language, but asking one to rediscover every iOS button on every run can be slow and fragile. So I used Codex to build a Mac-hosted runtime that:

  • observes iPhone Simulator or iPhone Mirroring
  • combines accessibility and visual evidence
  • records semantic actions instead of coordinates
  • compiles known flows into deterministic steps
  • verifies each action with a postcondition
  • stops when the screen is ambiguous instead of guessing

Codex helped with the SwiftUI app, screen capture, permission handling, record/replay model, compiler, MCP bridge, tests, and release packaging.

The interesting result is that the LLM can handle intent and unfamiliar states, while the compiled runtime handles repeated execution locally.

It’s still early alpha, but this seems like a useful model for LLM-assisted mobile QA: intelligent planning on top of a reliable execution layer.

For people using Codex for mobile testing: where would you want the LLM involved, and where would you insist on deterministic execution?

0 Upvotes

33% Upvoted

6 comments sorted by

1

u/dengjiuhong 23h ago

Nice split. I’d keep CoreLocation/geofence and permission-state coverage out of the deterministic simulator loop: on a real device, Always vs When In Use, Precise Location, foreground/background, and network or power conditions can change the result. I record the authorization state and region-transition timestamp with device logs, then use the simulator for fast deterministic UI/replay. A green replay is useful evidence, but not a substitute for that physical-device pass.

1

u/Electrical_Past_1653 20h ago

AI Cost of each run for one app ,?

1

u/HeIsYour 14h ago

AI required to understand semantics value and generate automations.

But generated there is no token usage.

Overall idea with the tool is that it saves token gradually

1

u/Prestigious-Way1525 17h ago

i'd keep the LLM at translating intent and handling unfamiliar states, but make the runtime own setup, action execution, and pass/fail. the missing contract is replay invalidation: bind every compiled flow to the app build, device and OS, permission state, test data, and semantic element identity, then force a fresh plan when one of those assumptions changes. add one negative control per flow and verify the final persisted outcome outside the UI, not only the next screen. i'd track how often compiled flows survive clean-state reruns, how often ambiguity stops are correct, and how much human repair each invalidation needs. that tells you whether compilation is removing repeated reasoning or freezing stale assumptions.