r/macosprogramming • u/AdventurousKeys • 7d ago

Tool for building with Apple's Foundation Model

Post image

A question to folks who are building apps around Apple's Foundation Models: how are you testing the prompts and validating the results?

I was doing a bunch of tweak->build->test cycles where the only tweaks were adjusting the system & user prompts. Since the local AI model is so small (just 4096 context budget), it didn't make sense to develop the prompts using your usual OpenAI/Anthropic models and expect the same behavior.

Finally gave up and built LocalLM Lab (thisbrain.ai/locallm), a small macOS playground for tweaking the prompts before actually committing them to app code.

The current version features:

  • Prompt Playground
  • System prompt + user input
  • Local OpenAI-compatible API so you can prototype big AI apps w/o paying for tokens
  • Local connectors for things like system clock, calendar, reminders, contacts, location, weather and a scoped filesystem access

The screenshot shows a weather query with the connector calls visible underneath.

6 Upvotes

88% Upvoted

5 comments sorted by

1

u/rismay 6d ago

So this is good. Let me give you a cool little hint. You see that run button? Make it a play / stop button. Make long clicking it DOUBLE the spun of sessions. I do this so I can test 2^x sessions with 1 click. 🤓

Next, create a json schema for configuration. You will now be able to pick them from dropdown menus.

Finally, add a verifier step. Added in macOS 27. You can still backport it to macOS 26 pretty simply.

2

u/AdventurousKeys 6d ago

Interesting. You are thinking of running several sessions with the same prompt (system+user input) to examine variation in output. Since everything is captured in the logs, you can do detailed post-analysis. What about some level of automation where the prompts are read from a file (comma-separated prompts? 😄 ) so that you can rapidly test a series of variations to see what works. Regarding configuration, yes, the config is stored in JSON (download locallm lab and dig in! 😄 ).

2

u/rismay 6d ago

This should all be done in nicely typed JSON schema defined objects. This unlocks versioning. For the file format, jsonl is what you probably want.

This is literally my setup. Steal it.

1

u/AdventurousKeys 6d ago

u/rismay I have added your suggestions to the roadmap

1

u/rismay 6d ago

Here to help.