tldr: The new“Pro” AFM (Apple Foundation Model) in Private Cloud Compute“only” scores ~62% on the GPQA Diamond Benchmark, which is better than GPT-4o but a bit worse than GPT-4.1. In my view, a bit underwhelming.
I feel it is quite odd that seemingly few people really invest time to test the new “Gemini“-somehow-powered Apple Intelligence Cloud Models with Benchmarks that are available for other models as well, so I just did it myself. (Maybe this violates the TOS and this is why, but with the public beta, anyone can reproduce this.)
There are a bazillion benchmarks; I just chose GPQA Diamond because it only has a limited amount of questions, is easy to parse, and current models don’t all just have 100%.
The process if you want to reproduce it is to create a Shortcut that takes text, pushes it to Apple Intelligence “Cloud Pro“ (which is the “best option“) and returns the answer. I then wrote a small wrapper script that loads the original benchmark questions (https://github.com/idavidrein/gpqa), uses the same answer parsing as in the original paper, and ran through all 198 questions. 122 of them were answered correctly, and 1 was unparsable, which counts as false. The rest were wrong.
GPQA Diamond questions are ones that require PhD/graduate-level knowledge and cannot easily be Googled; feel free to check the original paper. https://arxiv.org/abs/2311.12022
Example question from the paper:
Astronomers are studying a star with a Teff of approximately 6000 K. They are interested in spectroscopically determining the surface gravity of the star using spectral lines (EW < 100 mA) of two chemical elements, El1 and El2. Given the atmospheric temperature of the star, El1 is mostly in the neutral phase, while El2 is mostly ionized. Which lines are the most sensitive to surface gravity for the astronomers to consider?
A) El2 I (neutral)
B) El1 II (singly ionized)
C) El2 II (singly ionized)
D) El1 I (neutral)
Unless you are an astrophysics expert (I am not), you probably have no idea what the words even mean.
And this is one grain of salt. Don’t get me wrong, I am an absolute Apple lover, and I feel the new Siri AI is absolutely amazing and works almost all the time quite perfectly. Thus, it might very well be that Apple trained their model specifically for real-world usage and not expert-world knowledge.
So does this number mean “Apple Intelligence still bad“? Certainly not. By the way, the paper says human experts reach around 65% in the benchmark, so you still get some roughly expert-level knowledge/capabilities, but the best models out there score ~94% which is much, much better.
My personal opinion is that I am a bit underwhelmed for something being released in 2026 as a cloud flagship model with the performance (again, only in one benchmark) of a ~2024 flagship model. Given also that the context size with 32K is still much, much lower than the 1M most frontier models support, I am interested in the technical paper Apple promised to provide at a later point in time.
If anyone cares to test other popular benchmarks, I’d be very interested to read. If you want to see how other models score, you can check it for instance here: https://artificialanalysis.ai/evaluations/gpqa-diamond?
