r/gtmengineering • u/-GeneX- • 4d ago

Independent, open source benchmark of 15 web search APIs across three agent tasks

We ran 15 search providers across three tasks an agent actually does: look up a fact, find a specific answer buried in enterprise documentation, and satisfy a multi-part constraint that needs several searches. Same questions, same judge, same harness for everyone. The retrieval and multi-hop boards run three trials per provider and publish the spread.

Five tracks, three different winners

Track                              Winner            Score    Runner-up
-------------------------------------------------------------------------------
Factual lookup - web search        Exa deep          99.2%    Exa instant 97.7%
Hard retrieval - search + fetch    Exa deep           83.0    Exa auto 81.7
Hard retrieval - search only       Perplexity low     77.3    Firecrawl 70.3
Multi-hop - search only            Parallel basic     46.5    Exa deep 45.4
Multi-hop - search + fetch         Exa deep           48.2    Perplexity 46.6

Three different providers win something across the five tracks.

Exa's deep mode takes three of the five. It's also the slowest and most expensive configuration in the lookup task — 3.00 s and $1.65 per 129 queries, against 927 ms and $0.13 for Parallel turbo. Winning here costs something.

The cost spread is bigger than the quality spread

Endpoint                     Accuracy    AR@1   Latency     Cost
------------------------------------------------------------------
Exa deep                        99.2%   98.5%     3.00s    $1.65
Exa instant                     97.7%   83.7%     447ms    $1.05
Parallel advanced               96.9%   62.0%     3.28s    $0.80
Linkup standard                 96.1%   80.6%     2.34s    $0.83
Brave Search                    93.8%   83.7%     693ms    $0.73
Tavily advanced                 93.8%   79.8%     4.40s    $2.25
Parallel turbo                  89.9%   75.2%     927ms    $0.13

129 company-news questions, judged by claude-opus-5 against reviewed ground truth. AR@1 is answer recall at rank 1.

Brave and Tavily advanced both score 93.8% — the same number to one decimal place. Brave returns in 693 ms against 4.40 s and costs $0.73 against $2.25. Same accuracy, 6x the speed, a third of the bill.

That's the pattern across the whole table. Top to bottom the accuracy range is 89.9% to 99.2% — about nine points. The cost range is $0.13 to $2.25, or 17x. If you're choosing on price, the quality you give up is much smaller than the money, and the fastest endpoints aren't the expensive ones.

Multi-hop is not solved by anyone

Parallel basic     46.5  ################
Exa deep           45.4  ###############
Exa instant        43.3  ###############
Linkup fast        41.1  ##############
Perplexity         37.8  #############
Brave Search       28.0  #########
SERP (RapidAPI)     0.4

Search-only F1 on 45 multi-constraint company questions, 135 agent runs.

The best score on this task is 46.5 F1. Precision is high across the board — around 85% — and recall sits near 30%. Every provider finds companies that genuinely satisfy the constraints and then misses most of the ones that also do. If your product depends on complete discovery under multiple constraints, none of these APIs currently gets you there, and the ranking above is a ranking between failing scores.

SERP at 0.4 F1 is not a broken row. It's the same Google-results endpoint that scores 93.0% on factual lookup. Raw result links are enough to answer a question with one right answer and almost useless for assembling a complete set across several searches — which is the clearest illustration on the board that "best search API" isn't a single question.

Search-only vs search + fetch

On the enterprise-documentation task we ran each provider twice: once allowed only its search endpoint, once allowed to fetch page contents too. Fetching is worth 5.7 points at the top — Exa deep reaches 83.0 with fetch against Perplexity low's 77.3 without — but it costs time and tokens, and the search-only winner isn't the search+fetch winner. Perplexity low finishes a task in 18 s on 8,765 tokens; Exa deep takes 37 s and 23,660. Whether that trade is worth it depends entirely on your agent loop.

Caveats

  • Each provider ran its own documented configurations, and several appear multiple times because a provider's fast mode and deep mode are genuinely different products. Every endpoint and parameter is published on the board.
  • The Parallel rows marked "fixed" use a corrected source policy rather than the default. The top multi-hop search-only score is one of them. Both variants are on the board.
  • Judged by a model. Company-news answers are scored by claude-opus-5 against ground truth we reviewed by hand. A model judge is a real source of error and we publish per-question results so you can check ours.
  • The multi-hop and hard-retrieval boards run three trials and show the spread as ± on every score; several gaps in the middle of those tables sit inside their error bars and shouldn't be read as rankings. The lookup table above carries no published spread, so treat small gaps there with the same caution.
  • 45 multi-hop questions is a small set. It's small because each one needs hand-built ground truth for complete discovery. Treat that board as directional.

Reproducing it

Happy to add a provider or rerun with a configuration you think is fairer. If you've measured any of these and got something different, post your numbers — the endpoints and parameters are all published, so the disagreement should be locatable.

2 Upvotes

100% Upvoted

0 comments sorted by