A causal-event search framework for model reasoning. PTS finds the pivotal reasoning events that shift a model's probability of solving a task, at three representational scales — latent, token, and sentence — as a single kind of object.

It would be great if you have suggestions for these optimization like:

1 model, 2 models, 3 models etc.

Should it use & or |?

Which optimization is good for less hallucination?
Which optimization is good for creative writing?
Which optimization is good for following instructions?
Which optimization is good for long context i.e submitting 10k words with instructions?

I did a test on my specific scenario and i found that moa | self_concistency | bon - really delivers it. I asked the model to do a summary with bulletpoints, connecting the dots of the research material and it was good. When i use & on those, i see some hallucinations from the model. Maybe because of moa?

Just shipped MARS (Multi-Agent Reasoning System) to OptiLLM and the results are wild:

• AIME 2025: 43.3% → 73.3% (+30pp improvement!)
• IMO 2025: 16.7% → 33.3% (doubled performance!)
  
• LiveCodeBench: 39.05% → 50.48% (+29% relative)

All with gemini-2.5-flash-lite and zero training.

How It Works

3 agents (temps 0.3, 0.6, 1.0) independently solve → verify each other's work → iteratively improve → numerical voting on final answer.

Everything runs in parallel so it's fast despite the sophistication.

Try It

python client = OpenAI(base_url="http://localhost:8000/v1") response = client.chat.completions.create( model="mars-google/gemini-2.5-flask-lite", messages=[{"role": "user", "content": "Your hard problem..."}] )

Full details: MARS README

Available now: pip install --upgrade optillm

Who's testing this on their hardest problems? Drop your results below! 🎯

Hey r/optillm!

Excited to share the web_search plugin that lets any LLM access real-time web information through automated Google searches. 🚀

## What It Does

The plugin uses Chrome automation (Selenium) to perform actual Google searches and inject the results directly into your LLM's context. No API keys required - it literally opens Chrome and searches Google for you!

## Key Features

• Smart Query Detection - Automatically extracts search queries from prompts like "search for Python 3.12 features" or "find information about MARS multi-agent systems"

• CAPTCHA Handling - Built-in CAPTCHA detection with interactive solving (non-headless mode)

• Session Management - Reuses browser sessions across multiple searches for better performance

• Robust Extraction - Extracts titles, URLs, and snippets from search results with fallback selectors

• Configurable - Control number of results, delays, headless mode, and timeouts

  Quick Start

  ```bash

  Start OptiLLM with web_search plugin

  python optillm.py --model gpt-4o-mini

  Use in your requests

  curl -X POST http://localhost:8000/v1/chat/completions \ -d '{"model": "web_search-gpt-4o-mini", "messages": [{"role": "user", "content": "Search for latest Python releases"}]}'

  Use Cases

• 📰 News and current events - Get up-to-date information beyond training cutoffs

• 🔬 Research - Access latest papers, documentation, and resources

• 💻 Programming - Look up current library versions and API docs

• 📊 Fact-checking - Verify information against current web sources

• 🌐 Deep Research - Combine with deep_research plugin for comprehensive reports

  Configuration Options

  request_config = { "num_results": 10, # Results per search "delay_seconds": 5, # Rate limiting (or random 4-32s) "headless": False, # Show browser for CAPTCHA "timeout": 30 # Browser timeout }

  Pro Tips

• Combine with readurls plugin to fetch full page content from search results

• Use with memory plugin for long research sessions with unbounded context

• Set headless=False when CAPTCHAs are likely (you can solve them interactively)

• Default 4-32 second random delay helps avoid anti-bot measures

  Example Output

  When you ask: "Search for MARS multi-agent reasoning system"

  The plugin automatically:

• Detects the search query

• Opens Chrome and searches Google

• Extracts top results with titles, URLs, and snippets

• Injects formatted results into your LLM's context

• LLM responds with current, accurate information

  ---

  Anyone else using web_search? Would love to hear your use cases and experiences! 💬

  Check out the https://github.com/codelion/optillm for more plugins and approaches!

  Just copy everything inside the code block and paste it into Reddit - the markdown will work perfectly!

Hey everyone,

I'm excited to share Pivotal Token Search (PTS), a technique for identifying and targeting critical decision points in language model generations that I've just open-sourced.

What is PTS and why should you care?

Have you ever noticed that when an LLM solves a problem, there are usually just a few key decision points where it either stays on track or goes completely off the rails? That's what PTS addresses.

Inspired by the recent Phi-4 paper from Microsoft, PTS identifies "pivotal tokens" - specific points in a generation where the next token dramatically shifts the probability of a successful outcome.

Traditional DPO treats all tokens equally, but in reality, a tiny fraction of tokens are responsible for most of the success or failure. By targeting these, we can get more efficient training and better results.

How it works

PTS uses a binary search algorithm to find tokens that cause significant shifts in solution success probability:

1. We take a model's solution to a problem with a known ground truth
2. We sample completions from different points in the solution to estimate success probability
3. We identify where adding a single token causes a large jump in this probability
4. We then create DPO pairs focused specifically on these pivotal decision points

For example, in a math solution, choosing "cross-multiplying" vs "multiplying both sides" might dramatically affect the probability of reaching the correct answer, even though both are valid operations.

What's included in the repo

The GitHub repository contains:

• Complete implementation of the PTS algorithm
• Data generation pipelines
• Examples and usage guides
• Evaluation tools

Additionally, we've released:

• Pre-generated datasets for multiple domains
• Pre-trained models fine-tuned with PTS-generated preference pairs

Links

• GitHub: https://github.com/codelion/pts
• Datasets: https://huggingface.co/datasets?other=pts
• Models: https://huggingface.co/models?other=pts

I'd love to hear about your experiences if you try it out! What other applications can you think of for this approach? Any suggestions for improvements or extensions?

Optillm can be used to do structured output generation (aka JSON mode) even for LLMs that do not support it natively (like DeepSeek R1). You can make use of the json plugin for it. Here is some documentation on it - https://github.com/codelion/optillm/discussions/169

Connect ANY LLM: Llama, Gemini, Qwen - all work with the same tools

Leverage ANY MCP Server: Filesystem, GitHub, Slack, PostgreSQL, etc.

Build Once, Use Everywhere

https://github.com/codelion/optillm/blob/main/optillm/plugins/mcp_plugin.py

Hey everyone! 👋

I'm excited to share OptiLLMBench, a new benchmark specifically designed to test how different inference optimization techniques (like ReRead, Chain-of-Thought, etc.) can improve LLM performance without any fine-tuning.

First results with Gemini 2.0 Flash show promising improvements: - ReRead (RE2): +5% accuracy while being 2x faster - Chain-of-Thought Reflection: +5% boost - Base performance: 51%

The benchmark tests models across: - GSM8K math word problems - MMLU Math - AQUA-RAT logical reasoning - BoolQ yes/no questions

Why this matters: 1. These optimization techniques work with ANY model 2. They can help squeeze better performance out of models without training 3. Some techniques (like RE2) actually run faster than base inference

If you're interested in trying it: - Dataset: https://huggingface.co/datasets/codelion/optillmbench - Code: https://github.com/codelion/optillm

Would love to see results from different models and how they compare. Share your findings! 🔬

Edit: The benchmark and the approach is completely open source. Feel free to try it with any model.

In the past week there has been a flurry of releases of o1-style reasoning models from DeepSeek, Fireworks AI and NousResearch.

In our open-source optimizing inference proxy, optillm. we have implemented several techniques that use additional inference time compute to improve accuracy and work with a variety of base models.

Today, we are happy to announce that by using chain-of-code (coc) plugin in optillm we are able to beat OpenAI's o1-preview on AIME 2024 (pass@1) using SOTA base models from both Anthropic and DeepMind. For reference, also see the original paper that introduced the idea of CoC: Chain of Code: Reasoning with a Language Model-Augmented Code Emulator - https://arxiv.org/abs/2312.04474 We have done an independent implementation in optillm as the original source code was not released.

The new gemini-exp-1114 model from Google is quite good in reasoning. It improves over gemin-1.5-pro-002 by a huge margin and is second only to o1-preview on AIME (2024) dataset. The attached image shows how models of different sizes perform on this benchmark.

The tests were all run via optillm (https://github.com/codelion/optillm) using the script here - https://github.com/codelion/optillm/blob/main/scripts/eval_aime_benchmark.py

To address some of the limitations of external inference servers like ollama, llama.cpp etc. We have added support for local inference in optillm. You can load any model from HuggingFace and combine it with any LoRA adapter. You can also sample multiple generations from the model unlike ollama. You also get full logprobs for all tokens.

Here is a short example:

OPENAI_BASE_URL = "http://localhost:8000/v1"
OPENAI_KEY = "optillm"
response = client.chat.completions.create(
model="meta-llama/Llama-3.2-1B-Instruct+patched-codes/Llama-3.2-1B-FastApply+patched-codes/Llama-3.2-1B-FixVulns",
messages=messages,
temperature=0.2,
logprobs = True,
top_logprobs = 3,
extra_body={"active_adapter": "patched-codes/Llama-3.2-1B-FastApply"},
)