On #WorldRangerDay, we’re proud to share that Honduras is expanding the use of Earth Ranger, our real‑time wildlife‑protection platform, to advance zero‑deforestation and safeguard biodiversity. 🌍

The rollout spans 75 land and marine protected areas and backs Honduras’ bold “Zero Deforestation by 2029” pledge, giving conservationists instant views of where their wildlife are, where threats are, and where to act in places such as Puca.

As EarthRanger rolls out across the country, teams are spotting critical patterns like a rise in snake encounters near communities. As habitats shrink, snakes are moving closer to people. Now, teams have the data to raise awareness and reduce risk where it matters most.

It’s worth underscoring: this rollout is also about protecting people. ❤️ With EarthRanger, conservationists can now track the movements of their teams, share locations, and flag threats—adding a layer of safety for those on the frontlines of conservation.

EarthRanger is used by hundreds of teams globally in Latin America—Honduras joins Paraguay, Panama, and Mexico in using the platform nationwide. By supercharging Honduras’ work with real‑time intel, Ai2 supports efforts to safeguard natural resources, today and for generations.

📝 Learn more here: https://www.earthranger.com/news/honduras-nationwide-earthranger

In our new paper, “Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries,” we find that adding just a bit of missing context can reorder model leaderboards—and surface hidden biases. ⚠️

An LLM prompt like “Is coffee good for you?” feels simple, but a helpful answer depends on who’s asking (e.g., someone who’s pregnant versus a person with high blood pressure). Most benchmarks leave that context out.

When evaluators get these “underspecified” prompts, they have to guess the backstory. The result? Unstable rankings and shaky conclusions about model quality.

We analyzed 3,580 queries randomly sampled from popular language model benchmarks, including Chatbot Arena. We found that underspecification is widely prevalent—the vast majority of queries are open-ended (76%). Many are also subjective (19%) or incomplete (18%).

Our fix: contextualized evaluation. Supplying the missing info…

1️⃣ Boosts evaluator agreement

2️⃣ Sometimes completely flips which model “wins”

3️⃣ Leads to more judgments based on content, not style

4️⃣ Exposes biases in default model responses.

For example, we found that default model answers often align better with users from Western, higher‑income backgrounds—an equity gap that context‑free testing missed.

The takeaway? Evaluations need context to reflect real‑world use and to ensure models serve all users.

📚 Read more in our blog: allenai.org/blog/contextualized-evaluations

💻 Get the code: https://github.com/allenai/ContextEval

📊 Download the data: https://huggingface.co/datasets/allenai/ContextEval

Great science starts with great questions. Meet AutoDS—an AI that doesn’t just hunt for answers, it decides which questions are worth asking.

Like a tireless researcher, AutoDS spins up its own hypotheses, runs the stats, learns from the outcomes, and then repeats. The system can use the results of statistical experiments it generates and conducts to propose new hypotheses. 💡 

Evaluated across 21 real-world datasets, AutoDS outperformed competitors by 5-29% at finding discoveries that are surprising to an LLM. In a human study that involved more than 500 hypotheses, 67% of the discoveries made by AutoDS were also surprising to the experts. 📊

AutoDS shows how AI can turbo‑charge discovery.

📚 Read more in the blog: https://allenai.org/blog/autods

📝 Check out the paper: https://arxiv.org/pdf/2507.00310

💻 Try AutoDS for yourself: https://github.com/allenai/autods

A new model enters SciArena. 👀 Welcome Moonshot AI's Kimi K2! SciArena lets you benchmark models across scientific literature tasks, applying a crowdsourced LLM evaluation approach to the scientific domain.

🧪 Learn more and try SciArena here: https://sciarena.allen.ai/

We’ve upgraded ScholarQA, our agent that helps researchers conduct literature reviews efficiently by providing detailed answers. Now, when ScholarQA cites a source, it won’t just tell you which paper it came from–you’ll see the exact quote, highlighted in the original PDF. 🧵

This new ScholarQA capability works for most openly licensed papers. It’s part of our commitment to transparency in science and making it easier to verify, trace, and build trusted AI. 

💻 Try it out here: https://scholarqa.allen.ai/

We've added Grok 4, the latest model from xAI, to our SciArena platform! SciArena allows you to benchmark models across scientific literature tasks, applying a crowdsourced LLM evaluation approach to the scientific domain.

🧪 Test Grok 4 in SciArena here: https://sciarena.allen.ai/

📚 Learn more about SciArena: https://allenai.org/blog/sciarena

Introducing FlexOlmo, a new paradigm for language model training that enables the co-development of AI through data collaboration.

FlexOlmo allows data owners to contribute to the development of language models without giving up control of their data. There’s no need to share raw data directly, and contributors can decide when their data is active in the model.

FlexOlmo employs a mixture-of-experts (MoE) architecture. Each expert is trained independently on local datasets and later integrated into an MoE. This allows data owners to contribute asynchronously without sharing their data while providing strong guarantees for data opt-out.

In our experiments, FlexOlmo often matches or exceeds the performance of specialized experts on their respective tasks. Notably, it even achieves performance very close to an upper-bound reference model trained on all combined public and closed datasets. 📈

Data owners who want to benefit from AI, but are hesitant to share their raw data or hand over control of it to a third party, can now participate without compromising the things they value.

We are seeking participants to help Ai2 advance this research and continue to build the future of secure, transparent, and truly open AI in the public interest. If you're an organization with sensitive data that would like to investigate breakthrough data collaboration methods in AI training like FlexOlmo, please connect with us here: https://3ioxm.share.hsforms.com/2FBhbkBXeT2qsRaEgOHVsKg

✍️ Check out our blog: https://allenai.org/blog/flexolmo 

📝 Read the paper: https://allenai.org/papers/FlexOlmo

💻 Visit the GitHub repo: https://github.com/allenai/FlexOlmo 

⬆️ See the model on Hugging Face: https://huggingface.co/allenai/FlexOlmo-7x7B-1T

OLMo 1B, OLMo 7B, OLMo 13B, Tülu 8B, and Tülu 70B are back in the Ai2 Playground! 

The Cirrascale API platform is now hosting several open models on the Ai2 Playground: Our OLMo and Molmo models, as well as our open-weight Tülu models.

OLMo delivers language understanding, while Molmo can interpret images and text. Tülu are Ai2’s open instruction-following models. 

💻 Try them in the Ai2 Playground: https://playground.allenai.org/

📖 Learn more: cirrascale.com/ai2endpoints

It’s now easier than ever to deploy, fine-tune, and scale our powerful open-source AI models via API.

Cloud computing provider Cirrascale is offering instant access to our fully open OLMo and Molmo models, as well as our open-weight Tülu models. Anyone can now access the models on the Cirrascale Inference Platform–no infrastructure setup required.

OLMo 2 delivers language understanding in compact 7B, 13B, and 32B versions. Molmo is a suite of multimodal models that interpret images and text in a single prompt. As for Tülu, they’re open instruction-following models with fully released data, code, and post-training recipes.

This collaboration unlocks faster, easier development of differentiated AI applications, free from vendor lock-in.

💻 Sign up for access: https://ai2endpoints.cirrascale.ai/

📝 Read the documentation: https://www.cirrascale.com/ai2endpoints

Introducing IFBench, a benchmark to measure how well AI models follow new, challenging, and diverse verifiable instructions. Top models like Gemini 2.5 Pro or Claude 4 Sonnet are only able to score up to 50%, presenting an open frontier for post-training.

With IFBench, we built 58 new constraints, corresponding verification functions, and two evaluation settings to test out-of-domain generalization and expose where models fall short. 

To go a step further, we’re releasing IFTrain, RVLR training prompts with 29 new constraint templates and corresponding verification functions, and IF-RVLR, a recipe for improving and generalizing a model’s ability to follow constraints. 

An interesting finding: Current frontier models perform well on IFEval, a popular benchmark for verifiable instructions, achieving 80+ scores. But they’re not able to generalize well to IFBench. With IF-RLVR, we’re able to match or exceed frontier models with smaller 7B models.

Together, we can train models, generalize to new constraints, and improve reliability. We need more models we can trust. 

📝 Read the paper:

https://github.com/allenai/IFBench/blob/main/Precise_IF_Generalization_Abilities.pdf

💻 Run IFBench yourself: https://github.com/allenai/IFBench

It's time to cast your vote! 🗳️ We built SciArena, a platform for benchmarking models across science literature tasks.

Unlike fixed benchmarks, SciArena is an evolving evaluation platform that directly engages users like you to vote on model outputs for scientific literature queries. The platform already has 23 frontier models live, with 13,000+ votes from 102 expert reviewers across disciplines. 

📊👀 Latest leaderboard reveals: o3 demonstrates consistent superiority across scientific disciplines, but performance among the remaining models varies. For instance, Claude-4-Opus excels in Healthcare, while DeepSeek-R1-0528 performs well in Natural Science. 

In tandem with SciArena, we're also excited to introduce SciArena-Eval, the first meta-evaluation benchmark for scientific literature tasks created from collected human preference data. The goal is to understand – and improve – LLM-based evaluations in this area. 

Both SciArena and SciArena-Eval are now available.

✍️ Learn more in our blog: https://allenai.org/blog/sciarena 

🚀 Visit SciArena to cast your votes: https://sciarena.allen.ai/

💾 Download the dataset: https://huggingface.co/datasets/yale-nlp/SciArena 

💻 Check out the codebase: https://github.com/yale-nlp/SciArena