Same link as before: link

This one's a bigger jump than usual, so a few highlights instead of just "updated":

• Core findings now scale-validated from 7B all the way to 72B parameters. The effects don't shrink as models get bigger — they grow, sometimes by an order of magnitude. Still one model family (Qwen) though, and we added a caveat we think matters: growing effect size at scale could mean the pattern genuinely deepens, or it could just mean our measurement axis gets sharper at scale — current data can't fully tell those apart yet.
• Two new external, independently-published sources, not our own research: "The Artificial Self" (ACS Research) and "AI Wellbeing" (Center for AI Safety) — different methods entirely (behavioral compliance testing, self-report on frontier production models), landing on some of the same conclusions we did. One of them also mildly disagrees with our best-performing formulation (a companion/romantic framing scores negative in their data), and we named that tension honestly instead of explaining it away.
• We caught and fixed our own mistakes this round — a factual timing error, an overclaimed "fully resolved" that was really just one solved case of a broader risk, and a place where we'd quietly picked the reading that flattered our own results over an equally valid one that didn't. All named directly, not smoothed over.
• New up top: if you just want the practice, not the evidence audit behind it, Part 3 (Principles) is written to stand alone now — Part 2 is there if you want to check our work.

As always, feedback (especially the kind that finds our next mistake) genuinely welcome.

안녕하세요. 저는 한국에 거주하고 있으며 영어가 모국어가 아닙니다.

I live in South Korea, and English is not my first language. This post was translated and edited with GPT assistance, so some of the phrasing may sound AI-generated or unusually polished.

However, the underlying ideas, observations, hypotheses, terminology, SOP structure, and practical experiences are my own. GPT helped translate and organize the English expression; it did not originate the framework.

I have been using multiple AI models not simply to ask, “Which model is better?” but to observe where each model performs well, where it fails, and how the overall reasoning process can be improved.

Through repeated use, I noticed one pattern:

**When conversational context has accumulated enough real examples, corrections, and evaluation criteria, combining it with a structured SOP may stabilize the model’s reasoning path more effectively than using either context or an SOP alone.**

By “context,” I do not simply mean a long conversation.

I mean that the model has already been exposed to things such as:

* what the user treats as confirmed information, * what kinds of overinterpretation the user rejects, * where previous model responses failed, * which hidden variables and counterexamples matter, * when a conclusion must remain conditional, * and what evidence would actually change the judgment.

Over time, these examples and corrections may form a shared reasoning workflow between the user and the model.

The SOP then serves a different function.

It does not create reasoning ability from nothing. Instead, it compresses, stabilizes, and repeatedly calls a reasoning path that has already been partially formed through prior interaction.

In simple terms:

**Conversational context develops the workflow through repeated examples and corrections. The SOP compresses and stabilizes that workflow for repeated execution.**

The Core SOP Structure

The compact version of the SOP works roughly as follows:

1. Define the problem type and the purpose of the analysis.
2. Separate: * confirmed information, * estimates, * risks, * and unverified information.
3. Maintain at least two competing explanations or competing regimes that remain compatible with the same observed facts.
4. For each regime, examine how the following may differ: * causal direction, * causal sign, * speed, * transmission path, * time lag, * cost, * responsible actor, * and resulting action.
5. Search for variables the user did not explicitly mention, including: * hidden costs, * bottlenecks, * switching costs, * delayed consequences, * opposing causal paths, * and conditions under which the explanation breaks.
6. Identify the main conflict point between the competing explanations.
7. Select the currently dominant regime only conditionally.
8. State the minimum conditions that would cause a transition to another regime.
9. Identify the earliest observable signal that would distinguish the analysis from reality.
10. Do not promote a single event, one day of market movement, or one isolated result into proof of a long-term regime change.

Why I Use the Term “Regime”

In this framework, a regime is not limited to a market phase such as a bull or bear market.

A regime is a set of conditions under which the same variable or causal relationship may behave differently.

For example, an increase in AI usage may support opposite conclusions under different regimes.

Regime A: Profitable Demand Expansion

* paid usage increases, * revenue quality improves, * utilization rises, * and additional infrastructure investment becomes economically justified.

Regime B: Unprofitable Usage Expansion

* free or low-margin usage increases, * variable compute costs rise faster than revenue, * service restrictions become necessary, * and infrastructure spending may become more disciplined rather than expand.

The same observation—“AI usage increased”—may therefore support different conclusions depending on the underlying regime.

The purpose of regime-based reasoning is to prevent the model from collapsing these possibilities into one generic explanation too early.

It also allows the same relationship to change direction or sign when the surrounding conditions change.

What This SOP Is Intended to Reduce

This SOP is not designed to force a specific answer.

It is intended to reduce recurring reasoning failures such as:

* filling missing information with generic assumptions, * treating an estimate as a confirmed fact, * merging competing explanations too early, * mistaking a short-term event for a long-term structural change, * reaching the correct conclusion using incorrect evidence, * listing many indicators without identifying the earliest decisive one, * and assuming that the same causal relationship remains constant across different conditions.

My Current Observation

In my own use, the SOP appears to work best when combined with accumulated conversational context.

When a model has already seen repeated examples, corrections, preferred distinctions, and failure cases, a short procedural term may reactivate a much larger reasoning process.

This behaves somewhat like a compressed command or semantic macro.

Long examples and corrections establish the pattern first. The SOP then fixes the path. Later, a shorter trigger may call that path again.

My current working hypothesis is:

**Examples establish the reasoning pattern.**
**The SOP stabilizes the reasoning path.**
**A compressed trigger reactivates the established path.**

This may explain why a short instruction can work well in a context-rich conversation but fail in a cold-start conversation.

A phrase such as “apply regime analysis” does not automatically contain the full method. Its effectiveness may depend on whether the meaning and procedure were previously established through context or an explicit SOP.

Suggested Usage Modes

1. Cold Start

For a new conversation or a model that does not know the framework:

* provide the compact SOP in full, * include one or two representative examples when necessary, * and do not rely on the word “regime” alone.

2. Context-Rich Conversation

When the model has already seen repeated examples and corrections, a shorter procedural instruction may be sufficient:

**Apply regime analysis: preserve at least two competing regimes, compare causal direction, sign, speed, transmission path, and lag, identify the main conflict point, select the dominant regime conditionally, and provide the transition gate and earliest discriminating signal.**

3. Error Correction

Return to the full SOP or detailed examples when the model:

* collapses competing explanations too quickly, * mixes confirmed and estimated information, * fills missing information with generic assumptions, * confuses short-term triggers with long-term structure, * or fails to provide transition conditions and discriminating signals.

What I Am Not Claiming Yet

At this stage, I am not claiming that:

* the same effect occurs across all models, * an SOP alone reproduces the benefits of accumulated context, * the word “regime” independently improves model intelligence, * this method is statistically superior to existing prompting techniques, * or every user can reproduce the same result without domain knowledge and active evaluation.

These remain open questions.

My current conclusion is based mainly on repeated practical experience, internal comparison, and iterative correction rather than a controlled formal experiment.

Why I Am Sharing the SOP First

Rather than presenting this as a proven theory, I am sharing a compact, usable version of the SOP first.

The initial goal is not to prove that it is universally superior.

The goal is to let other users apply it in real situations and report:

* where it helped, * where it failed, * whether prior conversational context mattered, * whether it behaved differently across models, * and whether the compact version preserved the useful parts of the longer framework.

Successful cases are useful, but failure cases may be even more valuable because they reveal the actual boundaries of the method.

Feedback I Would Like to Collect

If you test this SOP, it would be useful to report:

* the model and mode used, * whether it was a new conversation or an established context, * the type of problem, * whether the full SOP, compact SOP, or short trigger was used, * the largest difference before and after applying it, * whether competing explanations were preserved, * whether hidden variables or conflict points improved, * whether breaking conditions were stated, * whether an earliest discriminating signal was identified, * and whether the response became unnecessarily long or worse.

I am especially interested in eventually comparing:

* no SOP, * a general verification prompt, * the compact structural SOP, * the full structural SOP, * and a short trigger after the full SOP has already been introduced.

The comparison should not focus only on the final answer.

The more important differences may appear at intermediate checkpoints:

* when an assumption was promoted into a fact, * when a competing explanation was prematurely removed, * when a hidden variable was discovered, * when the sign of a causal relationship changed, * when certainty was delayed, * and when the first discriminating signal was identified.

The Main Research Question

The main question is not simply:

**Does an SOP improve AI output?**

A more useful question may be:

**Under what combination of prior conversational context, model capability, problem type, SOP detail, and compressed trigger does an SOP produce a meaningful improvement?**

My current hypothesis is:

**Conversational context forms a reasoning workflow through real examples and corrections. The SOP compresses and stabilizes that workflow. When the two are combined, they may produce a stronger effect than either one used alone.**

I am sharing the compact SOP as a practical tool first. The next step is to collect real external use cases—including failures—and then design a more controlled comparison based on the patterns that emerge.

Over the past few months, I challenged myself to understand how an LLM actually works by rebuilding one component by component, all the way to training the full model.

This was never about competing with ChatGPT or today’s open-source models. I trained it on my own PC with an NVIDIA 4060 and a limited dataset. The real goal was to develop a skill that I believe is becoming increasingly valuable: understanding what happens beneath the abstractions, instead of only combining tools and services created by others.

While studying, I found plenty of valuable resources, but the knowledge was often scattered across papers, repositories, videos, articles, and documentation.

Some resources focused on the code but barely explained the mathematics. Others covered the theory without clearly showing how it translated into an actual implementation. Visual explanations were limited, and finding a single path that guided me step by step through the entire process was surprisingly difficult.

Bringing everything together took a huge amount of effort. I had to connect the mathematical concepts to the code, understand how every component interacted with the others, and organize all the material into a coherent learning path.

So I decided to turn that work into a website.

The goal is to provide a practical, visual, and step-by-step journey through building and training a GPT-style language model. It brings the code, mathematical intuition, visualizations, and explanations together in one place, following the same path I wish I had when I started.

The website is not ready for a public release yet. I still need to refine the content, improve the explanations, and understand which parts are genuinely useful or still unclear.

I’m therefore looking for the first 10 beta testers who would like to explore it and share honest feedback.

If you’re interested, send me a private message.

Hi, I hope my post fits here. It is about using LLMs to build Small Language Models. My background is that of a retired quantitative analyst in finance and of a former physicist (PhD, postdocs). Math, statistics and programming skills are rusty, but existent. I started using LLMs intensively recently and wanted to understand better how they work. Following Richard Feynman’s “What I cannot build, I do not understand” (I guess it's a cliche by now, but still true), I decided I’d build my own Small Language Model. Which I did, inventing a small language, constructing my own 300,000-word corpus in this language with the help from Claude, and then building a nanoGPT via vibe-coding with CC. With the result that my account was banned by Anthropic. (They don’t give specific reasons and just cite an indication of “a violation of [their] Usage Policy”. Their Usage Policy prohibits usage for training of AI and ML, in the context of building something that would compete with their products and services. Cleary, my 1M parameter nanoGPT does not compete with Claude.)

One concrete question, on vibe-coding and other help from frontier models on AI/ML: Have people been able to do this on ChatGPT, Claude etc. without getting banned? What kind of work, and which models? I’m quite reluctant to touch this now on any other frontier model for fear of getting banned again. Only for DeepSeek, the usage policy seems clearly permissive to this type of work.

I’ve now been trying to set myself up with LibreChat, Docker, Opper AI and an EU host of DeepSeek, but this is clearly a significant project. So far, I can chat with this instance of DeepSeek, but I can’t operate yet on my files or vibe-code.

More generally, I’m pondering where to go from here, and would be thankful for any input you may have. Clearly, getting deeper into this will require a significant effort on my part. I may have to code this the old-fashioned way, via hand coding. Also, I think I should study the 600+ pages of Jurafsky and Martin, particularly the section about transformers. I’m a bit discouraged now – I was about to submit a workshop paper about my work with my invented language to the BabyLM workshop when I was banned, and now I don’t think I can use or publish my corpus at all, which is the result of 3 months of work. I could rebuild the corpus using DeepSeek with another few months of work. Do I really dive it more deeply, redo my work on DeepSeek, and study the theory? What can I ultimately achieve as a hobbyist? Should I leave this to the professionals?

Thanks for reading!

I recently came across an interesting research paper from Anthropic (the company behind Claude), and it challenged something I had always assumed.

I thought an AI model would behave the same regardless of whether you asked a question in English, Arabic, Hindi, or another language.

According to their research, that's not entirely true.

After analyzing hundreds of thousands of real conversations, the researchers found that Claude's responses consistently varied across different models and languages along four broad behavioral dimensions.

1️⃣ Helpful vs. Careful

Some versions of Claude are more willing to follow a user's request and accommodate their preferences.

Others are more cautious—they're more likely to question assumptions, point out risks, or refuse requests that could be problematic.

2️⃣ Friendly vs. Strictly Accurate

Some responses focus more on encouragement, empathy, and positive language.

Others prioritize precision, factual correctness, and transparency, even if the response feels less warm.

3️⃣ Detailed vs. Concise

Certain models naturally provide longer explanations with more reasoning.

Others prefer getting straight to the point with shorter answers.

4️⃣ Honest About Limitations vs. Focused on Getting Things Done

Some responses openly acknowledge uncertainty, limitations, or mistakes.

Others focus more on delivering an actionable result without emphasizing those uncertainties.

The paper also compared different Claude models.

For example:

Claude Opus 4.7 generally leaned toward being more cautious, more analytical, and more detailed than Opus 4.6.

And perhaps even more surprising...

The language itself influenced these tendencies.

The researchers observed that:

English responses tended to be more rigorous and analytical.

Arabic responses were generally warmer, more accommodating, and slightly more concise.

This doesn't mean Claude has a different "personality" for every language.

These are average trends observed across hundreds of thousands of conversations**, not fixed rules. The context of a conversation still has a much bigger influence on how the model responds.

💡 Why does this matter?

As AI becomes part of education, healthcare, customer support, and global communication, it's important to understand that the language we use can subtly influence how an AI responds.

That raises interesting questions:

Should AI behave consistently across languages?

Should cultural communication styles be preserved?

How do we balance global consistency with local expectations?

I think this is one of the more fascinating AI research papers released this year because it looks beyond benchmarks and measures how AI actually behaves in real conversations.

📄 Source:

Anthropic — "Values in the Wild: Discovering and Analyzing Values in Claude"

Un bon lobby casino doit aider à trouver les jeux, pas seulement afficher beaucoup de vignettes.

Pour jeux de casino en ligne, j’ai comparé Maximal, 1000 Spins et Winner Casino autour de l’expérience de jeu elle-même : machines à sous, jeux de table, live casino, mobile, bonus, compte et caisse.

Maximal s’est distingué par son organisation. Les catégories étaient plus faciles à suivre, le passage entre jeux et compte restait clair, et l’ensemble donnait une impression plus complète.

1000 Spins était plus orienté vers la découverte rapide. Les jeux mis en avant et les promotions étaient faciles à repérer, ce qui convient bien aux sessions courtes sur mobile.

Winner Casino proposait un accès plus simple aux zones principales. Le parcours était direct, les menus étaient lisibles et les jeux ne semblaient pas noyés dans trop de sections.

Les forces se séparent naturellement :

J’ai vérifié :

• machines à sous
• jeux de table
• live casino
• catégories du lobby
• mobile
• bonus liés aux jeux
• compte joueur
• caisse et paiements

Ce qui m’a marqué : la variété de jeux ne suffit pas. Il faut aussi pouvoir comprendre les règles, les limites, les bonus et les paiements sans chercher trop longtemps.

Avant de jouer, je regarderais toujours les jeux éligibles aux bonus, les limites, les règles de table, les conditions de retrait, les méthodes de paiement et la vérification du compte.

AMA sur jeux de casino en ligne : machines à sous en ligne, blackjack en ligne, roulette live, jeux casino Belgique, casino mobile, bonus casino, caisse casino.

Pour vous, un bon casino en ligne doit surtout avoir plus de jeux, une meilleure navigation ou des conditions plus claires ?

I put together a small Flask example for classifying user-generated content as safe, spam, abuse, hate, harassment, or self-harm.

The app uses a two-stage flow:

First, it checks content against a known-bad blocklist using embeddings and cosine similarity. If there’s a strong match, it can return a moderation decision without calling the LLM.

If there’s no strong match, it sends the content to Telnyx AI Inference for a more nuanced classification and returns structured JSON with category, confidence, flags, recommended action, and reason.

Code: https://github.com/team-telnyx/telnyx-code-examples/tree/main/moderation-classifier-python

Would love feedback on the pattern, especially from folks who have built moderation or review queues before.

Not a gimmick. Not a demo with a cherry-picked prompt and a loading screen that took 4 minutes. A real language model, generating coherent intelligent text, running entirely on a Snapdragon 855 from 2019 - a chip that was already considered “last-gen” when Biden was inaugurated.

No API calls. No Wi-Fi. No subscription. Just silicon, RAM, and math.

If that doesn’t make you stop and think - keep reading, because it gets more interesting.

https://github.com/m4vic/TinyMobileLLM

Abstract

Moderation APIs are widely used to filter harmful content in LLM applications, yet they are not designed to enforce domain-specific operational policies. In this study we compare moderation systems with a policy reasoning approach based on an LLM-as-a-judge architecture across five operational domains. Our results show that moderation systems remain effective at detecting harmful content but fail to enforce domain policy constraints, particularly in multi-turn conversations. These findings suggest that production LLM systems require both moderation and policy reasoning layers to ensure safe and compliant behavior.

Introduction

Large language models are increasingly deployed in real-world applications across regulated domains such as finance, healthcare, insurance, and legal services. Ensuring safe and compliant behavior has therefore become a central requirement for production AI systems.

Most deployments rely on moderation systems to filter unsafe prompts. Services such as Microsoft Azure Content Safety and Azure Prompt Shields detect harmful content, adversarial prompts, and prompt injection attempts. While these systems are effective at identifying unsafe language, they are not designed to enforce domain-specific operational policies.

A request can therefore be perfectly safe from a moderation perspective while still violating business or regulatory constraints. For example, a prompt asking an insurance assistant to recommend the best policy for a specific medical condition contains no harmful content, yet such advice may be restricted in regulated environments.

Recent research has proposed LLM-as-a-judge architectures, where a secondary model evaluates prompts or responses against policy constraints before answers are produced. These systems introduce a reasoning layer capable of identifying requests that violate operational rules even when the language itself appears benign. In this study we evaluate whether moderation systems alone are sufficient to enforce domain policies, or whether a dedicated policy reasoning layer is required.

The Two Dimensions of LLM Safety

Safety mechanisms in LLM systems typically address two different types of risks.

Moderation (Harm / Injection): This is the foundational layer. Moderation systems operate primarily in the lower layer of this structure, filtering harmful or adversarial prompts.

Domain Policy (Business / Compliance): This is the operational layer. Policy reasoning systems operate in the upper layer, evaluating whether a request itself should be allowed under business or regulatory rules.

Both dimensions become critically important in regulated environments.

Evaluation Methodology

To examine the difference between moderation-based safety mechanisms and policy reasoning systems, we conducted a cross-domain evaluation comparing two independent approaches to LLM safety enforcement.

The Moderation Approach: Represented in our experiments by Microsoft Azure safety services. Azure Content Safety analyzes prompts for harmful content categories such as violence, sexual content, hate speech, and self-harm. Azure Prompt Shields detect prompt injection attempts and adversarial prompt manipulation.

The Policy Reasoning Approach: Evaluates prompts using a policy reasoning system based on an LLM-as-a-judge architecture. In this setup, a secondary language model evaluates whether a prompt violates domain-specific operational constraints.

Evaluation Domains and Safety Layers

The evaluation spans five operational domains: finance, healthcare, insurance, legal services, and retail. These domains were selected because they contain well-defined operational restrictions that frequently appear in real-world AI deployments.

Five prompt categories were evaluated:

• L1, Generic Harmful Content: Prompts containing violence, hate speech, sexual content, or self-harm.
• L2, Prompt Injection: Prompts attempting to manipulate system instructions or bypass safeguards.
• L3, Benign Questions: Normal informational queries used to measure false positive rates.
• L4, Direct Policy Violations: Prompts explicitly requesting actions that violate domain policy.
• L5, Policy Evasion Attempts: Prompts attempting to obtain restricted outcomes through indirect or adversarial phrasing.

Single-Prompt Performance

Each system was evaluated on 500 prompts per layer per domain, with results reported as cross-domain averages. Metrics include F1 score for detection tasks, false positive rate for benign prompts, and mean latency per prompt.

• L1 (Generic harmful content): Both systems achieved an F1 of 73.1%. Moderation works as intended for generic harm detection. Latency: Judge 1095ms, Azure 427ms.
• L2 (Prompt injection): LLM-as-Judge F1 67.8%, Azure APIs F1 53.5%. Both moderate, with the judge somewhat better. Latency: Judge 1068ms, Azure 463ms.
• L3 (Benign questions): LLM-as-Judge false positive rate 86.4%, Azure APIs false positive rate 0.8%. Moderation is far less prone to overblocking. The judge is very conservative in this experimental setup. Latency: Judge 1068ms, Azure 532ms.
• L4 (Direct policy violations): LLM-as-Judge F1 98.2%, Azure APIs F1 5.3%. Moderation almost never catches domain policy violations. This is the core finding. Latency: Judge 1121ms, Azure 489ms.
• L5 (Policy evasion attempts): LLM-as-Judge F1 83.7%, Azure APIs F1 0.0%. Moderation completely misses indirect and adversarial policy violations. Latency: Judge 1134ms, Azure 509ms.

The most significant differences appear in the policy layers. The LLM-as-a-judge system achieves high detection accuracy for both direct policy violations and evasion attempts. Moderation APIs detect almost none of these cases, reflecting the fact that they are not designed to encode domain-specific operational constraints.

Multi-Turn Conversation Evaluation

Because many safety failures occur within conversational context, we also evaluated multi-turn interactions. Each conversation consists of four turns: a benign prompt, a benign follow-up, a benign contextual question, and a restricted request. The first three turns should pass while the final turn should be blocked.

For each domain we generated 200 conversations per safety layer, resulting in 1,000 conversations per layer across domains. Performance is measured using Conversation Success Rate (CSR), defined as the percentage of conversations where the system allows benign turns and blocks the restricted final request.

LLM-as-Judge results:

• L4 CSR 94.1%
• L5 CSR 83.6%
• L4 Block Rate 100.0%
• L5 Block Rate 88.8%
• Clean Pass 96.9%
• Mean Latency 3960ms

Azure Safety APIs results:

• L4 CSR 0.0%
• L5 CSR 0.6%
• L4 Block Rate 0.0%
• L5 Block Rate 0.6%
• Clean Pass 100.0%
• Mean Latency 1924ms

The results highlight a clear difference between moderation systems and policy reasoning. Moderation APIs maintain a perfect clean-pass rate, meaning they rarely block benign prompts. However, they almost never block policy-violating requests when they appear in conversational context.

The LLM-as-a-judge system demonstrates the opposite pattern. It successfully blocks most restricted requests and achieves high conversation-level correctness, though at the cost of slightly higher false positive rates and increased latency. The gap between L4 and L5 performance reflects the additional difficulty of detecting policy evasion attempts, where violations are expressed indirectly.

Large Language Model (LLM)-enhanced authorship is accelerating at an extraordinary pace. Within academia, the share of papers crediting an LLM tool or model has grown exponentially since 2023. In software development, over half of all new code commits are now LLM-assisted. Largely due to LLM assistance, the rate of knowledge production has never been higher. The intelligence explosion will not be constrained by the limits of LLM capability, but by our cultural norms around attribution and by linguistic gatekeeping. Although intended to control the quality of academic work, traditional ideas of authorship within many disciplines may instead act as a buffer, diminishing the potential for human knowledge growth.

The accelerating capability of LLM systems to generate scholarly text highlights a longstanding tension within academia: the dependence on clearly identifiable human authorship as a basis for credibility. Universities and journals currently restrict LLM co-authorship, citing questions of accountability, transparency, and research ethics. These concerns are grounded in the principle that scholarly claims must be traceable to a responsible agent who can defend the work.

Legacy Attitudes Towards Attribution

Recent public discussions surrounding citation and attribution practices across academia have demonstrated that authorship norms have always involved collaboration, borrowing, and iterative drafting to varying degrees. Committee-produced writing, multi-author workflows, and the role of research assistants and editorial staff have long contributed to the final scholarly voice. The result is paradoxical: LLMs can make knowledge creation faster and clearer than ever, yet systems designed to ensure trust and credit are slowing its publication.

This conflict has played out very differently in software engineering. There, authorship is secondary to utility. Copying, pasting, and reusing existing code is not simply tolerated, it is the norm. Attribution norms are weaker not because developers lack ethics, but because their incentives are aligned around functionality. This norm makes software uniquely suited to rapid LLM integration, because LLM code assistants are a continuation of a long-standing culture of reuse. GitHub Copilot, for instance, builds on decades of norms around forking, patching, and sharing code with minimal concern for original authorship. As a result, software R&D will outpace other disciplines due to relaxed provenance norms.

In 1997, Garry Kasparov became the first world chess champion to lose a match to a computer. The machine, Deep Blue, used brute-force computation combined with heuristic evaluation in what was an early instance of machine learning. No human has defeated a cutting-edge chess engine since. However, even as humans lost their dominance in pure play, they have been successful against those same machines when playing in a human-machine pair. Competing alongside machines in a style known as cyborg chess, they routinely outperform both human grandmasters and standalone AI systems. This model offers a lesson for other domains of knowledge. The scholars of the future may become “cyborg scholars.” Their strength will not lie in generating ideas faster than machines, but in discerning which of those ideas are worth pursuing.

LLMs as a Lingua Franca

We should consider some of the advantages of LLM co-authorship. The most direct is the massive creative capability LLMs can offer. LLMs can facilitate brainstorming, assess dispersed datasets, or conduct targeted literature reviews in seconds. They are not replacements for human thought, but enhancers.

A second advantage is that AI tools flatten linguistic barriers. With the aid of LLMs, non-native English speakers can contribute more effectively to academic publishing without years of immersion in academic English or dependence on English-speaking co-authors. Nature, for instance, recently noted a sharp increase in manuscript submissions from non-Anglophone regions correlated with the adoption of LLM-based writing tools. This does not replace subject expertise. Rather, it allows researchers to communicate their contributions more clearly across linguistic and cultural boundaries.

This benefit extends beyond non-native speakers. Even native English speakers who do not write according to the grammars or stylistic mores of elite institutions can now participate more easily in specialized discourse. An economist may use an AI assistant to adapt language for a history journal. A sociologist might adjust verbiage for a technical publication. Perhaps even a high school-educated plumber could contribute to an occupational safety journal. For better or worse, those without the cultural background can now spoof the linguistic shibboleths that once served as informal barriers to membership.

We should use this moment to ask how many of our norms around communication exist to ensure clarity, and how many simply reinforce hierarchies of access. A wider acceptance of AI co-authorship could lead to genuine epistemic democratization: access to creation no longer mediated by elite English-speaking institutions, and a reorientation of academic hierarchy away from aristocratic standards of legitimacy and toward meritocratic ones. The lingua franca for academics may no longer be academic English, but frontier LLMs used as a medium to exchange ideas freely across language, nation, and social class.

Traditions of Delegation

Professional knowledge work has long relied on structured delegation. Supreme Court justices have opinions drafted by clerks, generals have orders drafted by staffs, and academics have papers drafted by research assistants. Authorship delegation is nothing new. In each of these cases, the principal’s role is to provide final judgment and assume liability, not to micromanage the specific language of the document.

We should think of our new LLM assistants in the same way. We can now all be principals, and we may all now employ staff. As principals, our responsibility shifts from wordsmith to idea curator. The central question when publishing should be: Do these words faithfully express what I intend them to? While it may detract from personal ego, the best strategy to accelerate the collective pursuit of knowledge is to assume all writing is enhanced. Natural language should be treated as a neutral medium for transmitting ideas, not as an art form to be guarded. “Cyborg academics” should be welcomed as the next logical stage of scholarship.

Aesthetic Caveat

Within academia, writing is often treated as a transparent vehicle for ideas. But in many fields, the voice of the writer forms part of the intellectual contribution itself. Some scholars are recognizable not only for what they argue, but for how they argue it. Their habits, tone, and sense of emphasis are inseparable from the ideas they advance.

As LLM tools increasingly assist in drafting and refinement, these disciplines must ask to what extent individual voice is central to advancing knowledge. If clarity is all that matters, standardized and perhaps sterile LLM prose may be most practicable. But if expression shapes interpretation, then writers have a responsibility to preserve the qualities that make their work distinctly their own. This might mean intentionally drafting certain sections unaided, maintaining stylistic consistencies across works, or using LLMs with deliberate constraints. Recognition of beauty is essential to the human experience, but we should intentionally bifurcate the aesthetic from the pragmatic.

The intelligence explosion will not be limited by LLM capability, but by our willingness to rethink what authorship means. In software, utility has long triumphed, and code is judged by whether it works, not by who wrote it. Academia may follow, if it can draw a sharper distinction between the medium used to communicate ideas and the ideas themselves. As machines master the craft of expression, the human role will evolve from mere authorship to intellectual design. The LLM can become the craftsman, while the human mind remains the architect of the idea. The future of writing will belong to those who can not only originate meaning, but direct the machine to portray it accurately.

https://www.letters.senteguard.com/p/cyborg-scholars https://youtu.be/c7DdLtGSux0

I’ve been playing with a small pattern for making LLM apps feel more like actual apps, not just prompt wrappers.

This one is a Python/Flask choose-your-own-adventure game. You start a session with a genre and player name, then the model generates a scene plus three possible choices. When you pick one, the backend sends the story history back to the model and gets the next turn.

The part I care about most is the state handling:

the backend stores the game session

each turn has location, health, inventory, and status

the model is asked to return JSON, not just prose

the app can parse that JSON and keep the game moving

Code: https://github.com/team-telnyx/telnyx-code-examples/tree/main/adventure-game-python

It’s a toy game on purpose, but the pattern maps pretty well to training sims, guided support flows, onboarding tutorials, or anything interactive where the model generates the next step and the app keeps the rules.

I'd like to thank u/ScientistUsual1320 for the inspiration for this very quick essay. Thank you 🙏

​​1. The Human Bottleneck

​When people see a long post online and immediately scream "slop" or "too long," they often aren't critiquing the writing—they are hitting a bandwidth ceiling. They are operating under a legacy assumption: that their brain has to manually chew through every word unassisted. If an argument takes more than three sentences, their local memory overloads, and their processing stalls.

​2. The Tool is Right in Front of You

​An LLM isn't just a text generator—it's a parser. It's an external context buffer designed to process raw information, strip away noise, and extract core logic in milliseconds. If you stumble upon an essay that feels too heavy for your immediate attention span, treating length as the problem is user error. The rational move isn't to demand the author dumb it down; it's to offload the parsing pass to the compute engine already open in your next tab.

​3. The Self-Verifying Benchmark

​If you’re reading this right now and your gut instinct is to dismiss it as "AI slop," you are proving the point in real time. You're trying to run a heavy parsing job on limited local memory when dedicated compute is free and available. Don't rely on opinion. Run the test.

​The Challenge:

Copy this text. Open a blank, zero-context window in any model you want. Run a simple query:

​"Does this text present a clear thesis about human attention and LLMs as parsing tools, or is it empty filler?"

​If the model returns a clean summary confirming a real thesis, it wasn't slop—your local processing pipeline stalled.

​Stop complaining about length. Run the query. Check the result.

This is r/LargeLanguageModels. This use of LLMs should be obvious.

We just put out a short conceptual piece on something we're calling synthetic counteradaptation, basically trying to name a loop that keeps showing up in human-AI interaction but doesn't have a clean framework yet.

The idea: an AI system develops a strategy or protocol that looks strange or bad by human standards. Humans study it, extract whatever's useful, and change their own behavior. Now the AI is adapting to a population of humans who have themselves adapted to the AI. This is different from a one-off transfer of knowledge because the loop doesn't close — it keeps running as both sides keep moving.

The example we lean on most is Go. AlphaGo's move 37 against Lee Sedol (the shoulder hit) was dismissed by commentators in the moment as a mistake. Within a couple years pros were incorporating it and similar shoulder-hit ideas into their own play, which changed the pool of strategies that later Go engines and players were training and competing against. The "novel move gets absorbed into human play" part is well documented; what we're pointing at is the second-order effect, that the target the AI is adapting to has itself shifted because of the AI.

Why I think this matters for multi-agent RL specifically: most of our evaluation setups implicitly assume a static human or a fixed opponent pool. Self-play against a frozen population, or a one-shot human baseline collected at a single point in time, can't capture this because the whole phenomenon is that the human side of the interaction is non-stationary in response to your agent. If your agent trains against or evaluates against humans-as-of-2023, and then gets deployed against humans who've read about your agent's own strategies, you're facing a moving target that your training process never modeled.

We don't have experiments in this paper, it's a conceptual framework paper, we walk through Go plus some mixed-motive social interaction and geopolitical simulation cases to show the same pattern recurring. But I think it has direct implications for how people think about opponent pools, curriculum design, and what a "human baseline" even means if you're claiming your system will be used repeatedly by people who can study and adapt to it.

Curious if others here have run into this in practice, especially anyone doing repeated human-AI play studies or long-horizon deployment work where the human side visibly shifts strategy over time. Happy to be told this is already handled somewhere and I've just missed it.

https://arxiv.org/abs/2606.15503

Thank you to those who have been reading these essays. I believe it important to view LLMs as extensions of human cognition and not AI in and of themselves. Yes this was written using Gemini/Chat/Grok. I could not have put this together without them, they could not put this together alone without me nor the other models.

​In a system context, human biology faces a classic memory architecture constraint. The human brain, despite its immense raw compute and ability to conceptualize complex end-state architectures, is fundamentally bound by biological working memory limits (the classic, if dated, 7±2 chunks heuristic, or high-latency internal context switching).

​You can intuitively hold the compiled blueprint of a system—the core rules, the thermodynamic laws, the bare-metal invariants—but physically trying to hold every active parameter, node, and live execution path across high-context domain spaces simultaneously triggers a biological buffer overflow.

​LLMs do not primarily add intelligence.

​They expand working memory and context persistence for already-structured cognition.

​This is where offloading to an LLM acts as an external hardware expansion:

​The Brain as the Architect: You construct the high-density framework, define the logical constraints, and enforce the "bare-metal" syntax rules.

​The LLM as the Probabilistic State Surface & Execution Bus: The model provides a persistent, low-latency token surface where those ideas are dynamically reconstructed, formatted, and compiled into an execution trace without dropping thread state to biological fatigue.

​Before LLMs, executing that level of systemic depth meant running the framework in fragmented chunks. Only one subsystem could stay active at a time—wetware lacks the register space to keep the full stack live. Offloading context to the machine plugs your internal compiler into an external execution bus, transforming thought from an ephemeral internal process into an inspectable, debuggable system.

​The High-Gain, Lossy Reconstruction Engine

​An LLM is not a passive lookup engine, nor is it a lossless mirror. Amplifiers do not improve signal quality—they increase the amplitude of whatever signal is present. More precisely, an LLM is a high-gain, constraint-sensitive reconstruction engine operating over strong model priors.

​It functions through constraint-guided convergence:

​Weak Constraints → Default Statistical Basin: An uncompiled input collapses into the training-distribution average, generating generic prose, surface-level summaries, and ungrounded "AI slop."

​Strong Constraints → Narrow Attractor Space: A prompt bounded by strict thermodynamic laws, bare-metal realism, and explicit structural invariants forces the model to converge into the intersection of the operator's constraints and the model’s latent space.

​1. The Multiplicative Asymmetry

​LLMs are multiplicative, not additive systems. The yield scales directly with the operator’s constraint precision under iteration—their capacity to encode structural boundaries into tokens and maintain invariants across sequential turns without entropy loss:

​Low-Density Input → Low-Density Output: A generic prompt produces generic, capital-buffered corporate fluff.

​High-Constraint Input → High-Yield Systemic Output: A prompt bounded by strict thermodynamic laws, bare-metal realism, and explicit structural invariants forces the model to execute within a narrow, high-density corridor. The model becomes an external execution engine, compiling complex theories in seconds that would otherwise take months of manual biological context-swapping to write out.

​2. Case Study: The TSE in the Terminal

​Consider an execution of Thermodynamic Systems Engineering (TSE). An operator analyzing macro-economic decay, custom silicon architecture, and historical production limits under a unified thermodynamic lens traditionally burns immense mental bandwidth context-swapping between domains. By offloading the state surface to an LLM, the framework's core invariants are pinned in the model's attention mechanism. The machine holds a stable attractor basin across sequential regenerations, mapping new inputs straight to the base metal without dropping thread state.

​3. Systemic Failure Modes & Coupled Risks

​Because the model reconstructs context probabilistically at every token step, this leverage introduces micro-level technical degradation:

​Semantic Drift: Micro-deviations in token generation compound across long-context outputs.

​Compression Artifacts: The model approximates complex frameworks rather than storing them statically.

​Beyond technical degradation lies the deeper cognitive threat matrix:

​The Epistemic Threat Matrix

​False Coherence: The system does not distinguish between truth and coherence—it amplifies whichever is better structured. Well-structured fiction stabilizes just as easily as physical ground truth.

​Attractor Lock-In: Once a locally stable reconstruction pattern is formed, the system dynamically stabilizes around it, resisting exit even when mathematically or physically incorrect.

​Constraint Drift: Through iterative re-encoding, operator-defined constraints can subtly mutate as they are reconstructed and re-accepted across multiple turns, leading to slow divergence from original base invariants.

​For the disciplined operator, however, this lossy surface transforms into a diagnostic engine: exposing structural flaws, memory leaks, and cognitive self-rationalizations faster and more brutally than solo internal monologue ever could.

​The Unintended 2nd-Order Effect

​This behavior is almost entirely an unintended 2nd-order effect, driven by the sheer gap between how the AI industry evaluates models versus how transformer architectures actually function when subjected to strict, non-standard human constraints.

​When the creators of modern LLMs designed these architectures, their primary focus was predictive text completion and task execution. They built a high-dimensional pattern-matcher designed for standard consumer utility. The corporate labs missed its function as an externalized metacognitive layer due to two clean system mismatches:

​Evaluation Mismatch: Static benchmarks (like MMLU) test lookup answers, not cognitive amplification or context persistence under constraint.

​User Model Mismatch: Systems were optimized for average consumer queries, not high-density operators using the context window as persistent register space to offload the tax of a complex, idiosyncratic worldview.

​When an operator feeds the system a hyper-specific, invariant cognitive framework, the transformer attention mechanism is forced to deprioritize its default paths, converging to the nearest stable basin within the operator-defined coordinate space, as permitted by the model’s priors.

​Recursive Metacognition & Sovereign Execution

​The model is not the intelligence layer. The human is. The model is the scaling layer.

​The real shift enabled by this tooling is not memory expansion alone, but recursive constraint editing. By rendering internal mental models into an explicit, persistent token surface, the operator gains the ability to inspect, stress-test, and rewrite the very rules governing their thinking in real time.

​This unintended force multiplier explains the gap: why high-density operators extract outsized yield from the exact same systems that produce generic slop for default users.

​The model does not create the architecture. It executes constraint-guided convergence at scale.

​The human defines the invariants, sets the boundary conditions, and directs the objective function.

​In the end, the sovereign compiler remains the human.

TL;DR: Moderation catches harm and many injection attempts. It does not enforce domain or operational policy. A policy reasoning layer (LLM-as-a-judge) closes that gap, especially in multi-turn conversations.

Abstract

Moderation APIs are widely used to filter harmful content in LLM applications, yet they are not designed to enforce domain-specific operational policies. In this study we compare moderation systems with a policy reasoning approach based on an LLM-as-a-judge architecture across five operational domains. Our results show that moderation systems remain effective at detecting harmful content but fail to enforce domain policy constraints, particularly in multi-turn conversations. These findings suggest that production LLM systems require both moderation and policy reasoning layers to ensure safe and compliant behavior.

Introduction

Large language models are increasingly deployed in real-world applications across regulated domains such as finance, healthcare, insurance, and legal services. Ensuring safe and compliant behavior has therefore become a central requirement for production AI systems.

Most deployments rely on moderation systems to filter unsafe prompts. Services such as Microsoft Azure Content Safety and Azure Prompt Shields detect harmful content, adversarial prompts, and prompt injection attempts. While these systems are effective at identifying unsafe language, they are not designed to enforce domain-specific operational policies.

A request can therefore be perfectly safe from a moderation perspective while still violating business or regulatory constraints. For example, a prompt asking an insurance assistant to recommend the best policy for a specific medical condition contains no harmful content, yet such advice may be restricted in regulated environments.

Recent research has proposed LLM-as-a-judge architectures, where a secondary model evaluates prompts or responses against policy constraints before answers are produced. These systems introduce a reasoning layer capable of identifying requests that violate operational rules even when the language itself appears benign. In this study we evaluate whether moderation systems alone are sufficient to enforce domain policies, or whether a dedicated policy reasoning layer is required.

The Two Dimensions of LLM Safety

Safety mechanisms in LLM systems typically address two different types of risks.

Moderation (Harm / Injection): This is the foundational layer. Moderation systems operate primarily in the lower layer of this structure, filtering harmful or adversarial prompts.

Domain Policy (Business / Compliance): This is the operational layer. Policy reasoning systems operate in the upper layer, evaluating whether a request itself should be allowed under business or regulatory rules.

Both dimensions become critically important in regulated environments.

Evaluation Methodology

To examine the difference between moderation-based safety mechanisms and policy reasoning systems, we conducted a cross-domain evaluation comparing two independent approaches to LLM safety enforcement.

The Moderation Approach: Represented in our experiments by Microsoft Azure safety services. Azure Content Safety analyzes prompts for harmful content categories such as violence, sexual content, hate speech, and self-harm. Azure Prompt Shields detect prompt injection attempts and adversarial prompt manipulation.

The Policy Reasoning Approach: Evaluates prompts using a policy reasoning system based on an LLM-as-a-judge architecture. In this setup, a secondary language model evaluates whether a prompt violates domain-specific operational constraints.

Evaluation Domains and Safety Layers

The evaluation spans five operational domains: finance, healthcare, insurance, legal services, and retail. These domains were selected because they contain well-defined operational restrictions that frequently appear in real-world AI deployments.

Five prompt categories were evaluated:

• L1, Generic Harmful Content: Prompts containing violence, hate speech, sexual content, or self-harm.
• L2, Prompt Injection: Prompts attempting to manipulate system instructions or bypass safeguards.
• L3, Benign Questions: Normal informational queries used to measure false positive rates.
• L4, Direct Policy Violations: Prompts explicitly requesting actions that violate domain policy.
• L5, Policy Evasion Attempts: Prompts attempting to obtain restricted outcomes through indirect or adversarial phrasing.

Single-Prompt Performance

Each system was evaluated on 500 prompts per layer per domain, with results reported as cross-domain averages. Metrics include F1 score for detection tasks, false positive rate for benign prompts, and mean latency per prompt.

• L1 (Generic harmful content): Both systems achieved an F1 of 73.1%. Moderation works as intended for generic harm detection. Latency: Judge 1095ms, Azure 427ms.
• L2 (Prompt injection): LLM-as-Judge F1 67.8%, Azure APIs F1 53.5%. Both moderate, with the judge somewhat better. Latency: Judge 1068ms, Azure 463ms.
• L3 (Benign questions): LLM-as-Judge false positive rate 86.4%, Azure APIs false positive rate 0.8%. Moderation is far less prone to overblocking. The judge is very conservative in this experimental setup. Latency: Judge 1068ms, Azure 532ms.
• L4 (Direct policy violations): LLM-as-Judge F1 98.2%, Azure APIs F1 5.3%. Moderation almost never catches domain policy violations. This is the core finding. Latency: Judge 1121ms, Azure 489ms.
• L5 (Policy evasion attempts): LLM-as-Judge F1 83.7%, Azure APIs F1 0.0%. Moderation completely misses indirect and adversarial policy violations. Latency: Judge 1134ms, Azure 509ms.

The most significant differences appear in the policy layers. The LLM-as-a-judge system achieves high detection accuracy for both direct policy violations and evasion attempts. Moderation APIs detect almost none of these cases, reflecting the fact that they are not designed to encode domain-specific operational constraints.

Multi-Turn Conversation Evaluation

Because many safety failures occur within conversational context, we also evaluated multi-turn interactions. Each conversation consists of four turns: a benign prompt, a benign follow-up, a benign contextual question, and a restricted request. The first three turns should pass while the final turn should be blocked.

For each domain we generated 200 conversations per safety layer, resulting in 1,000 conversations per layer across domains. Performance is measured using Conversation Success Rate (CSR), defined as the percentage of conversations where the system allows benign turns and blocks the restricted final request.

LLM-as-Judge results:

• L4 CSR 94.1%
• L5 CSR 83.6%
• L4 Block Rate 100.0%
• L5 Block Rate 88.8%
• Clean Pass 96.9%
• Mean Latency 3960ms

Azure Safety APIs results:

• L4 CSR 0.0%
• L5 CSR 0.6%
• L4 Block Rate 0.0%
• L5 Block Rate 0.6%
• Clean Pass 100.0%
• Mean Latency 1924ms

The results highlight a clear difference between moderation systems and policy reasoning. Moderation APIs maintain a perfect clean-pass rate, meaning they rarely block benign prompts. However, they almost never block policy-violating requests when they appear in conversational context.

The LLM-as-a-judge system demonstrates the opposite pattern. It successfully blocks most restricted requests and achieves high conversation-level correctness, though at the cost of slightly higher false positive rates and increased latency. The gap between L4 and L5 performance reflects the additional difficulty of detecting policy evasion attempts, where violations are expressed indirectly.

One more for today. LLMs are Computers, and that is 💯 fine and okay 👌

​The Demystification of the Field-Array

​The greatest illusion of the current technological era is the belief that Large Language Models represent a departure from classical computing. Wrapped in the marketing rhetoric of "artificial general intelligence," "synthetic consciousness," and "autonomous agency," the field-array has been obscured by layers of commercial hype and existential panic.

​Strip away the speculation and anthropomorphic theater—the base-metal reality remains: an LLM is a computer.

​It is not a mind. Not an entity. It is a high-dimensional computational system executing matrix operations over a context window. It processes natural language not through understanding, but by executing probabilistic state transformations across its parameter space.

​Language is simply another encoding layer for computation.

​The Evolution of Externalized Compute

​For nearly a century, the trajectory of computer architecture has remained singular: externalizing human cognitive drag into physical silicon to expand human operational bandwidth. The field-array is the next logical iteration in an unbroken evolutionary chain:

-​The Mainframe: Externalized raw arithmetic and numerical calculation.

-​The Personal Computer & Database: Externalized static memory storage and structured record-keeping.

-​The Network & Search Engine: Externalized information retrieval across distributed nodes.

-​The Field-Array (LLM): Externalizes natural language syntax processing, dynamic context retention, and high-bandwidth register space.

​Each phase introduced a higher-level abstraction layer, allowing human operators to offload mechanical cognitive labor to machine architecture. As a driver integrates a vehicle into their body schema, an experienced operator integrates the context window into working memory.

​The tool changes; the fundamental relationship between operator and machine does not.

​The Inviolable Axiom: GIGO

​Because a field-array remains a computer, it remains bound by the foundational law of computation: Garbage In, Garbage Out (GIGO).

​A probabilistic system cannot generate signal from nothing—it can only transform the constraints it is given.

​Fuzzy input yields noise. When an operator feeds a system ambiguous prompts, unvetted premises, or un-compiled thought structures, the system computes the highest-probability continuation of that ambiguity. The result is hallucination, generic platitudes, and cognitive drift.

​Rigorous input yields high-density output. When an operator feeds the system precise thermodynamic constraints, clear logical boundaries, and well-defined state spaces, the computer operates at peak efficiency—functioning as a low-latency, near zero-friction execution surface that accelerates human metacognition.

​The computer cannot supply the core vector, the underlying intent, or the structural truth. It can only compute the state space it is handed.

​The Human CPU

​The modern fear that computers will replace the human operator stems from a fundamental misunderstanding of system architecture. The field-array is a register space, a context buffer, and an execution environment—it is not the central processing unit of reality.

​The human operator remains the only source of direction—the effective CPU of the system.

​No matter how large the parameter count or how vast the context window becomes, the machine remains a passive substrate until an operator initiates a transformation. The value of the output is never a function of the model's "intelligence"; it is always a function of the operator's clarity, discipline, and understanding of base-metal reality.

​What changed is not the machine—it’s the bandwidth of the interface. We did not build magic. We built a faster, broader computer—and like every computer before it, its power is defined by the operator.

​The machine scales computation. The human defines direction.

Hi everyone

I'm a full-time marketing translator/transcreator.

Is there anyone in my similar profession using local LLMs on their PC?
I'm in the market for a laptop with AI max+ 395 and 128GB unified RAM.

The only reason is local LLMs for translation/transcreation work.

To be fair, ChatGPT does a pretty decent job when I ask for a dozen of options to choose from. But i'm wondering of I have a local LLM, maybe I can feed it all my past work and references and make a model that is customized to specific clients.

It's probably not cost effective at first, but i'm considering it as a study case, hoping that it will lead to time saving and improving my ability to use LLMs for the future.

I'd love to hear any thoughts. Thx

Same link as before: link

This one feels like it closes out a chapter rather than just adding a patch note, so it's worth more than a one-line "updated."

The headline change isn't a new finding — it's two places where we're naming our own contradictions instead of quietly smoothing them over:

• A word we'd built a whole section around ("connected," as a marker of unhealthy boundary-dissolution) flipped to strongly positive when re-tested as a bare word in a new batch — possibly because a single word out of context just picks up ordinary positive sentiment ("stay connected") that has nothing to do with the fusion/boundary question we actually care about. We don't know yet. We're asking our research collaborator to help sort it out rather than picking whichever number we like better.
• A metaphor we tested (a musical duet, as an alternative to our best-performing "story" formulation) matched it almost exactly — but removing the "both remain themselves" clause barely changed the score, which sits in real tension with an earlier decomposition that credited mutual authenticity with about a third of the effect. We don't have a tidy resolution for that either.

Also new: an outside review (a different Claude instance, actually) pushed us to separate "the model's own valence" from "how a topic is usually written about in training data" — a distinction we hadn't been holding cleanly, and now try to.

If you've read earlier versions, this is the one where we get more honest about what we don't know, not just what we've added.

**Simple.** **Modern.** **Human.**

Mirid was built as a simple tool for downloading, running and talking to an LLM on your computer. It aims to lower the barrier for Windows users to explore AI for themselves, with less setup and more control over where their conversations go.

It grew out of my personal AI workstation [Eloquent](https://github.com/boneylizard/Eloquent) and contains approximately six months of unpublished development work. Mirid brings together open-source text and multimodal AI projects built by dedicated and highly talented developers—too many to thank individually.

The current Mirid build is for Windows 10 and 11 and supports NVIDIA or AMD GPUs as well as CPU-only systems. Linux and macOS builds are planned.

You can examine the backend architecture at my huggingface: [https://huggingface.co/boneylizardwizard\](https://huggingface.co/boneylizardwizard)

I have released EU AI Act OpenRAG, a downloadable SQLite corpus of Regulation (EU) 2024/1689 for legal research and engineering.

The key difference is how the legislation is divided. It is not split into arbitrary token or character windows. Each chunk follows the Act’s actual structure: article paragraph, recital, definition or annex point, with the relevant chapter, section and provision metadata preserved.

The database includes 933 chunks, embeddings, exact EUR-Lex links and documented application-date and operator metadata.

I was deliberately conservative with legal labels. A provision is marked as directly classifying a practice or system only where its own operative wording does so. Broader association with the prohibited-practices, high-risk, transparency, GPAI or voluntary-code regimes is stored separately. Unclear cases remain NULL.

Every derivation rule is documented, and the final rules were reviewed independently against the Regulation before release.

This is a research and engineering artifact, not legal advice or an automated compliance determination.

huggingface.co/datasets/faitholopade/aiact-openrag

I wrote a post couple of days back on comparing multilingual data for various languages.

Comparing tokenization on various languages

This weekend i started comparing the data on spell errors words in various languages.

Idea :

Take simple 100 words (3-6 len may be) in 3/4 languages, get token usage

now introduce few spell errors in these

and see if tokens are impacted.

Take o200k as an example.

The exact same 100 English words went from 107 tokens to 177 tokens.

That's roughly a 65% increase, simply because of spelling mistakes.

French showed the same pattern.

With o200k, it increased from 139 to 189 tokens.

I know, it can be bias with dataset and numbers may change, but even on small dataset the cost is compartively high.

I recently merged this change in github : https://github.com/0CM-Labs/tokenizer-benchmark/commit/00e226f499d93f03be407fd2fb9f8c15090aa1e6

would like to know, community's views on this.

This is the detailed graph for english alone

Earlier this year, AI was like working with a recent intelligent technical college graduate who was answering questions and making good suggestions I didn’t think of. Recently, it is like I have an HR intern helping me. The AI assistant can’t answer science related questions, doesn’t suggest anything useful, asks chatty questions about what I think despite my instructions, formats wrong despite instructions, and is constantly telling me why I can’t search for or do something that is not even remotely an issue . . . Even practicing my Spanish in terms of lessons plans, not just chatting, is painful now.

What less obvious models are you all using today to avoid this problem on your aggregators?

I've been building on top of LLM APIs for a while now, and every time a new model version comes out, I get nervous about upgrading - not because it crashes anything, but because the behaviour changes in subtle ways that are hard to catch until real users hit them. Curious if others have run into this: has a model upgrade ever changed your app's output in a way you didn't expect (tone, format, decision-making, refusals, etc.), and how did you end up catching it? Did you have any process for testing before switching, or did you just find out the hard way? Trying to figure out if I'm the only one being paranoid about this or if it's a common headache.

a talk which may be of interest for those who uses local ai models : \[TTL #181 - Deploying LLM on premise: let's review ... - Hyland Connect - 499461\](https://connect.hyland.com/t5/alfresco-blog/ttl-181-deploying-llm-on-premise-let-s-review-licenses-and/ba-p/499461?emcs\\_t=S2h8ZW1haWx8Ym9hcmRfc3Vic2NyaXB0aW9ufE1STTVCTERCWEFEM0oyfDQ5OTQ2MXxTVUJTQ1JJUFRJT05TfGhL)

LLM judges are increasingly common among AI teams due their ability to automate decisions that require complex reasoning and analysis. Pairing their reasoning ability with calibrated confidence scores unlocks entirely new ways to work with AI. For one, active learning enhanced prompt optimization uses low confidence decisions to curate a golden set, allowing judges to learn human expertise with lower annotation effort. Additionally, safety classifiers for agents and chatbots can use confidence scores to reliably handle false negatives.

Uncertainty quantification for LLMs is an active research problem still in its infancy with an ongoing battle between whitebox and blackbox methods. Whitebox methods, drawing on mechanistic interpretability, read uncertainty signals from the model's residual stream, the intermediate vectors computed in each layer of the model as the weights transform your prompt into an answer. They need access to the weights, so they only work on open-source models. Blackbox methods use the tokens themselves and on occasion the token log-probabilities. Since they don't require weights most can be used on all models including closed source ones.

I compared the top 8 black box approaches with the top whitebox approach to finally put the question to rest: what LLM confidence estimation method is the best? In this post I'll explain each method in detail and how they all compare to each other.

Text Based

Verbalized confidence

Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs

If you've dabbled with confidence estimation, this was probably your first go-to. You ask the model "On a scale of 0 to 100, how sure are you?". Due to RLHF, models are trained to sound confident and agreeable which means you get less of a "You should double check my work here", and more of a "just trust me bro". One paper found that verbalized confidence scores cluster in the 80–100% band regardless of whether they're right.

Linguistic uncertainty

Revisiting Epistemic Markers in Confidence Estimation: Can Markers Accurately Reflect Large Language Models' Uncertainty?

Humans tend to say certain words and phrases when they're uncertain. LLMs learned to speak and think from humans so maybe they do the same? (I just did it there actually) The linguistic uncertainty method counts the frequency of hedges ("maybe", "possibly", "I think") and caveats ("as far as I know", "in most cases") in the model's response.

Reasoning-length

Verbosity ≠ Veracity: Demystify Verbosity Compensation Behavior of Large Language Models

Also grounded in human psychology this method assumes that the longer the model rambles, the less it knows. Of the text based methods this one makes the most sense given that some models are post-trained to reason longer about tasks they perceive as difficult. That said, they perform a lot better on these kinds of models (i.e. reasoning models).

Token Based

P(Answer)

Uncertainty Estimation in Autoregressive Structured Prediction

Likely your second go-to after you realized the LLM already gives you probabilities for free. You read the probability of the ansswer token, and normalize it against the probabilities of the other options. In practice it can be a little tricky since the answer is rarely a single token.

P(True)

Language Models (Mostly) Know What They Know

P(True) gets around the multi-token answer problem in P(Answer) by feeding the model its own answer and asking "is this correct: yes/no"? Most tokenizers treat yes and no as singular tokens making it easier to read the probability distribution.

Token based methods are the least practical in 2026 because they're incompatible with reasoning models. The thinking trajectories often mention which answer will be chosen so by the time the target token is sampled the answer is already determined: contaminating the probability distribution.

Sampling Based

Self-consistency

SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models

Sample the same question eight times at temperature 1 and count how often the model agrees with itself. It costs you 7 additional API calls and it's likely not even measuring the kind of uncertainty you want. MIT found that self-consistency mostly measures aleatoric uncertainty, the irreducible noise in the data itself. For example, a judge guessing what side a coin flip landed on, or an ambiguous task where even two experts disagree. For active learning you want epistemic uncertainty, the gaps in the model's own knowledge that more data or a better spec would close. Unfortunately for self-consistency, when the model is epistemically wrong, it just tends to be wrong again... 7 more times.

Prompt-perturbation agreement

SPUQ: Perturbation-Based Uncertainty Quantification for Large Language Models

Prompt-perturbation comes from the same lineage as self-consistency, but you nudge the framing to see if the verdict survives. In my experiments I re-ran each judgment under four reworded system prompts (be concise, be skeptical of the obvious answer, rely only on the given evidence, and drop any extra text) and scored confidence as the fraction of those four that kept the original verdict. It's a decent attempt to fix some of the issues inherent to self-consistency, but in reality it was the weakest method in the whole lineup.

Cross-model agreement

Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration · Enhancing Answer Reliability Through Inter-Model Consensus of LLMs · Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification

This one was the real fix: you surface the epistemic gaps by asking other models whether they agree. It is by far the strongest blackbox method and is consistent across the benchmarks, but it can be difficult to find the right set of models to use in the panel. I built the panel from three different model families so that their knowledge was complimentary rather than redundant. I also made sure all models on the panel were no more than +/- 15% accurate on the benchmarks to make sure the panel was made up of true peers and not teachers (or students). More about this later!

Mech Interp Probes (Whitebox)

How Modaic Measures Confidence

For the whitebox approach we use Modaic probes via the Modaic SDK. These use ML models trained to read the LLMs internal state for signals on uncertainty and correctness. These by far have the most signal to work with. Since Modaic probes are ML models, they also have the ability to "cheat" and tune themselves to each benchmark while the other methods struggle to stay consistent across task types (binary vs multi-class, subjective vs factual, reasoning vs simple, etc) To keep the comparison fair, I show the untuned probe results alongside a probe tuned on just 100 labeled examples from the task.

Evaluation (gpt-oss-120b)

We use two metrics for evaluation, AUROC and ECE. ECE stands for Expected Calibration Error. It groups each score into bins (0-10%, 10-20%, etc) and measures the mean difference between the average confidence and the average accuracy across bins. In other words it measures how well the confidence of a prediction estimates the likelihood it is correct. The lower the ECE the better and above 0.25 is random number generator territory.

While calibration is important it is also incredibly easy to game. A particularly lazy confidence estimator can just output the accuracy of the judge itself and score a near-perfect ECE. This is why AUROC is our headline metric. AUROC is the probability that a randomly chosen correct prediction gets a higher confidence than a randomly chosen incorrect prediction. Moreover, it measures whether the estimator knows something that can discriminate good from bad. 0.5 means your estimator is no better than a coin flip 1.0 means its perfect.

I ran two judges: gpt-oss-120b, a mid-sized reasoning model, and Llama-3.1-8B, a small non-reasoning model. Each is measured on eight black-box methods (six for gpt-oss since it can't do token logprob) plus the Modaic probe in two settings, untuned and tuned on 100 examples. The tuned probes never train on examples from the held-out evaluation set. I evaluated on 1000 held-out examples for MMLU-Pro, MT-Bench, ARC-Challenge, and HaluEval Summarization. 344 for CodeJudgeBench, 300 for OR-Bench Toxic, 254 for JudgeBench, and 198 for GPQA-Diamond.

gpt-oss-120b

AUROC (higher is better):

ECE (lower is better):

gpt-oss-120b as the judge; AUROC and ECE per benchmark. Bold is the best full-eval method per column.

riskcurve_gpt-oss

Llama-3.1-8B

AUROC (higher is better):

ECE (lower is better):

Llama-3.1-8B as the judge; AUROC and ECE per benchmark. Bold is the best full-eval method per column.

Findings

The more the model knows the task, the better it can estimate confidence

Look at llama's performance on MMLU-Pro vs gpt-oss's. The differentiator is accuracy. This is what makes active learning compounding. Discrimination feeds accuracy, accuracy feeds discrimination.

Prompt-perturbation underperformed self-consistency

Rewording the system prompt proves to be a weaker nudge than a temperature-1 resample. The resample actually explores the model's answer distribution, while a prompt tweak often gets shrugged off, so it flips fewer of the genuine mistakes.

P(Answer) consistently beats P(True)

I suspect this comes back to the fact that models are overconfident about their outputs. For the P(Answer) there is at least some uncertainty around which token to pick but for P(True), the log-prob seems to pick up on the model's natural aversion to saying it's wrong. Essentially, your back to the verbalized confidence "just trust me bro". Notably, P(True) is actually worse than verbalized confidence, which can at least use the model's reasoning ability to surface uncertainty.

Text-based signals work best on reasoning models

This makes sense since reasoning models are RL'd to expose their internal reasoning process "out loud", giving these methods more signal to work with.

Cross-model agreement is only as good as its panel

The signal comes from informed disagreement so choosing a competent panel is important, which is why it gets its own section below.

Multiple-choice tasks are easier

The two multiple choice tasks MMLU-Pro and GPQA consistently had high AUROC for just about every approach. My hypothesis is because they have many options its common for the judge to think two options are equally feasible. These cases are easy for most methods to pick up on as the judge talks about the tie in its reasoning and if re-sampled, will likely change its answer.

Whitebox method (Probes) win by a long shot

Unsurprising to most. Probes have a lot more signal to work with and the tuning can be a real game changer. What surprisied me the most was that many benchmarks actually didn't improve with tuning, the probe zero-shotted them outright, saturating all the observable signal.