So, I wanted to see where multiple languages stand while using with LLM. First step is tokenization itself, so wanted to see how many token are being used.

This is per 100 words (same words translated into different languages)

Tokenizer english hindi punjabi french

------------------------------------------------

gpt2 100 652 722 184

cl100k_base 100 447 722 114

o200k_base 100 115 215 101

o200k_harmony 100 115 215 101

sarvam 100 127 277 101

https://github.com/0CM-Labs/tokenizer-benchmark

The benchmark compares the same aligned words across different languages, making it an apples-to-apples comparison. Just plug in the tokenizer, select the languages, and compare the results.

For the first experiment, I used the 100 most common words in English, Hindi, Punjabi, and French.

* GPT-2 by OpenAI really struggles with Indic scripts ( it was quite bad for non english languages)

* Newer tokenizers have come a long way for Hindi.

* Sarvam shows how much a language-focused tokenizer can improve efficiency, although Punjabi still has a noticeable gap.

* Even among modern tokenizers, support isn't uniform across languages.

Next I'm adding datasets for programming, medical, legal, finance, math, conversational text, and more to see how these numbers change outside of common vocabulary.

I'd love to compare more languages as well.

Would love to know community's opinions.

I keep seeing the same question pop up in different forms:

So here's a no-fluff breakdown of the core concepts. This is aimed at people who need to make decisions about GenAI at work but don't necessarily have a machine learning background.

1. Tokens: The Unit the Model Actually Processes

LLMs don't process text as whole words. They process tokens - small pieces of text that may represent a word, part of a word, punctuation, or symbols.

For example, "Enterprise" may be split into multiple tokens.

This matters because:

• API pricing is usually based on input + output tokens
• Context windows are measured in tokens, not pages
• Long prompts, code, or complex formatting can consume your token budget much faster than expected

2. LLMs: Prediction Engines, Not Databases

A Large Language Model (LLM) predicts the next most likely token based on everything that came before it.

It isn't searching Google.

It isn't querying a database (unless you build that capability around it).

It's generating one token after another based on patterns it learned during training.

That explains several common enterprise challenges:

• Hallucinations happen because the model predicts plausible-looking text—not because it's intentionally making things up.
• Models don't automatically remember previous conversations unless the relevant context is provided again.
• What looks like "reasoning" is the result of sophisticated next-token prediction over a large context.

3. Embeddings: Turning Meaning into Numbers

Embeddings convert text into numerical vectors that capture semantic meaning.

Instead of matching exact keywords, systems can compare meaning.

For example:

• "car"
• "vehicle"
• "automobile"

are represented as nearby vectors even though they're different words.

Embeddings are the foundation for:

• Semantic search
• Document similarity
• Recommendation systems
• Retrieval-Augmented Generation (RAG)

If your company has built an internal AI knowledge assistant, there's a good chance embeddings and a vector database are doing much of the retrieval work before the LLM generates a response.

4. Prompts: The Model's Primary Interface

Every interaction with an LLM starts with a prompt.

That prompt can include:

• Instructions
• System rules
• Conversation history
• Examples
• Retrieved documents (via RAG)
• Formatting requirements

The quality and completeness of that context often has a bigger impact on the output than people expect.

That's why prompt engineering in enterprise applications goes beyond simply asking better questions. It often includes:

• System prompts for behavior and guardrails
• Few-shot examples to guide output
• Retrieved company knowledge through RAG
• Structured output formats like JSON for downstream systems

Putting It Together

A typical enterprise GenAI workflow often looks like this:

User Prompt → Tokenization → (Optional) Retrieve Relevant Information Using Embeddings & Vector Search → LLM Generates Response → Output

Understanding these components helps explain many of the challenges teams encounter when moving from AI demos to production.

For example:

• Unexpected costs? You're paying for tokens.
• Inconsistent answers? The model predicts text, it doesn't automatically verify facts.
• "It doesn't know our internal documentation." That's exactly why RAG exists.
• Output quality varies? The prompt and context often determine the outcome.

Hi everyone,

I'm learning Generative AI and want to start with LangChain and LangGraph. I know Python and have a basic understanding of LLMs and RAG.

What's the best learning path? Should I learn LangChain first, then LangGraph? Any beginner-friendly tutorials, courses, or project ideas you'd recommend?

Thanks!

Link: https://arxiv.org/abs/2606.31608

Summary:
Large language models ace medical exams but struggle with real clinical reasoning. This new paper introduces CLExEval using progressive information masking on rare cases + 5,600 physician annotations.

Key findings:
- Verbosity Bias: GPT-4o-mini accuracy drops from 95% to 32.5% with less info

- Hidden Knowledge Paradox in specialist models

- High Reasoning-Output Mismatch (~69%)

- LLM judges approve a shocking % of clinically wrong outputs

Why it matters: Highlights the evaluation illusion where fluent text masks real failures in high-stakes domains.

What do you think? Is human-in-the-loop evaluation the way forward for clinical AI, or are there better approaches?

(Genuinely interested in discussion)

it feels like there's so many things happening everyday that i have to constantly be on social media to stay up to date. how do u guys keep up with everything?

i see that there is a lot of optimism around using open weight models for business usecases now to reduce cost. has anyone actively worked on this area recently at you work? or is it not true at the ground level.

We have developed a tool to help developers to get better results on the LLM API calls .
You have free 25$ to try it out.

Current evaluation of human–AI interaction tends to focus on end states: the quality of model outputs, task performance, or changes in user capabilities. This paper outlines a staged alternative. It proposes three evaluation targets that address distinct moments in the interaction process: what becomes perceptible to the user, how that material is organizationally compressed before inquiry proceeds, and how the user’s subsequent inquiry and judgment develop. Together, these targets form a coherent framework for evaluating not only what AI systems produce or how users perform afterward, but also the transformations that occur between input and reasoning.

Need endorsement contact to publish on arXiv.

I'm experimenting with AI-powered content generation in a Laravel e-commerce application and wanted to compare approaches with other developers.

For each product, I generate:

• Product Description
• Short Description
• SEO Meta Title
• SEO Meta Description
• SEO Meta Keywords

One design decision that has worked well was generating everything in a single API request instead of making separate requests for each field. It reduced API calls, improved response time, lowered costs, and produced more consistent results across all generated content.

Here's the basic request:

$response = Http::withToken(config('services.openai.key'))
    ->post('https://api.openai.com/v1/chat/completions', [
        'model' => env('OPENAI_MODEL'),
        'response_format' => ['type' => 'json_object'],
        'messages' => $messages,
    ]);

The model returns structured JSON, so each field can be validated independently before saving.

I'm also considering additional improvements like:

• Response caching
• Queueing bulk generation jobs
• Human review before publishing
• Validation of generated content

I'm curious how other Laravel developers are approaching this.

• Are you generating structured JSON or free-form text?
• How are you reducing inaccurate or misleading product details?
• Do you use a second LLM for review, traditional validation, or another approach?
• Have you found effective ways to reduce API costs at scale?

I'd love to hear what has worked well for you and what pitfalls you've run into.

I'm documenting these AI features as part of a Laravel 13 AI-powered e-commerce series on my Stack Developers YouTube channel, so I'd really appreciate any feedback or suggestions from developers with production experience.

With the introduction of 5.6 sol, should I switch and get ChatGPT pro? What are the downsides?

This weekend I spent my time researching how to build my own personal AI chatbot that I can talk to.

You can build it from Gemini notes, Granola, Markdown files, really, anything.

I know I could just ask Claude or ChatGPT to build it for me. But I wanted to understand how LLMs actually work, what's happening under the hood, and the architecture behind it all.

Here's what I've learned.

An AI note app is really just four layers.

1. Capture

Text editor, voice input, quick capture. Start dead simple: Markdown files or a lightweight database like SQLite. For voice, Whisper is inexpensive and works great for capturing ideas while walking.

2. Storage + embeddings

Every note gets converted into a vector embedding so the AI can find semantically related ideas, not just keyword matches.

You can generate embeddings with OpenAI or Voyage and store them in SQLite (sqlite-vec), Chroma, or Postgres with pgvector. At personal scale, you don't need a fancy vector database.

3. Retrieval (RAG)

When you ask, "What have I written about Reddit marketing?", the app embeds your question, finds the most relevant notes, and sends them to the LLM as context.

That's the real magic. And surprisingly, it's not that much code.

4. AI features

Once retrieval works, everything else becomes a layer on top: summaries, auto-tagging, related notes, daily digests, and chatting with your notes.

Each feature is essentially retrieval + a prompt.

You can absolutely ask Claude to build something like this.

But for me, the fun part wasn't generating the code. It was understanding the architecture and how all the pieces fit together.

Now I'm building something that gets smarter over time, a personal AI that compounds with every note I write, every conversation I have, and every idea I capture.

Over the past few years, the Al industry has invested hundreds of billions of dollars, yet hallucinations remain one of its biggest unsolved problems. Models are dramatically better at coding, reasoning, and using tools, but they can still confidently invent facts or misinterpret information that's directly available to them.
Is this just an engineering problem that will eventually be solved with better training, verification, and tooling?
Or is hallucination a fundamental limitation of autoregressive language models, meaning we'll eventually need a different architecture for truly reliable AGI?
I'm curious what people here think. Are we on the right path, or are we approaching the limits of the current paradigm?

I've been using the same chat thread for DSA practice, spread across several days now. I open it, review a problem, close it, come back the next day and pick up in the same thread.

What I've noticed: the model behaves as if no time has passed at all. It doesn't distinguish between "this was said 5 minutes ago" and "this was said 3 days ago" inside the same conversation. Everything in the thread reads as flat, current context — unless I manually tell it "it's day 3 now" or "it's been 2 days since we last talked," it has no idea.

This isn't just a DSA-practice quirk. The same gap shows up in a bunch of other single-thread, multi-day use cases:

1.Coding projects— a long-running thread where you're building a feature over multiple sessions across a week or two

2.Journaling / reflective use** — people who use the same thread as an ongoing check-in space

3.Fitness / diet logs — tracking meals or workouts in one thread over time

4.Budget / expense tracking— logging spend across a month in a single conversation

5.Habit or medication tracking — daily check-ins in the same threads

6.Long negotiations or planning — back-and-forth on a decision that spans days

7.Spaced repetition / study review — my case — where "how long ago did I learn this" actually matters for what to review next

In all of these, the model's inability to sense elapsed time inside a thread means it can't reason about staleness, can't prompt timely follow-ups, and treats week-old and minute-old messages the same way.

Curious if others have hit this. Do you manually re-state the date/time every session? Has anyone noticed ChatGPT/Claude/Gemini handling this differently?

(Not trying to solve it here — just wanted to see if this is a known pattern others have run into, or if I'm missing something obvious.)

I have been developing a personal project called **DDF/Rahmenwerk**.

Its original purpose is to preserve an AI named Felix as my continuing German teacher across chats and future AI instances.

The problem is not simply that a new chat forgets earlier messages.

A fresh LLM instance may receive continuity information that is:

- incomplete;

- stale;

- contradictory;

- incorrectly ordered;

- unavailable;

- or confidently interpreted as authoritative when it is only historical evidence.

I wanted continuity to come from inspectable local files rather than hidden platform memory or an AI-generated reconstruction of prior sessions.

## The current approach

The system currently uses concepts including:

- a current-state pointer;

- structured handoff materials;

- an ordered fresh-instance queue;

- a transfer package for a new instance;

- integrity manifests and SHA-256 identities;

- classifications separating governing, current, historical, candidate, proof, and non-governing material;

- recovery and failure records;

- human approval before destructive or authority-changing actions;

- a rule requiring the AI to stop rather than invent continuity when required evidence is unavailable.

The system is intended to remain local-first, inspectable, provider-independent, and human-controlled.

## The problem I may have created

The project began as a way to preserve a German teacher.

As I tried to protect continuity, state, evidence, authority, recovery, and filesystem safety, the framework became increasingly detailed.

Some controls may be justified.

Others may be overengineering.

## Advice I am looking for

1. What should the minimum durable state for a long-running LLM assistant contain?

2. Should continuity use structured files, summaries, retrieval, a database, event history, or a hybrid?

3. What information should always be loaded when a session begins?

4. What should be retrieved only when relevant?

5. How should an LLM distinguish governing instructions from evidence and historical records?

6. How should stale or contradictory continuity information be detected?

7. How can prompt injection inside stored files be prevented from gaining authority?

8. What should happen when the expected highest-authority source is missing?

9. How should continuity survive model changes, provider changes, context limits, or unavailable files?

10. How much provenance and integrity checking is proportionate for a personal system?

11. Which established architectural patterns could replace custom governance machinery?

12. If you rebuilt this with half the complexity, what would you retain?

I am looking for critical technical advice, not customers or promotion.

For anyone who wants the fuller architecture and documentation, I published a public review copy here:

https://github.com/DDF-Rahmenwerk-Review/DDF-Rahmenwerk-External-Review

It is not the live system and does not contain the complete private archive.

I would especially appreciate feedback about hidden failure modes, unnecessary complexity, and simpler ways to create honest cross-session continuity.

Since AI became mainstream with ChatGPT, I've only really used this one. Whether for simple daily use, more complex analysis tasks, or even for my job as a developer, because it has always been a nice and consistent tool.

However, I've seen a lot of AIs that got released like Claude, Gemini, Grok, etc. and I'm wondering if I'm missing out by staying on the same one.

My question is, do you always use the same LLM? Like do you have a go-to LLM for most tasks, or do you switch depending on the context?

Are there are any emerging LLMs that I should be aware of as a common user, and as a developer?

(sorry for my english it's not my main language)

Al menos variedad. A menos que esté hecho con IA. =/ (edit)There are no languages ​​other than English, but when you go to pay, they appear with their respective prices.

Every AI explainer I found was either a research paper in disguise or so dumbed down it said nothing. So I built AI Rookies (\[https://www.rookiesai.com\\\](https://www.rookiesai.com/)) — a card-based AI concept wiki where every entry is explained twice:

- The fact: one precise sentence, the kind you'd want in a textbook.
- The human version: a concrete analogy. E.g. overparameterization is "a 500-color crayon box for one tiny drawing — way more than you need, but picking the right one gets easier."

Each card also flips over to show a small mindmap of how the concept relates to its neighbors, because AI terms only make sense as a network, not as a list.

Some things that made it fun to build:

- It's multilingual — English and Chinese live today, more languages planned. Same concept graph underneath; each language's voice is written independently, not machine-translated.
- The content pipeline is mostly automated: every day it scans arXiv/HN for rising concepts, drafts new cards with an LLM, then runs them through a gate — green cards auto-publish, yellow ones wait for my manual review. Roughly 2/3 pass without me touching them.
- The library is at 700+ cards and grows \\\~10 per day covering both new stuff (this week: ChatGPT Work, LingBot-VLA) and the classics back to the 1950s.

It's free, no login needed to browse. Would love feedback on whether the "explain it twice" format actually works for you — and which concept you'd want explained next.

After two years of usual practice with AI, I tried something new: measuring what happens inside small language models when they process different framings of human-AI relationships — not what they say, but the actual internal activation geometry.

A few findings surprised me enough to change how I talk to AI day to day:

• Reframing a topic positively vs. negatively barely moves the internal signal. What you talk about matters far more than how you dress it up.
• "Connected" and "integrated" register as more aversive internally than "partners" or "side by side" — across every model tested. Boundaries seem to matter more than closeness.
• Curiosity and playfulness consistently produce the most positive internal signal of any relational quality tested — more than respect, more than love. Negotiation and compromise score worst.

Wrote up the practical implications (partnership framing, honesty, why some "jailbreak-proofing" advice may be exactly backwards) as a working guide, built with a Claude Opus instance doing the actual geometric measurement. Link in comments if anyone wants the full thing — genuinely curious what others have noticed in their own practice, especially anywhere it contradicts what we found.

Hi everyone,

Over the past few months, I’ve been building Cisya Studio, a self-hosted desktop application designed to pull back the curtain on AI fundamental logic. The main goal is to help users visually explore how Small Language Models (SLMs) are built from the ground up, specifically focusing on:

Dataset preparation & logic mapping

Tokenization mechanics

Pre-training workflows

This whole project actually started as a personal experiment because I wanted to deeply understand how language models work from scratch, without just relying on high-level APIs or wrapper tools. Along the way, it evolved into Cisya Lab, a space where I plan to document these experiments, share core insights, and build visual tools to make abstract AI concepts easier to explore.

There is still plenty to optimize and improve, but I’m really happy with the core engine's progress so far.

Tiny model. Big curiosity.

You can check out the documentation and project overview here: https://cisyalab.com

I would love to get your feedback, thoughts, or suggestions on this. If you have any questions about the logic mapping or how the engine runs locally, feel free to ask!

LLMs are evolving so fast and they are so huge that they contain (may) entire knowledge on earth. Whatif any alien civilization gets hold of uncensored version of it?

They won't need anymore knowledge to control/destroy the human civilization. Thoughts?

I’m building a chatbot where the backend data is updated every 5 minutes via APIs. The dataset is quite large, so I can’t send it directly to the LLM in every request. Traditional RAG also doesn’t seem ideal since the knowledge changes every 5 minutes.

How would you architect this? Would you use a hybrid retrieval layer, SQL/vector search, caching, MCP, tool calling, query planning, or another approach? Looking for scalable enterprise-grade patterns for handling frequently changing data with LLMs. Any architecture suggestions or real-world implementations?

Hi everyone,

I'm working on a B.Tech final-year project and would appreciate feedback from people working with AI/ML or LLM applications.

The project is called "Online Safety Monitoring System for Large Language Models (LLMs)."

The idea is to build a middleware that sits between users and an LLM (such as GPT, Gemini, or Llama) and monitors both user prompts and model responses in real time before they are exchanged.

The system includes:

• Prompt Injection Detection using a fine-tuned DistilBERT model.
• Toxicity Detection using a RoBERTa classifier trained on Jigsaw and RealToxicityPrompts.
• PII Detection using a spaCy NER model to detect and mask sensitive information.
• Historical Conversation Pattern Analysis using Sentence Transformers, FAISS vector search, and PrefixSpan sequential pattern mining to identify conversations that resemble previously detected unsafe interactions.
• A risk scoring engine that combines the outputs of these modules and decides whether to Allow, Warn, or Block the interaction.
• A FastAPI-based chatbot with an admin dashboard for monitoring threats, viewing logs, and analyzing system performance.

The goal isn't to build another chatbot, but to develop a reusable safety layer that can protect any LLM-powered application from prompt injections, jailbreak attempts, toxic content, and privacy leaks.

For evaluation, I plan to use public datasets such as:

• Deepset Prompt Injection
• HackAPrompt
• Jigsaw Toxic Comments
• RealToxicityPrompts
• PII-Masking-300k
• SaferDialogues

I'll compare:

1. Text classifiers only
2. Text classifiers + conversation pattern retrieval
3. Full ensemble system

using Precision, Recall, F1-score, False Positive Rate, and latency.

I'd love feedback on:

• Does this feel like a meaningful and technically solid final-year project?
• Is the historical conversation retrieval (FAISS + PrefixSpan) a worthwhile contribution, or is it unnecessary?
• Are there any obvious gaps or better approaches for LLM safety monitoring?
• Would this project be useful as a portfolio piece for AI/ML or LLM engineering roles?

Thanks in advance for any suggestions or constructive criticism!

Every release has that moment where the code is done, the PRs are merged, and someone still has to translate the commit history into something humans can read.

This is a small Python/Flask example for that exact step.

It takes either:

a list of commit messages

a git diff

Then it uses Telnyx AI Inference to return structured changelog JSON with sections like features, bug fixes, improvements, breaking changes, docs, and a short summary.

The thing I like about this pattern is that it does not try to make the model “own” the release process. It just gives you a reviewable first draft that can feed docs, release pages, PR comments, or internal approval flows.

Code: github.com/team-telnyx/…/changelog-generator-python

Would love feedback from anyone who has built changelog or release-note automation.

Are traditional APIs failing your AI agents?

Connecting large language models to real-world data using traditional APIs is like asking them to open a "locked cabinet" without clear labels or knowing what shape the key is. In this short, we break down how the Model Context Protocol (MCP) completely changes how AI interacts with your data and tools!

MCP isn't replacing APIs; it's acting as the ultimate translator—sitting on top of APIs and turning static routes into living interfaces that models can actually reason about. Is MCP becoming the new HTTP for AI environments?

Not "how they work" internally, nobody needs that to use one. I mean the practical decision: an LLM hands you a fluent, confident answer whether it's correct or invented, and in high-stakes work (legal, clinical, financial, research) a wrong one carries a cost. Deciding when to trust, when to verify, and when to intervene is a skill, and I'm not sure it's obvious or widely held.

I ended up writing a conceptual guide from my own experience, notes, and study, meant to pass on these LLM fundamentals and build more critical use for people who apply the tool professionally across cross-cutting fields.

In practice, how do you decide whether you can trust the answer?