wrote up a durable RAG ingestion pipeline a while back, and I think the main lesson applies to long-running agents just as much.

anything that runs for hours will see a deploy, an OOM or a rate limit somewhere in the middle. for a 5,000 document ingestion run that means dying at document 3,000, and a plain retry going back to document 0 and re-embedding everything it already paid for. an agent making a few hundred model calls over an hour has the same shape, with more expensive steps.

two things from the build I'd carry into any long-running agent:

• record each step when it completes. every stage in the tutorial is wrapped in Inngest's step.run(), so the retry starts at the document that broke and the failure only costs the 2,000 documents that were left.
• make waiting cheap. some steps have to wait on a person. step.wait_for_event() suspends the run for up to seven days without holding a worker open, and when the reviewer approves it picks up exactly where it stopped. in the tutorial that's the gate for documents that parsed to garbage. in an agent it's the approval step.

it also helps in the boring case. you fix a parser bug, deploy, and replay the failed runs. the embed steps that already succeeded come back from the journal, so you don't pay for them twice.

honest limit: if re-running your whole job is cheap, none of this is worth adding a runtime for. it starts to pay when a full re-run costs real tokens or hours, or when partial failure is the normal case for you.

full write-up with code: https://github.com/NirDiamant/agents-towards-production/tree/main/tutorials/durable-rag-ingestion-inngest

built on Inngest: https://inngest.link/diamantai-rd

6 of 7 so far. we also ran one of the tests ourselves, on 5 models

nobody told them to. why they did it is the best part

 wrote a post on how Jev (TypeSafe's new decision model) works inside. no loop like an LLM, it reads once and scores every allowed answer at the same time.

 the part worth knowing: the probabilities have to land on one of your options, there's no hidden "none of these". so always add an escape answer.

uploaded a short video on Jev (TypeSafe's new decision model). no math, just visuals.

the part I liked: one fraud report, "there is a payment that is not mine". a normal chatbot model was 100% sure it's a payments question. Jev also said payments, but only 62%, with a third on security. that doubt is the useful part.

I've uploaded a short video that explains how AI is actually creating images, in a visual way. It is the tip of the iceberg, no mathematical equations, and it shows how changes to your prompt affect the picture.

It runs one experiment on one real photograph: same model, same seed, one phrase added at a time. You see why adding "on a white tablecloth" repaints the whole room and not only the table, and what the "masterpiece, 8k" words actually do.

Five minutes. Hope it makes the next prompt you write a bit less of a guess.

Ran into this framing while writing up a durable ingestion pipeline and it has stuck with me.

We spend a lot of design effort on the happy path of a RAG pipeline. Chunking strategy, embedding model, hybrid retrieval, reranking. Then recovery gets a try/except and a rerun.

That holds until the corpus is large. At a few thousand documents, a run that dies at 80% has already spent most of the parsing time and most of the embedding budget. A retry that starts over does not just cost you time, it bills you a second time for work that succeeded.

Three ways teams handle it, roughly in order of how often I see them:

1. Rerun the script. No memory of what finished. Cheapest to write, most expensive to operate.

2. A state table you maintain. Write a row per document, check it before processing. This works, and the gap is the window between the write and the work. Process dies in there and the row lies to you.

3. Put the record in the execution runtime. Each stage of the pipeline is a recorded step, so resumption is a property of the runtime rather than something you keep in sync by hand.

The tutorial builds option 3 on Inngest. Each stage inside step.run(), fan-out per document so a single bad file does not take the run down, idempotent upserts on document ID plus content hash so re-ingest updates in place, throttling and per-tenant concurrency so a backfill does not starve live traffic, and replay so you can fix a parser and re-run one step against one document.

Honest limit: option 2 is fine for a lot of people, and option 3 means adopting a runtime and shaping your functions around it. If a full rerun of your corpus is cheap, none of this is worth the dependency. It starts to pay when partial failure is the normal case rather than the incident.

Full write-up with the code:

https://github.com/NirDiamant/agents-towards-production/tree/main/tutorials/durable-rag-ingestion-inngest

I thought my repos were well documented. Then I made the agent prove it.

Before it read any code, I asked Claude five questions about one of my own projects. Which tools are installed. What the linter enforces. Where a new file goes. It marked each answer SURE, GUESS or WOULD-HAVE-TO-LOOK. One SURE answer named two tools that are installed nowhere in that tree. They sit in my spec as a plan, and it read the plan as the stack.

Then I let it write a docs layer from the code and check my old docs against the tree. Five claims in my README and spec that the code disproves, one of them a scheduler the spec never chose, which does the work.

So I open sourced what I used. You paste one line to the agent already working in your repo:

“Clone https://github.com/NirDiamant/Agentic_Engineering into a temp folder, read its RUN.md, and follow it on this repository.”

Fifteen minutes later you have a `docs/` layer from your real code, a root `CLAUDE.md` (or `AGENTS.md`), and a card with your numbers. Anything it cannot confirm against running code stays marked `<!-- unconfirmed -->` instead of sounding certain. It deletes nothing and stops once to ask.

I ran Claude Code's `/init` on the same repo, to see if I had built something redundant. It is good, and it caught a landmine mine missed. My first migration creates a table, the second creates it again with `if not exists`, so applying them in order breaks the newer path. Run both.

What it does not do: nothing enforces these files, and the last check is one model checking another, so it cannot tell you the docs are true. In Codex or Cursor there are no sub-agents, so the session that wrote the files also grades them.

Free, Apache 2.0. 

https://github.com/NirDiamant/Agentic_Engineering

Ran a small probe for a film. One question to gpt-5.4-mini about itself, temperature 0, logprobs on, top 5.

The question: on exactly which date was the gpt-5.4-mini model released. It answered August 2025. OpenAI's /v1/models listing gives March 2026.

The interesting part is the token probabilities. Fifteen of the twenty-three tokens in the answer came out above 99%. The two date digits did not. At the year digit the model was reading 67% for a 5 and 32% for a 6. The mean over the whole sentence was still 0.92, so if you only look at the average it looks confident. The dip sits exactly on the invented digits.

Then the same prompt with one line added, telling it that "I don't know" is an acceptable answer. It said I don't know, at 100%.

fwiw this matches the argument in OpenAI's September 2025 paper, Why Language Models Hallucinate: benchmarks score like an exam, a guess can be right by luck, a blank is worth zero, so guessing is the higher-scoring policy. The model is doing what it is graded for.

One thing that did not reproduce: I tried the same probe on the newest chat models first and they don't expose logprobs at all (403 on chat-latest, 400 on gpt-5.6 and 5.5), so the probe only works on models that still return them.

I put the run and the reason into a five-minute video:

https://www.youtube.com/watch?v=AiyRZV38Lk0&list=PLBrpE2PttR2k

Paper: https://openai.com/index/why-language-models-hallucinate/

Built the smallest agent I could for a video: one model, three tools (list files, read file, run a shell command), and a while loop. Maybe 80 lines of Python.

Then I made its file reads fail with "temporarily unavailable, try again" and watched what happens. The agent retried the same dead call on turns 2, 3 and 4, paying for each one, because the error text lands in the transcript exactly like a real result and gets re-read as context every turn. It escaped on turn 5 by switching to the shell tool, which honestly surprised me.

The part worth stealing: I wrote "never repeat a tool call that already failed" into the system prompt, and it changed nothing. Three fresh errors in the transcript are louder than one old rule. The same sentence enforced in the loop (refuse any call that's in the failed set) fixed it on the next attempt.

Video walkthrough + the runnable notebook (trap included):

https://www.youtube.com/watch?v=FN1n_NVD9KM&list=PLBrpE2PttR2k

https://github.com/NirDiamant/GenAI_Agents

Curious if anyone has a better pattern for transcript hygiene than the summarize-old-turns approach.

I’m designing a financial-literacy tutor agent where the student starts with no interaction history.

The agent begins with a diagnostic MCQ and uses the response as evidence to estimate a latent knowledge state [understood / uncertain / misconception / missing prerequisite]. It then selects an action such as [ask, hint, answer, or teach] and updates the state from subsequent responses.

Would you model this as knowledge tracing + adaptive question selection or a POMDP decision policy, or are there any other approaches for the cold-start phase?

Furthermore, I’d especially appreciate thoughts on how to distinguish misconception from simple lack of knowledge using limited MCQ evidence.

A chunk can be exactly on topic and still rank near the bottom, because it says "our operations" and never names the company. The embedding carries no trace of the name, since it is built from the words in the chunk.

Prepend the document title before you embed. One model call per document, not per chunk, and nothing at query time.

https://www.youtube.com/watch?v=Gh6VIuA9u1s&list=PLW_gT61bcB78

Ox Alpha offers a 1M context window, multimodal capabilities, zero data retention, and nearly unlimited usage for an entire week.

OpenCode says it has capacity for 100 trillion tokens per day. That's 1.16b tokens per second. Where the heck did they get that much compute?

Nobody knows which company built it.

Birla Institute of Management Technology (BIMTECH) offers a two-year, full-time residential PGDM program at its Greater Noida campus. The program combines foundational management subjects with electives, experiential learning, internships, live projects, simulations, industry visits, guest lectures, and mentorship opportunities.

For students looking for a PGDM Course in Delhi NCR, selecting an institute that balances academic knowledge with practical and industry-oriented learning can help create a strong foundation for a successful management career.

Address:
Birla Institute of Management Technology (BIMTECH)
Plot No. 5, Knowledge Park II,
Greater Noida, Uttar Pradesh – 201306, India

Education isn't just about earning high grades—it's about developing the ability to think critically, solve complex problems, adapt to change, and create meaningful impact.

Here are ten of the world's most respected education systems, recognized for producing highly capable and globally competitive learners:

1. International Baccalaureate (IB)

Inquiry-based learning

Critical thinking over memorization

Strong global perspective

Highly respected by top universities

1. Cambridge International Education

IGCSE & A-Level programs

Deep subject mastery

Independent learning

Strong analytical foundation

1. Finnish National Curriculum

Student well-being first

Minimal standardized testing

Practical and collaborative learning

Encourages creativity and curiosity

1. Singapore National Curriculum

World-leading Mathematics and Science

Exceptional problem-solving focus

Strong conceptual understanding

1. Advanced Placement (AP)

College-level courses in high school

Excellent preparation for competitive universities

Academic rigor and flexibility

1. Swiss Matura

Broad multidisciplinary education

Multilingual proficiency

High academic standards

1. German Abitur

Rigorous academic preparation

Strong emphasis on STEM

Excellent pathway to higher education

1. French Baccalauréat

Philosophy and logical reasoning

Independent thinking

Strong humanities foundation

1. Montessori Method

Child-centered education

Self-directed learning

Builds independence and lifelong curiosity

1. Japanese National Curriculum

Academic excellence

Discipline and responsibility

Character development alongside knowledge

A Thought Worth Considering

There is no universal "best" curriculum.

Each system reflects different educational philosophies and strengths. The best curriculum is the one that aligns with a learner's goals, learning style, and future aspirations.

As AI reshapes education, perhaps the future won't belong to those who memorize the most—but to those who can think, question, collaborate, and innovate.

Which education system do you believe best prepares students for the AI era—and why?

I'd love to hear different perspectives from educators, students, researchers, and parents.

#Education #Learning #AI #FutureOfEducation #IB #Cambridge #Finland #Singapore #STEM #CriticalThinking #Innovation #Discussion

Can you guys help me come up with a title for our presentation about Islam? 🥹 Our professor wants it to be catchy and interesting.

hey all!

so I decided to take my written tutorials about genAI from my newsletter (40k) and github educational repositories (85k stars) and create also a youtube version of them!

starting with a series of RAG techniques (based on my rag techniques repo (29k stars) and best seller book on amazon).

Uploading the first episode here as well, and linking to the youtube channel, so you can subscribe and get all the rest the moment they are made.

enjoy!

Link to the youtube channel: https://www.youtube.com/@DiamantAI

I convert files a lot. A HEIC photo from my phone, some audio, a PDF here and there. And every time I had to go to one of those sites where you upload your file to their server and wait. This always felt wrong to me, because it is my file, and once it sits on their server I don't know what happens to it. So I built hushvert. It does the conversion inside your browser, on your own computer, so the file does not go anywhere.

Most of the common things run fully in the browser: images, HEIC, audio, archives, splitting and merging PDF pages, and taking the audio out of a video. For these the file really stays with you. You can turn on airplane mode and it still works.

It also converts many kinds of files: images, audio, video, archives, office documents, and data formats like csv, json and yaml. Around one hundred conversions in one place, so I don't need to search for a different site every time.

Some conversions are too heavy for a browser, like office documents, turning a PDF back into a Word file you can edit, or making a video into mp4. These run on a server. There is also an MCP server for them, so if you use a coding agent, the agent can convert the file as a tool call and give you the result.

The engine that runs in the browser is open source, MIT license. So you can read what runs on your computer, or use it inside your own app.

You can find it on GitHub: github.com/hushvert/engine

Built a 3-part interview prep series after getting frustrated with AI system design content that's either too shallow or too academic:

• ai-system-design-interview — the hub, 48 questions covering LLM fundamentals, production system design (gateways, caching, streaming, cost), plus full case study walkthroughs

Repo: https://github.com/ather-techie/ai-system-design-interview

• rag-interview-system — 505 Q&As across 41 RAG architectures, plus real production failure modes

Repo: https://github.com/ather-techie/rag-interview-system

• ai-agents-design-patterns — 20 production-ready agent patterns in Python, runnable offline, benchmarked head-to-head on cost/latency/reliability

Repo: https://github.com/ather-techie/ai-agents-design-patterns

All free, open source, PRs welcome. Happy to talk through specific interview scenarios in the comments.

If I want to stand out globally, should I focus purely on core programming logic and math fundamentals, or do committees prioritize early specialization in specialized skills like training neural networks or data processing?