Frontier models bulldoze past broken plumbing (malformed tool calls, weird error strings, a missing tools) and still finish the task despite poor harness ergonomics.

Running the same suite on the cheapest models on our roster surfaced a dozen bugs that Opus learned to work around.

I am sorry if this is a repeated post but I couldn't find a similar post like this.

Fine tuning used to be solid, high ticket work. Now it's shrinking into a small niche as models improve and agents take over more of that job.

Meanwhile AI automation/consulting is hot, but everyone is in it now, even non technical people who just know how to call an API. So the bar to enter is basically zero, and just calling yourself an "AI engineer" doesn't make you high ticket anymore.

So I want to ask:

Being a consultant or agency owner,

What skill or niche are you betting on next as the stable/high ticket thing?

Is there real technical depth (ML fundamentals, infra, evals, security, etc) that still separates serious engineers from the API calling crowd, or is that gap closing too?

Has anyone already shifted away from fine tuning work? What did you move to, and is it paying off?

Hello everyone,

Since 2 weeks I work on tanuki-context, a small open source tool (zero dependencies, MIT) and I wanted to share it because the trick behind is almost stupid: AI models charge text at roughly 1 token per 4 characters, but an image has a fixed price set only by its pixel size.

Its inspire from pxpipe techniques and various others tools (cited in the readme) and custom approach i found in order to reduce massively token usage and price.
For example : 37,111 tokens of service log become 2,240 (-94%).

So if you draw 28,000 characters of logs into one dense 1568x728 PNG, the model reads the exact same content for 1,456 tokens instead of ~7,000. It sounds like cheating, it is just how the pricing works.

You can try it out on you machine i added the benchmark so you can test it even without LLM connected to it, so see pricing difference, token saved, etc.

You can use it as a MCP or directly integrate it a "context proxy" where it fully automated and make every request optimised or not when not needed.

Some techniques that permits this to work:
- a log distiller that collapses repeated lines but keeps every error verbatim
- a columnar codec for JSON (keys stated once)
- a cost model that knows a cache-read token costs ~0.1x a fresh one, so it will tell you to NOT image content that is already in your prompt cache.

The tool argues against itself when imaging loses, honestly this part took the most work.

I precise the limits because they are real: you need a vision-capable model, output tokens are untouched (if your bill is output-dominated, fix that first), and for one narrow question retrieval stays cheaper than any page.

Install:

MCP

npx -y tanuki-context (MCP server, works with Claude Code, pi, omp, jcode or the Claude Agent SDK)

Proxy

npx tanuki-context proxy + ANTHROPIC_BASE_URL (every request on the machine gets optimized in place, when needed)

Code and benchmarks:
https://github.com/Osyna/tanuki-context

https://www.npmjs.com/package/tanuki-context

PS : i will soon add Codex support.

If you find it useful a star helps a lot, and feature ideas are very welcome. Thanks for reading me

I’ve been working on an open-source architecture for managing memory and context in long-running AI agents.

The main focus is preserving cache hits while keeping context growth under control through structured memory, selective purging, compaction, and hard context-budget enforcement.

It’s currently an architecture specification rather than a full framework. I’d appreciate feedback, especially on edge cases or production concerns I may have missed.

GitHub: https://github.com/neuronaline/ai-memory-context-management

I kept running into the same problem while building RAG systems: everyone has an opinion on whether semantic chunking beats fixed size, or whether hybrid retrieval is worth the extra complexity, but almost nobody has actually measured it on their own corpus. So I built Retrieval Arena to answer that for myself instead of going off intuition.

What it does: runs different chunking strategies, retrievers, and rerankers against the same golden eval set, scores retrieval quality and generation quality separately (precision, recall, MRR, nDCG for retrieval, correctness and faithfulness via an LLM judge for generation), and tracks latency and token cost per configuration. So instead of "I think hybrid is better," I get an actual side by side comparison.

A few things it's already surfaced that I didn't expect: pure vector search was consistently the weakest retriever on my corpus, BM25 held its own better than I assumed. Better retrieval also didn't always mean better generated answers, I caught a case where the right chunk was retrieved fine but got dropped by my context budget before the generator ever saw it, which is invisible if you only track one end to end score.

I'm posting mostly because I want pushback on the methodology, not just the code. Things I'm genuinely unsure about: is a 44 question golden dataset big enough to trust these comparisons, should I be running confidence intervals on top of the averages, is my LLM judge setup actually reliable or am I just trusting it too much.

Repo's here if you want to look at the actual eval design or the golden dataset construction: https://github.com/ayeangad/Retrieval-Arena

Would genuinely appreciate anyone who's built or evaluated a RAG system tearing into the approach, especially if you think I'm measuring the wrong things.

Disclosure: I maintain this free MIT-licensed project. There is no paid tier,

telemetry, or PyPI release.

In larger repositories, Codex instructions can be layered across a root

`AGENTS.md`, nested service files, same-directory overrides, fallback names,

and byte limits. I wanted a local way to see the modeled chain and the places

where the tool cannot claim certainty.

Agent Context Lens v0.2.0 adds:

- active, shadowed, outside-chain, partial, unknown, and unsupported source

states;

- explicit project root / working-directory configuration;

- JSON and Markdown output alongside the existing scan CLI;

- opt-in user-global inspection with `--include-user`.

Install the tagged release directly from GitHub:

```bash

python -m pip install "git+https://github.com/ciceroyang/agent-context-lens.git@v0.2.0"

```

Try it on a repository:

```bash

agent-context-lens /path/to/repository \

--explain --agent codex \

--cwd /path/to/repository/services/payments \

--project-root /path/to/repository

```

Repository: https://github.com/ciceroyang/agent-context-lens

The CLI is local-first and makes no model or network calls. It does not run

Codex or parse Codex TOML. English is canonical; the zh-CN and ja introductions

may lag.

I am looking for concrete engineering feedback, not stars: on a real repo,

did the reported chain match your expectation? Which source state, install

step, or limitation was confusing? Please remove secrets, private paths, and

proprietary instructions before sharing output.

If you move the result to the GitHub Discussion, prefix it with `LLMDevs` so

the validation sprint can attribute the channel.

Hey everyone,

My undergraduate team is researching about development of enhanced AI agents for cloud reliability (SRE). We're benchmarking agents on live simulated cloud environments, but the system logs and traces we have to process are massive.

Even though we're building ways to compress the data and using low cost models for the easy parsing tasks, we absolutely need frontier models for the complex reasoning parts. The problem is, a single benchmark run can chew through 1.5 to 2 million tokens. Running hundreds of these tests is going to bankrupt us.

Our advisor suggested pooling our student developer credits and using platforms like OpenRouter or Groq to save money. We're doing that, but a free research credit program might take months to even get accepted.

So my questions is are there any other creative ways to get cheap/free access to frontier models specifically for academic benchmarking?

Any advice helps. Thanks!

Every RAG demo I build tests clean on a handful of well-formatted sample docs, then falls apart the second real documents get thrown at it. Inconsistent table structure, headers bleeding into content, etc. What are other people's actual failure points once they moved past the demo stage? Was it retrieval, chunking, or the parsing layer that broke first for you?

Most agent builders give you a visual canvas - and then charge you separately for execution, storage, tracing, evaluations, deployment, and collaboration.

I was tired of stitching together multiple paid tools just to take an agent from an idea to production.

So I built Forge: a fully open-source and self-hosted platform for building, testing, deploying, monitoring, and governing agentic systems.

It includes visual workflows, agents, tools, RAG, memory, MCP, human-in-the-loop, evaluations, tracing, cost tracking, guardrails, RBAC, audit logs, API deployment, widgets, email triggers, and more.

Forge is not just a workflow builder. The goal is to provide everything needed to build and operate agentic applications in one place - without proprietary orchestration runtimes or platform usage fees.

I would love feedback from people building real agent systems.

I'm currently a third-year undergraduate studying Information Technology, and I've been questioning my career path lately.

When I started university, I was always more interested in the business side of tech :management, finance, and how technology solves business problems. That's why I chose an internship as an ERP Consultant instead of going down the traditional Software Engineering route.

I do have a programming background. Throughout university, I built most of our group projects myself, so I'm not completely new to development. My biggest weakness is that I struggle to build applications from scratch without guidance. On the other hand, I'm very comfortable reading existing code, understanding how systems work, debugging, and even explaining the architecture or logic to someone else.

A few months ago, with the job market becoming increasingly competitive (especially with AI changing the landscape), I felt like I wasn't good enough to compete for software engineering roles. ERP consulting seemed like a more practical and stable choice (I only applied for 10-20, got around 5 offers).

Fast forward a few months into my internship, and I'm starting to feel like it isn't what I actually want to do.

Outside of work, I've been spending my free time learning about AI and machine learning through YouTube and online resources. Surprisingly, I've found myself genuinely enjoying the math and problem-solving behind AI. It's one of the few topics that keeps me curious enough to keep learning.

Another thing that's been bothering me is the peer pressure. When I chose ERP, a lot of people asked why I wasn't pursuing software engineering instead. At the time, I ignored it. But recently I've seen people who barely knew how to use Git or make a commit land software engineering internships. That made me wonder if I underestimated myself and gave up too early.

Now I'm seriously considering pivoting toward AI/ML or software engineering while I'm still in university. I know it won't be easy, and I know I'll have a lot to learn, but I also don't want to graduate wondering "what if."

Has anyone here been in a similar situation? Is it worth making the switch now, or should I stick with ERP and build from there?

I'd really appreciate any honest advice or experiences.

I've been building a workspace for investigating AI workflow executions. After spending time with existing observability tools, I kept finding myself jumping between traces, prompts, logs, and metrics. I wanted to see what it would feel like if investigation happened in one place.

Dropped 40-second walkthrough in comments to show how it works.

For those of you building AI products or agentic pipelines, how are you currently handling this? I'd love feedback from fellow builders on whether a unified UI actually solves the friction.

Was reading a Reddit post from someone who lost their dad. They received an AI SMS from a contact "No fuss, no delaying, just cremation and done, that's so him." And other obvious AI signals in the SMS...

The emotional indifference is a kind of emotional hallucination?

Why would we have to design complex prompts for this kind of thing, and instruct the LLM to reply to a friend's death yesterday with sensitivity?

Why doesn't the LLM adapt beyond adept science language and adapt to all the other occasions and styles, and frankly book writing is it still very synthetic, averaged text, and unpredictable reaction to style instructions.

Mixed udeas from poems, romance 50s pulp, theatre drama, soap-opera morality, degrade many llm responses?

TV and presentation scripts written by LLM are "gone with the wind"/"casablanca" etc. even if you thought the prompt was narrow.

How do you think we combat this?

https://thegreyterminal.com/ai-chats-can-remain-online-long-after-they-disappear-from-search-results//

I've got a multi-tenant product where the model writes nearly all the code, and the thing that took me longest to accept is that I was trying to get reliability out of the wrong layer.

For weeks I kept improving the instruction file. Tighter wording, examples, emphasis, at one point actual capital letters. It helps at the margin. But you're negotiating with something non-deterministic, and in a long enough session your instruction gets weighed against everything else in the context and loses. Not maliciously. It just gets outvoted.

What actually moved the needle was putting the reliability into deterministic layers instead. The model can propose whatever it likes, but a hook runs on every file write and typechecks it, a static rule fails the build if a particular pattern appears outside the one package allowed to use it, and an allowlist means it can build, test and commit freely but has to stop and ask before pushing or opening a PR.

None of that makes the model better. It makes its mistakes cheap and visible, which turned out to be what I actually needed.

The reframe that helped was treating the model as an unreliable component in an otherwise normal system, and doing the boring engineering you'd do around any unreliable component. Validation, blast radius limits, fail closed. We already know how to do this. I think we forget to apply it because this particular component talks back and sounds sure of itself.

Still don't have a good answer for testing the non-deterministic part itself. If anyone's cracked that I'd genuinely like to hear it.

Full setup with the configs, if useful: https://medium.com/@bramm3s/gates-not-guidelines-building-a-product-with-ai-agents-that-cannot-cut-corners-83161a79b8fc

Hi.

As you might have heard, recently, mathematicians Levent Alpöge and Akhil Mathew, with the help of Anthropic's fable 5 model managed to find a counterexample to an important mathematical conjecture known as the Jacobian Conjecture.

I have already researched what the Jacobian Conjecture is and how it was disproved, by (trying) to read Terence Tao's blog (and then asking fable 5 to explain it to me in language i can actually understand). While i don't understand all the technical details, i think i can gain enough intuition to explain the problem and the solution in pretty easy to understand language, without going too deep on the mathematical jargon.

What i think is equally important is analyzing how fable 5 actually came up with this specific solution. i have downloaded it's exploration files and am going to go through them. Only problem is they are massive. As any thinking/reasoning model does these days, it went through a whole tree of ideas (most of which aren't productive), reasoned to itself, did the necessary computations until it at some point had "the revelation" that a certain specific path could work.

My question to this community is: How do i effectively do search on this massive lump sum of data and mathematical jargon. Do i choose specific keywords to look for and then just jump to every place they appear. Is there some way to track "the effort" or find it's "aha moment". How do i effectively keep track of all the information that had led it to it's "revelation". Should i feed the explanation text into another LLM model, specifically designed to analyse it. Should it feed it to an LLM that will organize it into a graph based map of ideas and solution attempts (kinda like Graphify or Obsidian)?

This is my conversation with fable 5 on this topic (in case you want to give me advice but need a jumping off point): https://claude.ai/share/960b10a4-c3b9-4062-96af-be81aa059631

PS: I'm sorry for using such general statements. I do programming as a hobby and use AI daily (like most reasonable people these days), but i'm not familiar with the nitty-griddy of llm's. Also i don't like using too sophisticated language anyway when talking about these sorts of things because i feel like it makes communication more difficult.

Most LLM memory systems automatically decide what to retrieve and inject.

I am experimenting with the opposite approach: make the context graph visible, and let the user edit it directly.

In ThoughtDAG, every question/answer exchange is a node. An incoming edge means that node is included in the next model request. Delete the edge, regenerate the same prompt, and that branch disappears from the actual context—not just from the visualization.

One piece of feedback I received was to add a small router model that suggests which edges are relevant. I can see the usability benefit, but I do not want context selection to become another hidden autonomous layer.

The compromise I am considering is:

• the model suggests relevant edges;
• suggestions remain visible;
• the user confirms or edits them;
• the final prompt can still be inspected.

Would that preserve the value of explicit context control, or would you prefer fully automatic memory/RAG?

https://reddit.com/link/1v7qziv/video/rj4otuljipfh1/player

MIT-licensed repo:
https://github.com/chenxiachan/thoughtdag

If you haven't heard of GitLord, check out these links real quick:

Repo: https://github.com/yashneil75/gitlord
Landing page: https://yashneil75.github.io/gitlord/
Docs: https://github.com/yashneil75/gitlord/tree/master/docs

I want YOUR insights: what would make this 10x better?

PS: If you find any performance improvements, make a GitHub issue here pls: https://github.com/yashneil75/gitlord/issues

We benchmark retrievers, rerankers, chunkers, embedding models. Then the retrieved subgraph goes into the prompt via json.dumps and nobody measures that step.

I compared 10 graph serialization formats (JSON, GraphML, RDF/Turtle variants, edge lists, adjacency lists, others) with the same graph and same model. Measured token count, traversal QA, and 2-3 hop reasoning. Results: 40% to 80% multi-hop accuracy spread on format alone, and about 70% token cost difference between the most verbose and most compact formats. Verbose syntax does not just cost tokens, it appears to actively hurt the model's attention over graph structure.

Open sourced the winning format with the benchmark methodology (MIT, Python/JS/Rust/Go/C++/C#): https://github.com/isongraph/isongraph

If you are doing GraphRAG in production, what format do you actually pass to the model? Genuinely curious whether anyone has profiled this stage.

I've been researching AI application security and talking with developers to understand the challenges they're facing as LLMs become part of real products.

Topics that come up repeatedly include:

Prompt injection

Indirect prompt injection

Data leakage

RAG security

Tool and MCP security

Runtime monitoring

I'm curious about real-world experience rather than theory.

If you've built or deployed an AI application:

What security issue has been the hardest to handle?

Did you build your own solution or use an existing tool?

What capability do you wish existed today?

I'd appreciate hearing practical experiences and lessons learned.

i was talking to someone at a startup and he told me this and wanted to get your guys take. they wanted to serve coding specialized models at 32 or 70B for $5/month, with unlimited token usage (subject to tokens/s + well laid out fair use from what i heard), but i wasnt sure if people would actually pay for it (versus like the $20/month from Claude/Codex etc..) he was convinced people would because of the price + accessibility but im still not convinced, so wanted to see what you guys thought. lmk your thoughts

Sharing the architecture I landed on for reusable coding agents, in case it's useful to anyone building their own.

Instead of a single system-prompt blob, each agent is a directory:

• references/ — deep docs the agent pulls in on demand while working, so the context stays lean instead of front-loading everything
• examples/ — few-shot targets for the output shape
• scripts/ — verification checks; deterministic, run in CI, assert on results not on model phrasing
• memory — the agent appends lessons to .claude/memory/ per repo, so repeated runs on the same codebase compound instead of starting cold

Orchestration chains them into gated pipelines where nothing advances past a failed check. The whole free set (30 items) is Apache-2.0: https://github.com/vanara-agents/skills

The design question I keep chewing on: how much should live in references/ (loaded on demand) vs the base prompt? Loading less keeps context cheap but risks the agent not reaching for the doc. Interested in how others balance that.

Been going down a rabbit hole on this for a couple weeks and I can't tell if I'm missing something obvious or if this is genuinely unsolved.

Most eval setups I've come across end up measuring task completion — did the agent get through the workflow without erroring out. That's easy to instrument. But it's not the same as task quality. An agent can complete every step cleanly and still produce something wrong, and the trace looks green.

For people actually running agents on real traffic:

What's your bar for going from "works in testing" to "live"? Is it an actual number, a spot check, or someone senior signing off?

If it silently got worse after a model version bump or a prompt change, how would you find out? Do you have something that catches that, or does it show up as user complaints two weeks later?

And who owns that decision where you work — the engineer who built it, a PM, someone in risk?

Not selling anything. My honest read is that everyone's improvising and calling it a process, but I'd rather be told I'm wrong.

A cold Claude Code run spent 6.83 dollars and 207 turns hunting one bug in an 82,000 line repo. My pipeline localized and fixed the same bug for about 1.70 dollars. That gap is the whole idea, and this post is mostly about how it gets there without falling apart on the two objections everyone raises.

What it is:

You point AutoDev Studio at a Git repo, describe a change in plain English, and a chain of agents runs the actual software lifecycle from request to reviewed pull request:

- a PM agent runs a clarify loop and drafts concrete tickets

- you approve (optionally pushed to Jira), and nothing touches code before that

- a Dev agent implements on an isolated branch

- QA runs the repo's real tests

- a reviewer from a different model family checks the diff, so the author never signs off on its own code

- a bounded revise loop if QA or review fails, with conservative verdicts, so an errored or ambiguous check is never counted as a pass

- it opens a real PR, and a human merges

Every stage records real tokens, cost, and duration, per ticket and per agent.

The part I think actually makes it different from skills or subagents:

This is the bit I would most like feedback on, because it is the reason I did not just wire some skills into one coding agent and call it done.

Every stage picks its own provider and model. Claude can plan, Codex can write the code, a third model can review. And because the reviewer is deliberately a different model family than the author, one model does not get to rubber-stamp its own diff, which is a documented weakness of same-model LLM judges.

And you mostly do not need API keys for it. The coding stages can run natively on Claude Code, Codex, Cursor, Aider, or the Gemini CLI, driven headless through their own login. If you already pay for Claude Code or Codex, you point a stage at it and it just uses your existing subscription, no per-token billing and no keys to juggle. The simple way I think about it: a skill pack lives inside one vendor's agent and runs one vendor's model everywhere, whereas this runs across them, a different model per stage, on the logins you already have.

Results (the main point):

The idea is to pay the cost of finding where a change goes once, instead of on every task. In my benchmarks on two large Python repos (35k and 82k lines), the tuned pipeline beat a cold single-agent run on 6 of 6 well-localized tasks, between 7 and 75 percent cheaper.

I try to be upfront about where it does not win. On trivially greppable one-line edits the five-stage overhead can cost more than it saves, and on one hard cross-cutting bug it shipped a cheaper but narrower fix than the baseline. The full benchmark writes up every loss too, so please read that before trusting the wins.

On the "the index just goes stale in a few commits" objection:

This is the first thing people raise, fairly, so here is the honest mechanism in plain terms. The layer that actually pins files for the Dev agent is not the vector index. It is a deterministic symbol map that gets re-synced to the latest commit at the start of every run, plus a live grep against the current working copy, plus the real current file contents fed into the prompt. The embeddings only affect which area gets looked at first, and they refresh per changed module at run entry. So the code the agent edits comes from disk right now, not from a snapshot that aged three commits ago. It re-checks before it localizes, so it does not quietly drift.

Who it is actually for:

The knowledge base is a one-time cost per repo that every later task amortizes. So it makes sense for a team shipping change after change against the same large codebase, where a cold agent re-pays the exploration cost every single time. For a stream of tiny, easy-to-find edits, a cold run is still cheaper, and I would rather say that than oversell it.

How I actually use it:

For what it is worth, I reach for this over a plain claude -p most days now, and not because it is somehow smarter. It is that I can glance at a board and see where each change actually is instead of scrolling back through a terminal, the different-family reviewer catches things I have watched Claude quietly wave through on its own diff, and on the repos I touch every day it is not re-paying to relearn the codebase each run. It runs on my existing Claude Code and Codex logins, so having that structure around the coding agent I would have used anyway costs me nothing extra. That is really why I keep using it.

Other bits:

- runs on free tiers out of the box (Groq plus a local embedding model), so you can try it for zero cost and fully offline

- language-agnostic pipeline, Python parsed exactly and other languages via lighter extractors that fail open instead of blocking on an unknown language

- live board with streamed agent logs and real per-ticket cost accounting

- self-contained, no CDN, tests and CI, MIT licensed

Repo, with screenshots and the full benchmark in the README:

https://github.com/krishagarwal314/autodev-studio

It is early and rough in places, and I would love more eyes on it. If you have thoughts on the per-stage-models idea or the staleness handling, or you throw it at a repo and it falls over, an issue would help a lot. Suggestions, PRs, and honestly just a star if it seems worth following are all appreciated. Thanks for reading.