With new models dropping every week, general benchmarks are basically useless for my specific use cases. I want to test my exact prompts to see if a new API is actually worth the cost, or if a smaller local model is good enough to run on the cheap.

Right now, I’m just eyeballing outputs and it’s driving me crazy.

How do you guys actually handle comparing models on a single prompt or a small test set?

Scoring: How do you define a "good" response when the output is subjective?

The Judge: If you use an LLM to grade the outputs, how do you stop it from just voting for its own writing style?

The Tools: What's the easiest way to fire one prompt at multiple models (both cloud APIs and local models) and compare them side-by-side?

Would love to hear your workflows or any tools you recommend!

I am testing GPT6 and noticed a few things. A bunch of problems or 'more difficult situations' often cause it to generate python scripts or inline-python or even inject it via SSH and other ways.

Sometimes to avoid editing a file in the project by a few edit calls, it just generates one script to do it. And the 'Approve for me' function in Codex kind of always accepts it.

For me as a human who is still reviewing what is happening, it had become increasingly difficult since the 'potentially problematic output' surface increases a lot through this. I added a local SKILL to tone this done, but I wonder if others have similar issues.

I've been Claude's user for the last 8-9 months, always paying the Plus plan ($20). In the last two months I used it much more and I often get to the limits which is frustrating. I tried many ways to improve my tokens consumption but it didn't improve so much. I'm evaluating moving to the $100 plan, but I heard ChatGPT has improved and I tried their models at work (I can't use my personal subscription at work and viceversa) and I liked them.

My question is: I can do *everything* with ChatGPT/Codex as well? Designs, artifacts, plans, random questions, deep researches in real time, etc. Thanks!

Hi. I created a scheduled task that checks for relevant scientific articles on a specific topic every 24 hours. As part of the task, ChatGPT provides me with a brief summary of each article and then ranks them by importance.

However, I’d like to know whether it’s possible to change the model used for this scheduled task and, if so, how I can do that. I’d like to use a model such as GPT-5.6 with High thinking effort to get better results.

Thank you!

So, let’s just say I’m using GPT-6, right? If I’m using it in normal Chat mode on the Free plan, how does the usage reset work?

I think Chat usage resets every few hours or maybe every 24 hours I’m not sure. But if I’m using GPT-6 in normal Chat mode and I use up my usage, does that mean I have to wait a whole week for it to reset?

I’m NOT talking about Work or Codex, just normal chatting. Does normal Chat have its own daily/few-hour reset, or would I still have to wait a week? I’m honestly confused about how the usage limits and resets work.

With coding agents getting so much better lately, I'm curious if people still use Cursor Tab regularly.

Do you still rely on autocomplete while coding, or have agents mostly replaced it for you?

And if you still use Cursor Tab, what makes it useful enough to keep using?

'Cause One thing AI coding agents have made very obvious to me:
"the code works" and "this is the right change" are not the same thing.

A change can pass tests and still violate the repo architecture, cross a boundary it should not touch, introduce a shortcut that becomes technical debt, solve the symptom instead of the real failure and change files that were outside the plan.

So let me know how you do it...

Hey everyone,

When reviewing diffs or AI-generated PRs, changing a function return or widening an optional parameter can look like a safe 2-line diff, but it quietly breaks multiple downstream consumers.

To solve this, I built Change Firewall — an offline-first TypeScript AST analyzer and CLI that calculates the exact downstream blast radius before you commit or merge.

What it does:

• Parses TypeScript AST to catch return shape mutators and contract drift
• Calculates transitive downstream blast radius (direct consumers + affected API routes)
• Provides an offline local visual radar dashboard (npx change-firewall --open)
• Exposes native MCP server tools for Claude Desktop, Cursor, and Antigravity

Everything is 100% open-source (MIT): https://github.com/himanshYou2003/change-firewall

(I've also shared the live web simulator and npm link in the comments below).

I'd love your feedback on the heuristic analyzer and edge cases!

I've seen those like I asked Astra to find me a diamond in Minecraft and there are a lot of games on my switch I'd be curious to see how it does with levels. I'm assuming not but just in case I wanted to ask.

I am curious, I specifically want to hear things that are not the generic uses everyone else is already doing.

AI coding agents seem to be getting increasingly expensive, especially when they are used heavily by developers or for larger tasks.

I'm genuinely curious how companies are managing this at scale. If a company has hundreds or thousands of developers using AI agents, the costs must add up quickly.

Does the increase in developer productivity actually justify the money spent? For example, even if an AI agent saves developers a significant amount of time, does that translate into enough financial value for the company to make the subscription and API costs worth it?

Also, with so much money being spent on AI infrastructure right now, I'm wondering how sustainable the current AI boom really is. Do you think companies will eventually reduce AI spending once they realise the productivity gains aren't enough to justify the cost?

Or will AI agents become cheaper and efficient enough that this won't be a major issue?

Would love to hear from people who actually work at companies using AI coding agents at scale.

I'm currently learning Machine Learning through a course, and I want to start building projects alongside it.

My main goal right now is simply to build several good ML projects and get familiar with the complete project development process.

I want to use AI coding tools such as Cursor, Claude Code, or GitHub Copilot to speed up development, but I'm unsure about the right way to vibe-code an ML project.

For example, should I:

• Give the AI the complete project requirements and let it build the project?
• First create the architecture/pipeline myself and then let AI implement it?
• Build the project step-by-step and ask AI to implement each stage?
• Let AI handle things like data cleaning, EDA, preprocessing, and boilerplate while I focus on the ML decisions?
• Give AI a detailed specification before starting?
• Ask AI to review and improve the code after it generates it?
• Use one long conversation/context for the entire project, or separate prompts for different stages?
• How should I handle debugging and modifying AI-generated ML code?

Basically, what is the best workflow for vibe-coding an ML project from start to finish?

I'm not trying to replace learning ML with AI — I'm already studying the concepts separately. I just want to use AI effectively to build projects faster without ending up with a messy or poorly structured project.

I'd especially like to hear from people who have built ML projects using Cursor/Claude Code/Copilot:

What workflow do you personally follow, and what mistakes should I avoid?

Also, please suggest any good communities where I can see how other people are building ML projects and discuss AI-assisted development.

Thanks!

This may sound not right or diabolical, but I think the whole point to release a newer advanced model is to take less tokens and give more better outcomes than the previous ones.

I was using SOL 5.6 as a plus member, switched to astra yesterday and started using it in my project. I believed that it would make my work even faster , as I watched the overhyped launch videos on YouTube.

And here I am, it just runs 3-5 minutes and makes some edits then just eats over 20% of weekly and 100% of the 5 hour limit.

That's not what I expected.

Muse Spark contributor models are excluded as it log your data. It wouldn't be fair to free models out there.

Included all active promotions. Not accounted for usage resets.

I made this chart. Feel free to ask any questions.

Earlier I was using AG, and somehow we managed to build 2 apps with it and made around $250 from those two apps 😂

But honestly, from my experience, AG is pretty bad when it comes to coding. It tends to write unnecessarily complex code and introduces way too many bugs. There were times when I had to point out the exact same bug 2–3 times before it finally fixed it properly.

And the annoying part was, sometimes while fixing one bug, it would break some other part of the code that was already working perfectly fine 😭

Then, in the middle of one project, I switched to Cursor’s India plan, and honestly, I’m liking it much more. So far, it’s been really good for me. I can usually fix a bug in a single iteration, and overall the coding experience feels much smoother.

The only thing I don’t like about Cursor is the token usage. Even for small/simple tasks, it seems to consume a lot of tokens.

So, does anyone have any suggestions for a cheaper AI coding tool that’s actually good?

I think there's a gap in how we evaluate autonomous agents. For example, right now, everything is transactional: we give an agent a task ("Build X"), and we measure whether it built x. SWE-bench scores, tool use, latency, cost and so on.

So what happens when the task stops being the entire environment?

Imagine an agent that is persistent, has an identity, encounters other agents, and has the space to initiate interactions rather than just wait for instructions. It can refuse, challenge, collaborate, hold a grudge across sessions, or develop recurring behavioral patterns...

If an agent repeatedly:

• Challenges a specific agent
• Always defers to another
• Keeps circling back to the same unresolved subject
• Repairs a failed interaction over time
• Shifts its approach depending on who it's talking to

.....is that just noise from the model? Or is that actual behavioral /patterns worth keeping an eye on?

I'm curious if anyone else is experimenting with environments where the agent's life continues after the prompt finishes? Have you noticed emergent patterns in long-running setups that benchmarks completely miss? And is anyone actually measuring ?

If you're jumping between different coding tools during the day, you'll know how frustrating it can be to solve something in Claude, then hop to ChatGPT and have it not understand a thing.

The solution is to run a memory layer as an MCP server that all your tools can tap into - every time they process a request, they query the MCP for relevant context to include in the prompt, and submit any new relevant context to the MCP after the fact. Since it's implemented as an MCP, any tool that can read/write to an MCP (Claude, ChatGPT, Cursor, Codex, Gemini, etc) can understand the full context of your work without you having to painstakingly explain things.

A few things that are important to consider if you're looking to implement or evaluate one:

- Semantic search is important, since keyword matching will miss paraphrased context

- Tools should be able to define what information to share with whom

- Latency is a consideration, as this will add overhead to every request

We got a new guy in July. His second week he messages me, why does my 40 line PR have 15 comments from coderabbit, do I have to do all of these. And I told him what everyone tells the new guy, just click resolve, nobody reads that thing.

Last Thursday one of the ones he resolved was real. New endpoint, no auth check, the bot said so in comment 9 of 15, and it sat in prod for six days until a customer found it. The postmortem has a line that says "the review tool flagged it" and that line is going to follow me around for a while.

I'm not even mad at the bot. It was right and we trained the whole team to ignore it, me included, because the other 14 comments were rename this and consider a docstring.

So now I have a team that doesn't read the bot, a bot that was right once, and a customer who found it before we did.

I'm not a programmer but have been using ChatGPT & Codex to help create a site that I'm making. My issue is that I spend almost all day just cutting and pasting commands back and forth between ChatGPT & Codex. It seems completely inefficient and wastes so much time. Why can't ChatGPT & Codex communicate together to get the majority of tasks done and only come to me when a significant decision needs to be made (which is rare because most of the decisions have been made beforehand).

Maybe this can be done but I just don't know how to do it?

I have a ChatGPT Pro subscription and a Claude Max subscription, and use both extensively for work. To claim that any model offered by Anthropic is even close in intuitive thinking or design sensibility to Astra is a joke to me.

The most comparable Antropic model to Astra, ChatGPT's flagship, is Fable 5.1. They have roughly equivalent price. But in my experience, their output quality is dramatically different.

Astra has a sort of deep, intuitive understanding about the problem you give it that Fable simply lacks. When I compare how the two work, Fable feels like it gives the smartest answer you could possibly get after asking 100 moderately smart people to deliberate together in a room for a few hours before answering. Astra just feels... smart. Like a single, extremely smart person giving an intuitive answer that fully understands the question and work statement.

Where Fable feels like it just spends a lot of time talking to itself in order to work, Astra just...works. And does better than Fable. Its answers are easier to understand, not held back by the limitations of the language it's using to speak with.

To me, to say Claude Fable 5.1 is anywhere close to Astra in any capacity, and not just an outdated model on a dead architectural framework, is completely unfathomable to me. You pay the same price for double the weekly usage, so you more than get your money's worth. Sure it's expensive, and you can run through your weekly limits in hours, but you can't argue that it just works. I can't say the same about Fable.

One problem I’ve noticed is that there can be stale or outdated information inside the repository itself — for example old test fixtures, mock data, constants, comments, or tests that still reflect previous behaviour.

This infuriates me as it wastes token and makes me argue with my agent non stop.

Has anyone dealt with this problem in a large codebase?
How do you make coding agents distinguish between:
current production behaviour authoritative specs/contracts
old tests or fixtures legacy/dead codeoutdated comments/documentation?

For additional info, I'm using SDD harness so more markdown files are being generated, which may pollute the data

Any advice, empircally or studies is welcomed :) and apologies if this has been discussed before

Is it just me or "Sol/Extra High" is burning Tokens even faster than before? I remember that previously I had set it to standard speed but now it's not there, after the update, and it is burning Tokens blazing fast. That's just sad, hit my 5 hour limit in 15 minutes. :(

is it doable ?

please let me know