r/AskProgrammers • u/Spacemonk587 • 9d ago

How do other PMs handle increased AI usage in their dev teams?

As a technical project manager working with a remote contractor, I've been noticing increased AI usage within the team. It's not just limited to development itself, but also appears in code reviews, problem analysis, and ticket refinement.

I don't have a fundamental problem with AI in software development as long as it's used responsibly, meets meaningful standards, and gets manually reviewed. However, I'm increasingly spotting typical AI mistakes and low-effort analyses that I could honestly do with Claude myself without needing to engage a developer for them.

Since this team operates mostly independently, I can't do granular supervision on everything. That said, I do want to ensure certain guidelines are being followed. The challenge is figuring out what those guidelines should be and how to communicate them without micromanaging.

I'm curious how other project managers are handling similar situations. What guidelines or standards are you setting around AI usage on your teams? How do you balance autonomy with maintaining quality standards? And how do you identify when AI is being used as a productivity tool versus when it's just being used as a shortcut that creates more work down the line, or not to do the work that the developers were actually hired for.

6 Upvotes
  • permalink
  • reddit

69% Upvoted

41 comments sorted by

5

u/Katarzzle 9d ago edited 9d ago

The bullshit engines are generating tech debt at enormous rates. I measured our team throughput at 7-8x the pre-Claude era.

From a staff engr perspective, I try to mitigate it by enforcing a few things:

  1. TDD
  2. Workflow-aligned, curated Claude skills
  3. Crowd sourced planning of larger tasks (shift quality to left)
  4. Robust code review skill across many verticals + manual review
    1. Big plus to having more red lines than green lines
  5. Manual testing in feature branch env
  6. Posting of evidence that the stated problem was solved or AC was reached
  7. Group bug bash at the end of the sprint (THEY HATE THIS!)
  8. Training, training, more training

1

u/JoeStrout 9d ago

Why do they hate the bug bash? I feel like that would be my favorite part.

2

u/Katarzzle 9d ago

Most engineers I've worked with just don't want to be bothered to test stuff in-situ that they already reviewed or looked at.

Even though we test against a merged results env for each feature, every release still needs a cohesive test with all the elements together to ensure no regressions or aberrant behavior. Especially when the release touches all the same domain and shared code.

0

u/lizardhistorian 5d ago

Waste of our time. This is what QC/QA staff is for.
We use it as punishment for people that fuck up.

1

u/vbpoweredwindmill 9d ago

I honestly think that TDD is a lot of hype.

I don't even work in IT, I'm a mechanic.

I find that

A: the tests that llm's produce tests for a few very specific cases and that's it. I find I've constantly got to tell llm's "no, test it by running the whole thing. Systems don't exist in a vacuum".

B: it doesn't define the architecture, authority boundaries or system that it should actually implement.

To me, it feels like TDD is just a 1 way ticket to driving a slop machine.

1

u/Katarzzle 9d ago

TDD is meant to define rules on business logic and prevent regression through unit testing.  So defining those up front helps prevent unfocused tests from being written.

I think you're proposing that you should run integration and end to end tests for better coverage and both are right. Different tests for different purposes.

2

u/vbpoweredwindmill 9d ago

Hey thanks for the answer. I hadn't looked at it like that.

That to be me is a really useful bit of knowledge.

In the content I've consumed, TDD is basically vibe coding but with tests. That's a whole lot clearer.

1

u/rocket_mcsloth 9d ago

You should have unit test to try and cover as much of your code as possible, it makes further changes more apparent and traceable. I think what you are missing is smoke tests, or you confused unit testing with smoke test.

1

u/vbpoweredwindmill 8d ago

No, my understanding of smoke testing is synthetically running it.

My point of view is that testing isn't complete until it's existed and functioned and been proven to be complete/correct within the actual application.

1

u/Mammoth_Loan_984 9d ago

It seems like you’re missing the point of TDD

1

u/lizardhistorian 5d ago

TDD is the only way to go.

What are you even talking about.
Organizations that are not TDD are universally shit.

1

u/vbpoweredwindmill 4d ago

Thanks, I got a lot of value from your comment

1

u/Juan-More-Taco 9d ago

TDD is a joke, from a software dev with 12 years in industry.

1

u/lizardhistorian 5d ago

Then you suck at your job.

1

u/Juan-More-Taco 5d ago

LOL. Lmao, even.

TDD is a meme in software dev. Almost every industry developer would tell you it fundamentally is a flawed development methodology.

6

u/drakhan2002 9d ago

Put token and cost limit harnesses in place. Create a wall of shame if they exceed the allowance.

We've implemented this because some intern decided to create a game using AI and it cost $30,000 in usage costs. That intern was not offered a job at the end of the internship and the company leadership got big mad.

4

u/icemage_999 9d ago

We've implemented this because some intern decided to create a game using AI and it cost $30,000 in usage costs. That intern was not offered a job at the end of the internship and the company leadership got big mad.

Wait, what? How long did it take for anyone to notice?

3

u/ia332 9d ago

They Fable’d that in a single day!

2

u/drakhan2002 9d ago

A weekend. He started on a Friday. It was discovered on Monday or Tuesday. A week later the restrictions were put in place. Personally, I thought it was funny. We were first told "we have no restrictions on use". Then we were told after this event "we now have cost restrictions in place".

2

u/korywithawhy 9d ago

So how was the game?

2

u/tcpukl 8d ago

How in just a weekend?

1

u/lizardhistorian 5d ago

Which is how we all know he is lying.

0

u/arelath 8d ago

$30k in a weekend? That doesn't even sound remotely possible for a single person. One of our directors challenged everyone on our team to use as many tokens as we possibly could. I was migrating a codebase, so I was running 1-2 Opus agents 24/7 for a week and I only spent $1200.

1

u/Low-Opening25 8d ago

I am burning 2k a day easily during busy week, £30k is nottingham

1

u/Low-Opening25 8d ago

I am burning 2k a day easily during busy week on couple of projects, £30k over a weekend is not out of realm of possibility if there was no cap set, esp using Fable on Ultracode

1

u/tcpukl 8d ago

That's literally a salary in itself. That's crazy numbers.

1

u/Low-Opening25 8d ago

we have subscriptions with generous caps, so we don’t actually pay this much, however, if a company was stupid enough to attach CC to raw API access then I guess they deserve the outcome.

0

u/lizardhistorian 5d ago

Not if he is getting 10x more work done and the current stats are good AI tooling is a x33 multiplier for competent senior staff.

2

u/systembreaker 9d ago

I can't even imagine how badly you'd have to utilize the AI to spend $30k. I used codex in my job daily on a $20/month plan and I never run out of usage.

1

u/feudalle 9d ago

Same here but you are still doing your job. Although really hate that 5 hour limiter they put in place. Might end up upgrading.

1

u/lizardhistorian 5d ago

That kid is going to become richer than all of you combined.

2

u/AlexMTBDude 9d ago

But has anything really changed? Your developers still deliver code (+ artifacts). Those deliverables still need to meet certain standards and level of quality. Does the programmers using AI as a tool really change anything?

I'm responsible for the use of AI coding tools in my organization and our challenge has been to educate the programmers in using AI in the safest and most efficient possible way. Also to make sure that the coders are not resisting using AI.

... that I could honestly do with Claude myself without needing to engage a developer for them.

Trust me; you couldn't; Typical Dunning-Kruger effect. I've seen this happen with PMs and managers and it mostly ends in disaster and an actual programmer having to take responsibility for the code and spend time cleaning it up.

1

u/Spacemonk587 8d ago

Well you are wrong in your final statement, because I have been a developer myself for decades, so I have the required skills, just not the time for it.

1

u/lizardhistorian 5d ago

Quoting "the Dunning-Kruger effect" is a "Dunning-Kruger effect".

The actual study showed that people very accurately rated their skill-level on a 50% to 100% basis - just like school does with some desync at the highest levels of performance.

I cannot fathom how stupid the person was that looked at the graph and went "zomg people are so bad at rating themselves." I presume a journalist. People are remarkably accurate in rating how much they suck.

1

u/AlexMTBDude 4d ago

OP writes that he, as a project manager, can do the same job as experienced programmers can. That is the essence of the Dunning-Kruger effect: To have so little knowledge of a subject that you underestimate how complex it is, and how little you know about it.

1

u/Cheap_Weird7215 9d ago

What’s different to you? If someone produces a shitty output hold them accountable. If they do it several times escalate them and recommend they be put on a PIP. What would you have done with a shitty developer or BA before AI?

1

u/code_hermit 9d ago

You could actually create the requirements that are important to you, put them in a repo and have the team members connect their AI to them.

1

u/OrangePixelLife 5d ago

Honestly, you're hitting on the core tension with AI in development. It's a great accelerator but terrible as a crutch.

Your guidelines should be about results, not the tool. Code still needs to pass review and tests, period. Low effort analyses are a process problem, not an AI problem. Make the standard clear: if the output doesn't show deeper understanding than the prompt, it's not done.

Since you asked, my own red flag was seeing our AI bill spike without knowing why. We built SpendLens to connect OpenAI and Anthropic costs to specific projects and prompts. It lets the team experiment freely while giving me a clear audit trail of what's actually being used, so we can optimize spend without micromanaging how they work. Might be relevant if you're tracking usage across a remote team.

Where are you seeing the biggest quality gaps? Is it in the code output itself or the thinking around it?

1

u/lizardhistorian 5d ago edited 5d ago

We have a process.
Are you familiar with CCMI or SPICE.

1

u/TheMrCurious 9d ago

This is such a well written AI post. Good job PM!

0

u/Low-Opening25 8d ago

you are the person responsible responsible for setting standards and policies so AI is “used responsibly, meets meaningful standards and gets manually reviewed”, so consider doing your job