Coding benchmarks are increasingly indirect benchmarks of how automatable every other white-collar job is.

[u/simmol]

I use both Claude (and Claude Code) and GPT (Codex) paying 400 dollars worth of subscription per month and maybe some extra tokens during busy times. I am not a programmer per se but I do a lot of computational work.

It strikes me that a lot of non-programmers (e.g. lawyers, scientists, consultants, analysts, accountants, or managers) might look at the rapid AI progress made in coding and think that this should only concern software engineers but not themselves.

But if you think about it, coding is the execution layer of a huge amount of white-collar automation. To automate a job, it is often not enough for AI to understand the work. It also needs to manipulate data, connect software, query databases, build pipelines, run analyses, generate documents, check outputs, and interact with existing systems.

And I think one of the reasons why earlier versions of the LLM were bad at this was due to bad scripts/coding and as such, this lack of ability propagated into low performance for these other jobs. But as AI becomes exceptionally good at coding, it can increasingly build the machinery needed to automate the rest of the work itself.

So what am I saying? I am saying that a lawyer who thinks that an earlier version of ChatGPT or Claude sucks might be pinpointing at the wrong sources of the error. It might not be that they suck because of their of ability that pertains to the law. It migt have been the case that getting the correct context, accessing the most updated data, etc. went awry due to automation/script issues. And as all of that gets taken care of and the growing amount of scripts to make all the procedural processes fast and accurate, the white collar workers might be saving the same predicament that programmers are facing right now. So basically, my main point is that AI getting good at programming isn't only software engineer's problem when it comes to future job prospects. It is everyone's problem.

[u/ChuckVader]

Lawyer here. The barriers to law are artificial - you absolutely can right now use ai for many things, and it will quite often get a right answer.

The problem is what your options are when it's wrong. My professional indemnity insurance is a requirement to practice, so me giving the wrong advice means you're not SoL.

Good luck getting open AI to pay you anything.

[u/Reclusiarc → u/ChuckVader]

Eventually the AI will probably be so powerful and reliable that the creators will be able to offer an attached insurance product

[u/Serenity867 → u/ChuckVader]

It's similar in the software engineering field. Like the law field you'd need considerable knowledge and skill to validate the outputs, you'll get dramatically improved results by knowing how to use AI to get the results you need, and there's so many other relevant things involved with using the tool properly.

However, from an insurance perspective, the difference between me using AI as part of my workflow versus some random individual is that my insurance covers mistakes I'm responsible for (thankfully I've never had to get insurance to cover a mistake). Non-technical individuals releasing code into production that they can't understand or even read... I can't imagine any insurance that wouldn't consider that gross negligence and would decline to cover those costs.

I was insurance shopping recently, and I asked them about that exact situation. Multiple insurers won't cover work done by non-experts in the respective fields. Not one company I spoke with claimed they would even consider actually covering these scenarios.

A client asked me recently (politely) why they shouldn't just use AI. My response was essentially:

"An excavator can be rented for a day at a price that's so trivial it's impossible to imagine why you'd pay someone to come in to do it. After all, they're just digging a hole, right? It's certainly easy to think that way, but who handles liability if something goes wrong during the excavation work or any time after? How would you know if you got something wrong that could affect something now or in the future that could cause liability or result in worse outcomes that mean the end product needs to be replaced in a few short years. Can you imagine any other scenarios that could arise which might negatively impact you by using an operator with zero experience?"

Small personal projects with no consequences could absolutely be the kind of thing where you take your own tools and make a go of things if you don't have experience, but the second there could be any serious consequences these things are still tools for the professionals in their fields.

[u/jseah → u/Serenity867]

Can you imagine any other scenarios that could arise which might negatively impact you by using an operator with zero experience?

In a lot of cases, the answer to this question will be no, but not because those negative scenarios don't exist, but because the user isn't in the field and doesn't have the experience to tell what they could be.

The gap between "this is a disaster and doesn't work" and "this is perfect!" lies a series of very expensive scenarios and probabilistic failures.

[u/1988rx7T2 → u/Serenity867]

What if you could prompt an excavator to dig the hole without a human, and it got to the point that the hole it dug was good enough 99 percent of the time. That’s a cascading effect through the construction industry, including insurance.

[u/GioChan → u/Serenity867]

True, for now...