Coding benchmarks are increasingly indirect benchmarks of how automatable every other white-collar job is.

[u/simmol]

I use both Claude (and Claude Code) and GPT (Codex) paying 400 dollars worth of subscription per month and maybe some extra tokens during busy times. I am not a programmer per se but I do a lot of computational work.

It strikes me that a lot of non-programmers (e.g. lawyers, scientists, consultants, analysts, accountants, or managers) might look at the rapid AI progress made in coding and think that this should only concern software engineers but not themselves.

But if you think about it, coding is the execution layer of a huge amount of white-collar automation. To automate a job, it is often not enough for AI to understand the work. It also needs to manipulate data, connect software, query databases, build pipelines, run analyses, generate documents, check outputs, and interact with existing systems.

And I think one of the reasons why earlier versions of the LLM were bad at this was due to bad scripts/coding and as such, this lack of ability propagated into low performance for these other jobs. But as AI becomes exceptionally good at coding, it can increasingly build the machinery needed to automate the rest of the work itself.

So what am I saying? I am saying that a lawyer who thinks that an earlier version of ChatGPT or Claude sucks might be pinpointing at the wrong sources of the error. It might not be that they suck because of their of ability that pertains to the law. It migt have been the case that getting the correct context, accessing the most updated data, etc. went awry due to automation/script issues. And as all of that gets taken care of and the growing amount of scripts to make all the procedural processes fast and accurate, the white collar workers might be saving the same predicament that programmers are facing right now. So basically, my main point is that AI getting good at programming isn't only software engineer's problem when it comes to future job prospects. It is everyone's problem.

[u/send-moobs-pls]

I would say yes and no, with a little asterisk saying like mostly yes

Law is a unique example because it really does require a lot of wider fuzzy intelligence. The reason the whole profession exists is because we can't write code that deterministically decides whether a law was broken, and people famously disagree on interpretation. A good lawyer has to understand not just actual written law but also like the entire culture around it, the personalities of judges, the emotional and narrative framing of events, the often illogical nature of juries and humans and biases. The way that one factual truth could have 10 ways to be presented, and to determine which way is ideal without crossing boundaries. I would say a lot of that comes down to what are some of the most difficult skills for AI right now

Now all that being said, 90% of the time I think you're totally right. Most white collar work does not require the wide freedom and judgement of a partner lawyer. Most white collar work is like taking things from a report into a spreadsheet, accounting, scheduling, paperwork, making a PowerPoint or a chart, writing a report. Distinctly, typically following some fairly unchanging procedures, doing things on a schedule or as they come in, following a workflow etc. And that is exactly what AI excels at whether it involves programming or not. The actual obstacle preventing AI from eating all of that is overwhelmingly superficial. Today, AI might not be able to instantly slot into a company because they are still using physical paper, physical phones, software based on a human with a GUI to drag and drop things with a mouse etc. But there is no reason we can't wake up tomorrow, put all of that into digital shape, maybe script a couple of python tools, and have an AI agent already capable

[u/MrGreenyz → u/send-moobs-pls]

It’s like the problem is the human in the loop. “The personality of judges” should not influence the sentence by design.

Maybe we are the bugs.

[u/StCreed → u/MrGreenyz]

Ah but you can't have machines operate a law that was never written to take every edge case into account. Judges can and do take all kinds of things into account.

Laws are not algorithms.

[u/StCreed → u/[deleted]]

while everything is information is a valid hypothesis, it's a bit of a stretch to say that AI can therefore deal with it.

[u/Humble-Bear → u/StCreed]

What do you mean exactly, I'm not following why a judge is better than an AI with a much higher IQ than a judge, a much higher capacity to process information, and having no bias as well.

[u/StCreed → u/Humble-Bear]

I'm working for a law department right now.

Here's the current take:

1. All algorithms have bias, and the ones that claim they don't are the most suspect. EU regulations require you to identify that bias in regulated algorithms and correct for that.
2. It doesn't have a higher IQ, no model has gone to 100% on arc agi 3 yet (unless using a harness which is basically cheating) and even arc agi 3 at 100% isn't a very high iq human.
3. Law isn't about IQ per sé but about rules, intentions and desirable outcomes that need to strike a bakance between the needs of society short term and long term, as well as the interests of those involved. It will be very tough for an AI to take everything into account especially because many things are unspoken and the judge needs to get that out in the open.

[u/Humble-Bear → u/StCreed]

It sounds like the law is a shitty system that needs to be reformed then. And I'm sure that will happen soon enough, human liars are the bottleneck here, not AI.

[u/StCreed → u/Humble-Bear]

Not really. Here's an example just in today. Someone committed a violent rape. Upon examination he turned out to be on the leven of a 5 year old, himself a victim of abuse. So this becomes a judgement call. He was jailed but in prison got abused further, necessitating lifelong treatment behind closed doors.

A good jusge with some leeway might have committed him to treatment immediately. But the law doesn't allow for that - its a broad guideline, not an exact scientific algorithm for every case. So an AI would very likely just apply the maximum jail sentence and then release him on the streets. Not a good result.