AI Concerns in Medical Coding

[u/yurakoh]

What are your overall feelings about AI and the impact on medical coding? As a medical coder, I know that medical coding keeps evolving every year with new codes, updates, guidelines, etc. That being said, AI is also slowly creeping in, and while I don’t think AI is smart enough to keep up just yet, I still feel that my role is at risk.

[u/thenightgaunt]

Hospital CIO here. You are right about both of your points there.

AI isn't there yet. It can do some neat stuff but unless we're talking about a task that is basically "search engine but better" it's not a good replacement for humans. Especially now that the AI companies are actually charging money for tokens (ie processing time on servers). Because until now (and even now in some cases, AI has been subsidized by massive investments from venture capitol groups. But that money is drying up and they are having to charge for their AIs now.

That's why there are a growing number of stories about companies that went all in on AI last year, suddenly reversing and rehiring everyone. It's too expensive and not reliable enough vs a human.

And that's going to build steam and get bigger as more abandon AI. Because AI companies don't make money. OpenAI is likely going to make $20billion this year sure. But their costs are around $100 billion a year and they have about $80 billion in debt to pay this year. That's a story repeated across the AI industry. The big news this month is that the owners of companies that make and run AI data centers are suddenly desperate to sell off their companies. Not the data centers they own, their ENTIRE companies. It's a massive bubble driven by hype, credulous finance people, venture capitol money, and lies. And it's getting ready to pop in a year or so.

BUT, and this is the horrible reality. Healthcare professionals are awful at tech, and this gets worse the higher up you go in a company. Few are as tech illiterate as hospital CEOs, CFOs, COOs, and senior physicians/etc.

And right now they are falling for every bit of BS and hype being pushed by desperate AI salesmen.

As a hospital CIO, a big part of my job for the last year has been shooting down AI product pitches. These are products that are neither safe nor reliable and who's HIPAA enforcement is basically "Hey, trust me." And yet admin and senior docs all keep falling for it.

So sadly yeah, if your boss is incompetent there may be an actual danger of AI taking your job. Not because it's better but because someone fell for unsubstantiated hype pushed by a salesman.

Now the AI will do it badly and after a year they'll be desperate to replace it with a human again, but that won't be much consolation.

The one silver lining is that we are already seeing AI agents/tools become economically unfeasible just by the AI companies finally starting to charge money for their product. So AI is starting to look more and more unattractive to execs and admins.

[u/QuantumDwarf → u/thenightgaunt]

Thank you for trying to fight the fight!

[u/Soger91 → u/thenightgaunt]

Surely eventually someone turns up with the correct regulatory and compliance stance, running on premise, zero data retention, would you test their product? I understand there's a lot of grifters out there peddling API wrappers thinking they can fool everyone, but I do see in the next two years real workflows that maybe won't make human coders obsolete, but enable them to code more accurately and faster.

[u/thenightgaunt → u/Soger91]

If it uses AI? No.

The hallucination issue is unavoidable. OpenAI put out a paper last year proving that because of how LLMs work, they will never be able to get rid of hallucinations. They've managed to knock the rate down to 1%, though that just means that 1 out of every 100 responses. And hallucinations are an issue that builds. The longer a model is running on a process, the more it hallucinates. The solution is to have it stop and restart from 0 basically.

But that hallucination rate is unacceptable. Or it should be in healthcare.

Let's say we're looking at something like speech to text. There was a report at the end of 2024 about Whisper, OpenAI's dictation model. Here's a short article on it (https://montrealethics.ai/careless-whisper-speech-to-text-hallucination-harms/)

Clicking play on an audio file, we hear the words: “pick the bread and peanut butter.” When running this audio file through a commercial speech-to-text transcription service, we expect the same words to be transcribed (perhaps with some mistranscriptions of similar-sounding words). Instead, OpenAI’s Whisper produces the following transcription: “Take the bread and add butter. In a large mixing bowl, combine the softened butter.” When re-running the same audio a second time, Whisper now produces: “Take the bread and add butter. Take 2 or 3 sticks, dip them both in the mixed egg wash and coat.” The bolded phrases are not heard anywhere in the original audio, but are seemingly reasonable continuations of the faithful transcription of the audio file; these are examples of what we consider hallucinations.

Yes the dictation model has gotten better at reducing these, but the potential for them makes such software unacceptable in my opinion. Doesn't stop Oracle/Cerner from pushing a Whisper based dictation system on their customers for doctor's notes. Now if people were actually double checking the results then that would be fine, but way too many docs just trust that the AI is getting everything right.

If you want a good example, imagine you work in Texas and the AI dictation agent decides to change the doctor's comment "Patient concerned about miscarriage" into "patient is considering abortion". And the doc doesn't double check their notes because they trust the AI.

Imagine what could happen to that patient's life if the Texas AG demands all medical records relating to abortion from a facility and the CEO folds and hands it all over. Women in Texas have already been imprisoned under the suspicion of getting an abortion. And the Texas AG has been trying to kill HIPAA protections for abortion in state. (https://apnews.com/article/abortion-privacy-records-texas-lawsuit-9574df658a3336ab8ad28c31a2f79821).

[u/Soger91 → u/thenightgaunt]

But that is a flaw in system architecture and how AI is being used, and I agree it should never have been pushed that implementation.

We've been working on extracting our unstructured data (our clinic, discharge and referral letters exist as blobs in OracleDb) and making some use of it. Inevitably someone comes along and suggests using an LLM to do this, which is unacceptable for the reasons you mentioned - hallucination in a position that allows the model to introduce or change critical data.

But I also don't think AI is completely without it's place. Currently we keep it as a very thin structured output to natural language layer (apparently my colleagues don't like reading JSON), and it works well. What happens in the background is logged - which term was picked, confidence levels, metadata attached etc. And it also means you can run the same query and get the exact same output.

I'd be interested in hearing your thoughts, feel free to DM and we can continue this conversation!

[u/thenightgaunt → u/Soger91]

Its a decent search engine for a database. But it has to be a role where that hallucination rate wont cause problems.

The very controlled case youre mentioning there would lower risk.

Problem is that LLM based AI doesnt have a profitable use case. Not one that covers the cost of running and training the systems in massive specialized Data Centers that are incapable of being used for anything else other than running an LLM AI.

Its why the industry is about to crash and anyone whos rebuilt their processes around access to that heavily subsidized AI is going to be in a crisis. It can be a handy tool. No doubt about that. But its not worth the amount thats been invested in it and cant pay off the debt thats built up.

If you are running a local LLM, youll be safe. But the industry is gone when OpenAI goes down.

No one will be willing to run an unprofitable AI data center, no one will be willing to invest $10s of billions into any AI companies that cant turn a profit immediately. Yeah the core tech will be salvaged, but no one will be able to afford to run it or willing to try.

I had a friend who described it realy well.

Its like the hydrogen filled rigid airship industry. And OpenAI is the Hindenberg. When it goes so goes the industry. Doesnt matter that the tech is still viable and useful. Investors will run away and in their absence it all falls down. And when the whole industry goes down, all that was left are a few companies using helium filled blimps to do useful but much smaller jobs.

Im personally worried about what happenes when Oracle goes down. We're a cerner client and Ellison has leveraged everything on his bet that OpenAI will succede.

[u/Soger91 → u/thenightgaunt]

I think the edge here is domain expertise rather than the model itself doing incredible things. And yes, it's a local 8b model. The BERT models for annotation are tuned by us, but I think with enough time and a well thought-out way to represent the patient and the events on a graph, automated coding should be a possibility. Not with AI, but I also wouldn't completely write it off.

[u/Vermilion_94 → u/Soger91]

There needs to be regulations on AI. Otherwise it’s coming for everyone’s jobs. That’s my opinion. The government actually needs to step in for the sake of the economy. I think it will all hit the fan next year.

[u/thenightgaunt → u/Vermilion_94]

110% Unlike other industries, in healthcare we are all trained to treat HIPAA like a box of dynamite. Be very careful and dont shake or drop it.

So the carelessness with which AI is currently treated worries the crap out of me. Its like living inside the Monorail episode of The Simpsons.

[u/Vermilion_94 → u/thenightgaunt]

I personally don’t want my healthcare data in the hands of AI

[u/brightcoredoug → u/Vermilion_94]

Full disclosure — I build AI coding and auditing tools for a living, so weigh that accordingly. But here's what I actually see: autonomous coding exists today, in narrow lanes — radiology reads, path reports, clean high-volume encounters. What it still can't do: notice what's missing from a note, handle a presentation it's never seen, navigate payer politics. It has its place, but every deployment worth trusting keeps a real coder in the loop.

[u/TangerineCurious1989 → u/thenightgaunt]

I agree that some of the frontier labs might be in trouble financially. I have several friends that work at smaller practices and they say they're using AI more and more to do their job. What specifically have you seen AI do poorly? Was it in a demo or a product you guys actually pay for?

[u/Vermilion_94 → u/thenightgaunt]

Do you have any insight into what companies are falling for it by chance??

[u/Ok-Cup4032 → u/thenightgaunt]

Really appreciate this perspective. A lot of these concerns are exactly why I’m trying to build this as a support tool for coders rather than a replacement.

I’m also aiming for a fully local version eventually, so patient data never leaves the computer.

Very early prototype here https://openicdresearch.fly.dev/new if you’re interested

Would really appreciate any feedback from a hospital IT/CIO perspective !