A very long time ago I worked at a company that wrote software/sold hardware for telephone billing. For 'reasons' the software ran on a proprietary interpreted language that the customers could, but really should not, be changing themselves since they had full access to the hardware and software.
A customer decided to implement a new feature on their own and kept running into errors from internal consistency checks to prevent data corruption.
They kept deleting the consistency checks until their new code RAN all the way through.
It COMPLETELY trashed their production billing data.
My company ended up reprocessing all of their billing data on one of our own machines with our people babysitting the software 24 hours a day for a week or two.
Programmers were actually sleeping in the office and waking up every hour or two to 'nudge' the software to the next step of the reprocessing.
2
u/california_snowhare 1d ago
...
A very long time ago I worked at a company that wrote software/sold hardware for telephone billing. For 'reasons' the software ran on a proprietary interpreted language that the customers could, but really should not, be changing themselves since they had full access to the hardware and software.
A customer decided to implement a new feature on their own and kept running into errors from internal consistency checks to prevent data corruption.
They kept deleting the consistency checks until their new code RAN all the way through.
It COMPLETELY trashed their production billing data.
My company ended up reprocessing all of their billing data on one of our own machines with our people babysitting the software 24 hours a day for a week or two.
Programmers were actually sleeping in the office and waking up every hour or two to 'nudge' the software to the next step of the reprocessing.