Mistake morphology

Imagine blowing over $200,000 with one errant keystroke, not to mention jeopardizing your employer’s relationship with a $38 billion account!

That story reminded me of my own most costly mistake in computing. Well, at least the apparently most costly mistake — sometimes I shudder to think just how much money was probably lost thanks to a few bad design decisions, but those dollars never get spelled out in a P&L.

Back in 1978, I was a lowly operator in the Data Processing department for the university I attended. My regular shift overlapped the beginning of the university’s work day, and one of my daily tasks was to start the program for the cashier’s office. Yes, this one program ran on a single minicomputer (a Data General Eclipse C-330), accepted inputs from several terminals, and recorded the transactions — which were posted to the database at the end of the day.

My instructions for starting the cashier system included just two steps: delete one file, and run the program.

One day I happened to work a different shift starting in the middle of the day, and the system had just been rebooted from a crash when I arrived. The operations manager was busy getting everything back up, and he asked me to go over to the console and start the cashier’s program.

So I deleted the file, and started the program. (Bet you can see where this is going…)

A little while later, the cashier’s office called. All of their transactions for the day were missing!

My boss, the operations manager, ran over to the console and feverishly scanned back through the console log. Then suddenly he stopped and hung his head, eyes closed. Watching him, I began to get a very uneasy feeling that starting the cashier’s program in the middle of the day might require slightly different steps than those I had employed.

Yes, I had deleted that day’s transaction file. The file I deleted each morning was the transaction file from the day before. But nobody had ever told me what that file was nor why it required deletion, and the procedures for restarting the program were not documented anywhere. So I didn’t lose my job, but unfortunately the cashier’s office lost most of a day of data entry — and probably a lot more time figuring out exactly what transactions needed re-entering.

Nowadays it seems laughable that manual deletion of a transaction file would be a standard step in a daily procedure, but back then programmers were paranoid about letting software automatically delete any file. Besides, they wanted to get the transaction file on the nightly backup, just in case. It could have all made sense, if only they had communicated their design a little better.

Since then our trust in software to manage its own internal affairs has grown to the point where we expect not to be bothered with any details other than our own preferences. File management, cleanup, and software updates (even to the O/S) should just happen as needed. So we’ve migrated our vulnerability from the hands of the ignorant operator to those of the programmer who fails to anticipate every possible scenario. And along the way, I managed to transform myself from the one into the other.