One of the facts of life is that most people like to be accommodating. Customer has a small request … so you say “sure, that can be done, it’s just a 5 minute fix”.
How many times have we said this, only to regret it later? And how many times has this desire to accommodate the client ended up causing problems? The last time for me was 8 years ago and, believe me, never again!
Here’s how the nightmare played out…
1. Request comes from customer to make a small change – just a line of literal text at the top of the page of mobile service bills.
2. For reasons too embarrassing to air here, I agree to do it.
3. It’s now 5pm and I have until 6pm to make the change – plenty of time
4. The text literal was isolated to a single procedure and was implemented as a hard coded constant. What luck! All I had to do is extend a 20 character string to 35 characters. Done in a second, compiled in flash and promoted in an instant – and all in under 5 minutes.
5. Take my family to dinner feeling happy and content.
6. On leaving the restaurant the mobile rings. Could this be thanks from the customer?
7. Starts off that way… Over 50,000 bills had been generated by the bill run that evening and the new literal was correct on all bills.
8. Then the bad news… The chargeable amounts were incorrect on all bills. Each bill included 5 separate phantom tariff increases. Ouch!
The procedure that was modified controlled the processing of any applicable tariff increases during the billable period. The increase rates were held in a 5 element array together with their date of effect. The array is initialised to zeros and only populated when a specific process notifies of an actual tariff increase. No notification meant zero increase. Trouble was that the array was never deliberately initialised. Prior to the modification, another procedure called just before the offending procedure initialised a space in memory which was never utilised to date. When this memory was disposed it was next allocated to the offending procedure which had the effect of initialising the tariff increase array by default. By inserting the longer string constant the location of the tariff array in memory was effectively pushed down past the point where the benefits of the default initialisation were enjoyed.
It sure took a lot longer than 5 minutes to fix the collateral damage caused by the “5 minute” fix. Never again folks!
Have you ever had a “5 minute” fix that actually took 5 minutes?