Back to newsletter 089 contents | All Javva's articles
Day 1. Okay, okay, I made a silly mistake. I know. I admit to it. I tested my changes, then made a change after that, but I didn't test that change. It was stupid, I admit it. I have no defence, it was just sloppy. The change was just commenting out a line of code that was not needed. I had sort of tested it already, just not committed the change to the repository. So when I realised that, I added the comment-out. Sadly, without noticing, I had commented out a required semi-colon too.
The change, including the commented out code, was reviewed. But it's easy to miss a semi-colon, so the review passed, and the code was checked in. Not much of a problem, really. Except this meant that the code no longer compiled. The build process flagged it before the end of the day, but I was in a meeting so it held over to the next day.
Day 2. It's not hard to fix a missing semi-colon. But today was a busy day, with production issues that came first (a single thread in a client JVM was taking more and more CPU over the day, until it was maxxing the CPU causing really bad performance. Tracking down the cause was interesting, but perhaps a story for another day. Though in short, this was the first memory leak I've ever seen which showed up this way - a chain of intensive lookups got longer and longer over the day because someone had forgotten to remove the unnecessary elements of the chain. Each individual lookup wasn't that long, but by the time you had several hundred thousand in each iteration, and iterations triggered several times a second, we had moved into sucky-slow land).
Anyway, I squeezed my semi-colon into a few gaps when I got a breather, and by the end of the day, I had managed to fix the prod issues, and I had sneaked in the semi-colon code fix, tested and reviewed. Great. Only, now, the code branch was locked. Well that's no problem. Three people can authorise unlocking a bug fix for a locked branch. My immediate boss was one - but he was out sick. So I contacted unlocker alternate-one, Mr. ProjectLead. He was a busy guy, so I assumed it would rollover to the next day. And it did
Day 3. He was a very busy guy. By mid-afternoon, I'd reminded him twice. Before the end of the day I had contacted unlocker alternate-two. He was on holiday. So I suppose it was more accurate to say that before the end of the day I had failed to contact unlocker alternate-two. By now Mr. ProjectLead had gone home, and at this point the full brain-deadness of the procedure I had to follow suddenly hit me. A single character change to our codebase - a required simple syntax fix that was stopping the branch compiling - was going to take three people (me, the reviewer, the unlocker), and four days to apply. And I don't even work for a government department!
Day 4. The brain-dead procedure is completed - but not until the afternoon. I was tempted to request a security review of the code change too, just to see what would happen. But I bit my tongue, held back, and waited with infinite patience.
Day 5. ShortCutJoe, who has worked on the same project for a few years, tells me that nobody uses the brain-dead procedure. "It's brain dead. Whenever we hit a locked branch, we close the bug undelivered, open a new one with the 'QA fix' button ticked, and re-apply the same changes. If you tick the 'QA fix' button when you open the bug, it automatically unlocks the bug for delivery to locked branches". I briefly considered my alternatives - do something that would result in prison; scream; or go for a walk and get a coffee. As I sipped my coffee, I day-dreamed of working for a cowboy dotcom company again. Ahhhh, somewhere where adding a semi-colon to the codebase takes half an hour from the compile error to the fix reaching production. That's the way to program.
BCNU - Javva The Hutt.
Back to newsletter 089 contents