JProfiler Get rid of your performance problems and memory leaks! Modern Garbage Collection Tuning Java Performance Training Courses COURSES AVAILABLE NOW. We can provide training courses to handle all your Java performance needs Java Performance Tuning, 2nd ed The classic and most comprehensive book on tuning Java Java Performance Tuning Newsletter Your source of Java performance news. Subscribe now! Enter email: Training online Threading Essentials course JProfiler Get rid of your performance problems and memory leaks!

Back to newsletter 114 contents | All Javva's articles

Internal customer number 6 (no relation to the Prisoner) had a problem. I say 'had' judiciously, as I solved it. Just giving the ending away there, sorry if you prefer some suspense in your stories. Does knowing the ending spoil it for you? Personally, I quite like watching the same film a few times, knowing what happens doesn't spoil it at all, instead it means I know what to look out for, I think I appreciate it even more. And you have to watch the Sixth Sense twice to properly appreciate it - unless someone has already given away the ending, which is a shame. Hmm, I seem to have contradicted myself there. Still, consistency is the refuge of ... consistent people. And they don't matter.

Now, where was I. Ah, yes, Internal customer number 6. Internal customer number 6 had a new release of their product. It was a big release, with a great many changes that had somehow managed to get chunked together. They noticed during QA that things seemed to have slowed down a bit, and some quick testing and comparisons against the current prod version showed that, indeed, not only had everything slowed down quite a bit, but memory usage seemed worse too. Internal customer number 6 asked for my services to find out why they were having problems. Such is my genius that it took only a day to resolve. Well, I say a day, but that's actually the time it took together with testing, lots of emails, preparation, handholding, recommendations, etc. The actual analysis went like this:

Me: Is the QA system up and running
6: Yes.
Me: Can I make one of the processes generate a stack dump a few times?
6: Sure.
Me: can I just check which threads are using CPU using the system tool?
6: Go ahead.
Me: (after doing the above) Okay, give me 10 minutes to look at this ... hang on, hmm, this method in class X, can you pop it up in your IDE?
6: (does so).
Me: Isn't that part of your messaging layer?
6: Yes, it is.
Me: Well, I'm wondering about that intern() call there - that's not happening on every incoming message is it?
6: Oh ... maybe ... let me just check the last version ... ah, that was introduced in this version. Do you think that could be part of the problem?
Me: I think that could possibly be ALL of the problem. Can you test the system with that change rolled back?
6: Will do.

And a couple of hours later, I had confirmation of the most impressive performance bug I had ever seen. Across the full system, that one line change would have cost about 30 CPU cores and several gigabytes of memory.

And the moral of the story is ... funny code is going to get in to your system sooner or later, so you really need a performance regression test or one day you'll get embarrassed by your release.

BCNU - Javva The Hutt. (LinkedIn profile http://www.linkedin.com/in/javathehutt feel free to connect me, my email as javva@ this java performance domain).

Back to newsletter 114 contents