Back to newsletter 081 contents

Threads, Threads, Threads Everywhere

I recently read Heinz Kabutz's The Law of the Distracted Spearfisherman. The basic piece of advice is "know what your threads are doing". So I decided to follow Heinz's advice on our main system. It's several years into deployment, and threads are everywhere. With dozens of different types of Java processes, I decided to start on the most important components. It has over 500 threads and it is a pain to work out what even half of them are doing. So I wrote something that parsed them all, and tried to categorize them.

My week consisted of asking one developer after another pretty much the same things. The conversations would go like this:

• me "what does this thread do?"
• them ...
• me "why does it do that?"
• them ...
• me "why do we have three/four/fifteen/etc threads doing that?"
• them "we do?
• me "oh, you didn't realize there were that many?"
• them "No. Can you send me the dumps" ... then after a bit ... "ah, looks like it shouldn't have that many, I've raised a bug"
• me "How do the extra threads affect the app?"
• them "they don't really, the extra threads are mostly idle, they are just inefficient"

Basically either threads were being spawned unnecessarily, or they weren't cleaned up properly, or they were sitting there as part of a pool. Now the real kicker for me was the analysis showed IT DIDN'T MATTER. Yes, despite many Java processes each with many idle threads sitting around, the OS didn't really care. Sure it was having to do some extra scheduling and context switching - but all the processes were coming in under the SLAs. So I had a most uncomfortable conversation with the project manager which went something like this:

• him "we seem to have about 15 bugs raised about too many threads. We need to prioritize these - which are hitting the performance the most?"
• me "well ... I'm not entirely sure to be honest. It's difficult to know without removing them first and comparing the new performance."
• him "yes, but how are these extra threads impacting current performance, you've analysed their impact?"
• me "yes, it's all part of that analysis. Generally, they are idle and don't really have a CPU cost except for a little extra context switching. They also take up some memory."
• him "ah, so how does that affect the apps?"
• me "well, there will be extra load on the OS, and some threads might not get scheduled as quickly as they should be"
• him "right, so which ones are the most important to get rid of those delays? How much are we getting impacted by that?"
• me "well, I can't really tell. I can see the thread wait's if I analyse the lock frequencies, but none of the SLAs are being broken, so it's more a case of reducing known inefficiencies than eliminating a specific problem." -- I sounded sheepish even to me.
• him "So, if we just leave these threads, we don't really have a problem with the CPU. Anything else?"
• me "Well they each take memory resources too. Both from the Java heap and the native memory"
• him "enough to make these bugs high priority?"
• me, sheepishly "no".

So that was the upshot of quite a bit of work. We have too many threads. Lots of them are useless. And it doesn't really matter - at the moment anyway. Nevertheless I agree with Heinz, it's good to know what the threads are doing, and I expect this analysis will come in useful in the future when there are problems - and I'll be able to do the same analysis much more quickly.

Javva The Hutt.

Back to newsletter 081 contents