Java Logger Memory Leaks
Recently we switched from Log4J to the java.util.logger package (for this entry it will be called the "Java Logger"). Why, you might wonder, and I don't have a good reason to give other than the illusion the Java Logger package gives as being more standard. I would like to stress the word illusion for a minute here.
Since the title ruins any chance for a surprise ending, I will be brief, but give a few points about the tools I used. The starting point was that the entire back-end system slowed down significantly; no exceptions found in the logs, nothing stopped working, everything just got 20-40 times slower. So, what can be done?
- No profiling While profilers are great, for memory leaks occurring in a production environment which cannot sustain the effect of performance drop given by on-line profilers, they're quite useless. Also, with the abundance of information these profilers might give it could take a long time before finding any clues.
- GC influence Since it's a "heavy" back-end system, the GC could affect its performance if code was written in an non-optimal way. I decided to take a look at how hard it was working, using
jstat -gccause. It showed that the GC was working constantly, and that the old generation area was full so nothing could be created. I am still wondering why no OutOfMemoryException was thrown; it might have something to do with the origin of the bug, in the logger device. - Memory dump I started with dumping the memory, and hoping to excavate which objects filled the memory up. It was useless; the application is working with a couple of GB of RAM; saving the dump was easy, but opening it with a profiler on a stand-alone system was almost impossible. Eventually I installed a Linux 64-bit machine and opened it there, but reached nowhere, again due to the abundance of information.
- Statistical data Seeing that the dump was useless by itself, I decided to take a look at the memory objects using
jmap -histo:livecommand. It showed me what objects have the most instances and which take the most size in memory. It didn't provide a lot, but it gave a starting point: the Logger object had over 600,000 instances!
System.currentTimeMillis) added to one of the logger name requests. Which leads to two conclusions:
- Never use dynamic names for loggers! Simple enough, but apparently wasn't too clear for the users of our log library.
- The request for that dynamic logger was an old request. The bug apparently started only when we switched to the Java Logger, which meant there's a difference in the way Log4J and the Java Logger work. It seems that the Java Logger keeps a lot of information as cached, and that it was at least known regarding the
Levelclass.
Comments (13)
jstackcommand should show the threads at any given time, so it needn't be within the application's code. Regarding YourKit, I tried that and tried JProfiler - both didn't yield good results. I don't know why - it just went out of memory after working for 30 minutes trying to open the file (both with the newest versions). @Les Stroud: You could also dump only the live objects, if you can't deal with a full heap dump. The dump option has a flaglivefor exactly that. Does it help?