Java Logger Memory Leaks

· CodePerformancememory leakbuglogger

Recently we switched from Log4J to the java.util.logger package (for this entry it will be called the "Java Logger"). Why, you might wonder, and I don't have a good reason to give other than the illusion the Java Logger package gives as being more standard. I would like to stress the word illusion for a minute here. Since the title ruins any chance for a surprise ending, I will be brief, but give a few points about the tools I used. The starting point was that the entire back-end system slowed down significantly; no exceptions found in the logs, nothing stopped working, everything just got 20-40 times slower. So, what can be done?
  • No profiling While profilers are great, for memory leaks occurring in a production environment which cannot sustain the effect of performance drop given by on-line profilers, they're quite useless. Also, with the abundance of information these profilers might give it could take a long time before finding any clues.
  • GC influence Since it's a "heavy" back-end system, the GC could affect its performance if code was written in an non-optimal way. I decided to take a look at how hard it was working, using jstat -gccause. It showed that the GC was working constantly, and that the old generation area was full so nothing could be created. I am still wondering why no OutOfMemoryException was thrown; it might have something to do with the origin of the bug, in the logger device.
  • Memory dump I started with dumping the memory, and hoping to excavate which objects filled the memory up. It was useless; the application is working with a couple of GB of RAM; saving the dump was easy, but opening it with a profiler on a stand-alone system was almost impossible. Eventually I installed a Linux 64-bit machine and opened it there, but reached nowhere, again due to the abundance of information.
  • Statistical data Seeing that the dump was useless by itself, I decided to take a look at the memory objects using jmap -histo:live command. It showed me what objects have the most instances and which take the most size in memory. It didn't provide a lot, but it gave a starting point: the Logger object had over 600,000 instances!
I tried to look through all the usages of the logger, but it was impossible. While we have an abstraction layer of our own for logs (for our own reasons, nothing that the commons-logging library could do), going through all the invocations would have been impossible. I used the fact that we had the abstraction layer to write the unique names of the loggers requested to a file, and apparently a user decided to have the timestamp (System.currentTimeMillis) added to one of the logger name requests. Which leads to two conclusions:
  1. Never use dynamic names for loggers! Simple enough, but apparently wasn't too clear for the users of our log library.
  2. The request for that dynamic logger was an old request. The bug apparently started only when we switched to the Java Logger, which meant there's a difference in the way Log4J and the Java Logger work. It seems that the Java Logger keeps a lot of information as cached, and that it was at least known regarding the Level class.
I figured that figuring the memory leak's cause through a production system's statistical data would be interesting to at least some, hence the post; hope it was. Please let me know through comments!

Comments (13)

Aviad
@Markus: I'll give it a look! Sorry for taking so long to answer... Been having problems with the blog recently..
Markus Kohler
@Aviad Yes you can try to figure out the class with class statistics dumps, but then you have to do additional steps anyway, unless you are very lucky. So why not just take a heap dump put it in to MAT and just wait for the analysis to show you the root cause? http://dev.eclipse.org/blogs/memoryanalyzer/2008/05/27/automated-heap-dump-analysis-finding-memory-leaks-with-one-click/
Aviad
@William: I forgot to give an answer to something that you've asked. I agree that there was an abuse of the logging system; I even stated it, and maybe the title of the post is confusing a bit because it puts all the blame on the Java Logger. However, I do believe that the Java Logger could have gracefully avoided this becoming an issue, much like Log4J did (and it did, since we didn't find this problem until we made the switch). I could also look at the bright side and say that things turned to the better since we found something that probably disturbed the system for a long time without noticing it (because of the creation and deletion of many objects in Log4J).. :)
Aviad
@Markus: Eventually, JProfiler showed it to me. All of the objects were cached by the LogManager. Again, next time I might try MAT first, but still I think that minimizing the search criteria before dealing with a memory dump is better still. @William: The user of the logging library is the developer group, of course. This kind of bug, compared to others, is something that could be dealt with using simple maintenance (i.e. restart the server every X days). It was more important to go to production than to deal with this bug (I'm sure you can find an equivalent of the situation in your work-place, too). Regarding caring about logging: we log a lot, however we turn most of it off by default and only turn it on when needed. I think that using logging in your code has a linear relationship to the maturity of code, if you ask me, since it allows you to examine theories for production-time bugs without needing to restart, sending the application into a profiling session, etc, but by just getting some information dumped to a file at runtime. My opinion, obviously. You can have yours. :)
William Louth
This does beg the question what kind of performance testing do you do prior to production? Such low hanging fruit should never have made it into such an environment (is it really production?) unless your reference to a "user" in the blog entry does actually mean a "user" and not a developer. Using a timestamp for a logger name is a horrendous idea and should NOT reflect poorly on java.util.logging.* package but more so on the "user" able to make such calls directly or indirectly programmatically or via configuration. Markus is right any basic heap analyzer or runtime diagnostics tool could find this is a matter of seconds. The problem you have described is not a memory leakage at least not within the library itself more like an abuse. Would you not agree? I don't really care too much for logging as there is an inverse relationship between the amount of logging and the quality of the code executing.
Markus Kohler
I can almost guarantee that the Eclipse Memory Analyzer will show you why those 60000 objects are still alive. As Kirk pointed out you might "only" have a problem with softreferences. There are some special commands in MAT for analyzing them as well.
Aviad
@DY: The production system can't work with a thread dump every couple of seconds... Also, a jstack command should show the threads at any given time, so it needn't be within the application's code. Regarding YourKit, I tried that and tried JProfiler - both didn't yield good results. I don't know why - it just went out of memory after working for 30 minutes trying to open the file (both with the newest versions). @Les Stroud: You could also dump only the live objects, if you can't deal with a full heap dump. The dump option has a flag live for exactly that. Does it help?
Les Stroud
I've experienced something similar recently. Essentially, the instance of tomcat would get near the max heap and start garbage collecting. It was efficient enough to find a balance where it could collect about as fast as the memory leak could fill space. As a result, it spills over into a cpu constraint. Unfortunately, we do not have a cluster (that was my first suggestion). I couldn't do a full heap dump. The heap dump itself consumed too many resources on the server leading to more contention for the other running processes. I was able to get a histo, though. The histo led me to the general areas where the memory leak was occurring. After that, I have been banging the heck out of the apps in test specifically around areas that exercise the leaking objects. I have not yet been able to completely eliminate the condition, but I have impacted it's frequency. So, I'll be interested to hear if you come up with some other ways to analyze your issues.
DY
I have tried yourkit with big heap dumps and it works well for even a 2 GB dump on on my 32-bit laptop. My company has a java based intranet product which can consume a lot of memory at times and give OOM, so by default we set +HeapDumpOnOutOfMemoryError option in the tomcat service. It helps in quickly troubleshooting customer's system and suggest solutions like turning off some bells or moving to 64-bit. Another tool we ship by default is a way to get thread-dump. 4-5 thread-dumps taken 10 seconds apart are the quickest way to figure out where the application is spending a lot of time or what is causing the slowdown http://googleismygod.blogspot.com/2008/10/jvm-thread-dump-simple-yet-effective.html
Kirk
If you can't setup a test environment, you'll need to find this in production. The best way to profile memory in production requires that you cluster your application. Take one node (or add an extra node) and trickle traffic at it. IOWs, try to make up for the hit in performance by reducing load. Use a sampling profiler (one that doesn't try to capture every object creation event) such as NetBeans profiler. YourKit will also serve you well in this case (commercial). Franks (excellent) blog is about perm space leaks. If you've not got a custom level class you're not going to see an OOME due to classes not being unloaded. However, I don't think you have a memory leak. The biggest reason is, you're not seeing an OOME being thrown. If you had a memory leak you'd see an OOME as applications do *not* recover from leaks. Trigger for an oome is 98% of time in GC with less than 2% recovered. Your app is recovering just enough to keep going. It's struggling but.... I'm going to take an off the wall guess here and suggest you look at soft, weak, or phantom (finalize) reference processing. If I were to bet money, it would be on soft reference processing. If you turn on verbose gc with PrintGCDetails you should see reference processing stats.
Daniel Schneller
If you need to dig through Gigs of heap dumps in the future, have a look at the Eclipse Memory Analyzer. It was first developed by SAP to debug their app servers. It handles very large dumps much more gracefully than any profiler I have seen, as it first creates sort of a database/index system without needing to load the whole dump at once. Later it acts very quickly using this performance-optimized data structure. If I remember correctly, you should have no problems diagnosing a 64bit heap dump larger than 2GB even on a 32bit system.
Aviad
@Kirk: I agree with what you say. We do have a clustered environment, which helped test the system: the other nodes of the production environment took the strain while I was running all sorts of statistical tests on the damaged system. Regarding the OOME, you might be right. I did try the PrintGCDetails and it didn't show anything that could lead me to conclude a soft reference was the issue. However, looking at the Logger.getLogger method's implementation led to the LogManager.demandLogger code, which seems to use weak references to keep the loggers. I couldn't see in the code anything heavy being called if the reference evaporated, though. @Daniel: Thanks! I'll try the Eclipse tool - I still have the dumps on disk. I'll let you know how it worked...
sandeep
Nice article Did you get a chance to try the tool from appdynamics, it is super cool and easy to use, gives all info for memory leaks http://appdynamics.com/documents/white_papers/WP_MEMORY_LEAKS.pdf