Leden 17, 2019

When you flush the CodeCache

In the previous blog post, I wrote about our problem with JVM CodeCache. This cache is used for saving compiled machine code and if your Java application generates new code on the fly, it will eventually fill.

In our case, it filled pretty quickly with code, which was very soon obsolete. The profiling showed that when the cache filled up, it didn't flush. This shut down compilers which resulted in performance degradation.

After some time, I came back to the problem and solved it. I can't remember now if I tried at that time the most obvious thing - turn on the flushing. I probably did it but because some cache cleanups were visible before it filled, it's possible I convinced myself the flushing is happening and I didn't turn it on. Anyway, let me describe you all steps which, at the end of the day, helped us.

As I wrote before, we are using Janino compiler to compile many generated classes. Janino was defined as a singleton in our application. My idea was, that machine code is maybe stuck in CodeCache because compiled classes are never freed by a garbage collector. I still don't know if the idea is correct but it's still a memory leak no matter what. Compiled classes are stored in ClassLoader inside compiler instance and this instance lives through the whole application lifetime. It's maybe not a big issue if you have a lot of memory but I still create the instance of a compiler for each compilation as the performance hit isn't big in our case.

And of course, the flushing has to be on manually (-XX:+UseCodeCacheFlushing) because it's not active by default. After this, I finally saw the "saw" in a plot. Interesting thing is, that after the second peak, I still observed the error in the log that compilers were turned off. Even though the cache was flushed. It's possible they were started later and this wasn't logged but because it happened only once and  I don't want to invest more time to this problem, I'm happy with what we have now.

Our team

Leden 17, 2019

When you fill the CodeCache

When you fill the CodeCache

As a Java developer, you don't need to know internal details of JVM most of the times. The virtual machine is not a trivial piece of technology and learning internal details can be scary. But there are times, when you find something interesting in the logs, which seems to be important. Like the message:

CodeCache is full. The compiler has been disabled. Try increasing the code cache size using -XX:ReservedCodeCacheSize=

The compiler has been disabled? Yikes. That sounds bad, but the application is still running as expected. Fortunately, this message likely won't jump at you as it is related to specific application usage which isn't very usual in my opinion. But what CodeCache is anyway and how can be filled?

JVM uses several types of caches and CodeCache is one of them. As your application is being executed, your code (to be precise bytecode) is interpreted and some parts are compiled by JIT (just-in-time) compiler to native code, which can be directly executed by CPU. The code interpretation is a relatively slow process but if this section of the code isn't invoked frequently, it is fine. The compilation to native code takes additional time (and even more time with optimizing), but if the code is executed frequently, the speed gain is worth it. And how does the CodeCache fits into the equation? Well, the compiled native code is saved to the CodeCache.

Your application has usually a fixed size. The CodeCache is growing after the application was started but at some point, every critical section of code is already compiled so the growth stops. The default size of the cache should be enough for most applications but if not, it can be increased. Not to mention the cache should flush compiled section of code when it's full to create some new empty space. And according to our profiling, this flushing really happens. So how can you fill it to the point of stopping the compiler?

In one of our projects, we use Janino compiler to compile many generated classes and even though they are used only for a brief moment to generate data, they are still compiled to native code and saved to cache. But for some reason, the code of generated classes was never flushed. This is indeed a weird behavior because these problems with CodeCache were related to JDK 7 and addressed in JDK 8.

To be more precise, the flushing was observable just before the cache was filled. Once it was, it stayed that way. It's easily possible the sweeper was invoked but no code was flushed. It's hard to tell why is the code stuck because the sweeping process isn't transparent to the user and the decision if the method should be flushed is calculated at runtime using several parameters.

There are several solutions how to fight with full CodeCache but not all are possible:

  • Assign more memory to CodeCache - this is, however, temporary solution as it will still fill in our case
  • Minimize the number of generated classes and code per class - another temporary solution
  • Separate the class compilation to a different process which will be killed when done - we can do this because our classes are used only briefly to generate data
  • Mark generated classes for interpretation only - can be done for static classes but probably not at runtime
  • Disable JIT during the class compilation - can be set when the application is started, can be probably done using interface Compiler.disable() at runtime but not sure if it's supported in all JVMs
  • Investigate why are classes stuck and solve the source of the problem


For now, we aren't sure which way to take. It will probably involve more cache profiling so it's possible we will return to this topic in some of our future blogs.

Our team