Code cache flushing causes stop in compilation and high CPU

FULL PRODUCT VERSION :


FULL OS VERSION :
Linux server 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon Jun 27 19:49:27 BST 2011 x86_64 x86_64 x86_64 GNU/Linux


A DESCRIPTION OF THE PROBLEM :
When migrating from JDK 1.6 (1.6.0_30) to JDK 1.7 (1.7.0_65) we experienced high CPU (some time) and slowness/responsiveness in our application.

Setting in JDK 6 was:

-XX:+UseCodeCacheFlushing (this became a default in jdk 1.7)

In both cases I enabled "-server" flag

The issue is not experienced in jdk1.8.

I did some analysis using JConsole and I have observed different behavior of Code Cache flushing between JDK 1.6 and JDK 1.7. The Code Cache flushing seems not to work properly in Java 1.7 (while in Java 1.6 it works fine)

I stared profiling my application (using JConsole: Memory->Code Cache), monitoring the Code Cache (which default size is 48MB, -XX:ReservedCodeCacheSize=48M) and I have discovered that my application uses the entire Code Cache heap.

 In JDK 1.7: after Code Cache fills, Code Cache flushing starts, marking older methods to be potentially eligible for flushing in favor of newer methods. Actually it seems that after the Code Cache flushing, JIT compilation stops at all causing degradation in performance. Neither old or new code seems to be compiled and it flushes the cache till it reaches very low level, around 3M. Some times compilation starts again, sometimes it doesn't.

Below you can find a test executed with jdk 1.6 and jdk 1.7:


Note that in both scenarios the same jvm options have been used:
   -XX:ReservedCodeCacheSize=48M (default)
   -XX:+UseCodeCacheFlushing (default only in jdk 1.7)


This bug seems to be fixed in JDK 1.8. I have found a similar bug fix (8006952). I am not sure it is the same bug, as it is written that it affects only Java 8.

http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8006952

Thanks,
Sandro Sansone

THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: Did not try

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1) Set a limit size of Code Cache to make sure you application load more than allocated, for example
-XX:ReservedCodeCacheSize=30M

2) start your application

3) as soon as code cache get full, it start flushing and high CPU is experienced. Sometimes low CPU and slowness are experienced.

EXPECTED VERSUS ACTUAL BEHAVIOR :
I would expect a similar result happening in JDK 6 or JDK 8:

- the code cache flushes some code and compile a new one instead of going down only without compiling new code.
REPRODUCIBILITY :
This bug can be reproduced always.

CUSTOMER SUBMITTED WORKAROUND :

Temporary solution:

A workaround for this problem I found was increasing increasing the Code Cache size and be sure the Code Cache flushing does not happen . In my application the Code Cache size average seems to be 55MB, so I solved by setting it to 80MB.

I have used the following JVM option:
   -XX:ReservedCodeCacheSize=80M