Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8046246

the constantPoolCacheOopDesc::adjust_method_entries() used in RedefineClasses does not scale

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: P4
    • Resolution: Fixed
    • Affects Version/s: 7, 8u60, 9
    • Fix Version/s: 9
    • Component/s: hotspot
    • Labels:
    • Subcomponent:
    • Resolved In Build:
      b55

      Backports

        Description

        The issue was reported by the HP.

        Please, see the email from the support engineer Kevin Brown:

        Hi Serguei,

        I have an issue from our source licensee HP, and while it's not reproducible on Solaris/Linux I'm trying to advise HP on how to further debug it on HPUX/PaRISC.

        Vladimir indicated the method in question "adjust_cpool_cache_and_vtable()" was a JVMTI method and I hear you are an expert in this area. Could you review the data below and let me know if you have any advise?

        Thanks,
        Kevin

        -------- Original Message --------
        Subject: Re: HP's high CPU utilization investigation...
        Date: Thu, 08 May 2014 17:52:59 -0700
        From: Vladimir Kozlov <vladimir.kozlov@oracle.com>
        To: Kevin Brown <Kevin.L.Brown@oracle.com>


        Hi Kevin,

        The bug I mentioned to you:

        https://bugs.openjdk.java.net/browse/JDK-8041984

        but it looks like it is not related to your problem.

        > Question: By design, is re-compilation(i.e. compilation while reloading
        > instrumented classes) so very CPU intensive (~100%) Or is it expected
        > for large no. of classes ?

        When class is redefined we throw away (deoptimize) all related compiled
        methods. If new methods are hot they will be recompiled again after
        executed in Interpreter.

        The method adjust_cpool_cache_and_vtable() is not JIT compiler related.
        This method is in jvmti code. Ask serviceability group about it.

        That code was significantly changed since jdk7.

        Regards,
        Vladimir

        On 5/8/14 5:20 PM, Kevin Brown wrote:
        > Hi Vladimir,
        >
        > Thanks for the quick chat and willingness to look at this. I was hoping
        > you might have some insight or could at least better direct me on
        > investigating a high CPU usage issue that HP is reporting. Their
        > customer reported the issue initially with Weblogic on HPUX, and HP's
        > testcase without Weblogic (attached) successfully reproduces the problem.
        >
        > *But* it is only reproduciable on HPUX. When I run the test on Solaris
        > or Linux, it does not show the same high CPU usage. (never gets above
        > 12-13%) I cannot formally file a bug since we don't see it our our
        > platforms, but HP is hoping we can provide guidance.
        >
        > Do you have any suggestions on further debugging, VM options to tune
        > with, or areas to look into?
        >
        > Here is my full analysis thus far...
        >
        > Problem statement:
        > 100% CPU utilization is seen when the number of classes need to be
        > compiled are of the order of a thousand or more. Below are the numbers
        > that could be of interest to you when tested on HPUX.
        >
        > o High CPU time: 8-10 mins
        > o No. of loaded classes : ~1600
        > o No. of redefined classes: 443
        > o Calls to evaluate_operation(): 658
        >
        > What HP did on HP-UX:
        > HP could produce the above numbers through dynamically instrumenting an
        > in-house application that they generally use for testing. (the attached
        > testcase)
        >
        > JLE testing on Solaris and Linux:
        > I (Kevin) was unable to reproduce the results on Solaris or Linux. I
        > tested with JDK 7u51 on Linux x64, connecting JConsole to badapp.jar
        > file. I do not see similar results, after 30 minutes of testing, the
        > CPU usage never goes over 12.9%
        >
        > I suggested HP to try setting -XX:CompileThreshold to a lower setting
        > but they claim this delays the high CPU usage since compilation
        > eventually happens and goes to 100%.
        >
        > JVM Call hierarchy:
        > Here are the functions called from jvm, the
        > adjust_cpool_cache_and_vtable() is the busiest method consuming the
        > highest CPU.
        >
        > VMThread::run()
        > ….
        > VMThread::loop()
        > ...
        > evaluate_operations()
        > ..
        > evaluate()
        > ...
        > doit()
        > ...
        > redefine_single_Class()
        > ....
        > classes_do()
        > ....
        > adjust_cpool_cache_and_vtable()
        > ….
        >
        > Question: By design, is re-compilation(i.e. compilation while reloading
        > instrumented classes) so very CPU intensive (~100%) Or is it expected
        > for large no. of classes ?
        >
        > Thanks,
        > Kevin

        As we can see the HP reported a performance problem in the VM_RedefineClasses::adjust_cpool_cache_and_vtable().
        Also, the performance profiling information (see attached file HP-high_cpu_caliper-report.zip received from HP)
        shows the most overhead is in the constantPoolCacheOopDesc::adjust_method_entries().

        Look at the following fragment:
        Function Summary
        ---------------------------------------------------------------------------------------------------------
        % Total Cumulat Secs Msecs
           IP % of IP in Call per
        Samples Total Samples Func Count Call Function File
        ---------------------------------------------------------------------------------------------------------
          19.52 19.52 65739 20.59 62723 0.33 libjvm.so::VM_RedefineClasses::adjust_cpool_cache_and_vtable(klassOopDesc*,oopDesc*,Thread*) jvmtiRedefineClasses.cpp
          15.21 34.73 51230 16.05 27027 0.59 libjvm.so::RelocIterator::initialize(nmethod*,unsigned char*,unsigned char*) relocInfo.cpp
          14.52 49.25 48907 15.32 44635 0.34 libjvm.so::constantPoolCacheOopDesc::adjust_method_entries(methodOopDesc**,methodOopDesc**,int,bool*) cpCacheOop.cpp

          Attachments

            Issue Links

              Activity

                People

                Assignee:
                sspitsyn Serguei Spitsyn
                Reporter:
                sspitsyn Serguei Spitsyn
                Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                  Dates

                  Created:
                  Updated:
                  Resolved: