Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-6525425

(ref) heavy lock contention during object serialization on Solaris10/T2000

    XMLWordPrintable

    Details

    • Subcomponent:
    • Resolved In Build:
      b01
    • CPU:
      x86, sparc
    • OS:
      linux, solaris_10

      Backports

        Description

        We have a customer who is looking to deploy an application on several
        hundred T2000 machines. Unfortunately, during load testing we found the
        performance of the WebLogic SIP Server to be pathetic on that hardware.
        We traced the issue down to a bottleneck that occurs in the JDK 1.5.0_6
        through _11. A summary of the issue is below. The bottleneck was not an
        issue on x86 hardware or other non-Niagra based Sparc architectures.

        The issue we have run into is heavy contention on
        java.lang.ref.ReferenceQueue.lock during object serialization on Solaris
        10 SunOS 5.10 Generic_118833-17 sun4v sparc SUNW,Sun-Fire-T200, Java(TM)
        2 Runtime Environment, Standard Edition (build 1.5.0_06-b05) Java
        HotSpot(TM) Server VM (build 1.5.0_06-b05, mixed mode) and Sun T2000.
        The following stack trace illustrates the contention; in our tests we
        observe many threads waiting on the same semaphore :
         
        "pool-1-thread-23" prio=10 tid=0x00282d18 nid=0x41 waiting for monitor
        entry [0xb07fe000..0xb07ffaf0]
                at java.lang.ref.ReferenceQueue.poll(ReferenceQueue.java:82)
                - waiting to lock <0xe7569aa0> (a
        java.lang.ref.ReferenceQueue$Lock)
                at
        java.io.ObjectStreamClass.processQueue(ObjectStreamClass.java:2206)
                at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:253)
                at
        java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1035)
                at
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)
         
        "pool-1-thread-29" prio=10 tid=0x002895a0 nid=0x47 waiting for monitor
        entry [0xb01fe000..0xb01ffbf0]
                at java.lang.ref.ReferenceQueue.poll(ReferenceQueue.java:82)
                - waiting to lock <0xe7569aa0> (a
        java.lang.ref.ReferenceQueue$Lock)
                at
        java.io.ObjectStreamClass.processQueue(ObjectStreamClass.java:2206)
                at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:253)
                at
        java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1035)
                at
        java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:13
        75)
                at
        java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1347)
                at
        java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1
        290)
                at
        java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1079)
                at
        java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:302)

        Our analysis of the root cause revealed that the ObjectStreamClass
        static Cache data members are the bottleneck.
         
          To help with troubleshooting the problem, we have created a reproducer
        (attached) and a patch to ReferenceQueue.java (attached). While we are
        not proposing the patch as a final solution, it helps illustrate the
        problem. The reproducer requires WLS 9.2 to compile and run. The results
        without the patch are:

        bash-3.00$ ~/bea.sparc.30/jdk1.5.0_11/bin/java -server Serialize 32
        120000 1000
        Warming up with : [32] threads, for : [30000] ms, [1000] requests at a
        time
        Running with : [32] threads, for : [120000] ms, [1000] requests at a
        time
        Throghuput : [6.0] (ser-deSer / ms)

        With the patch we can fully utilize the CPU and we see over 3x
        improvement in throughput :

        bash-3.00$ ~/bea.sparc.30/jdk1.5.0_11/bin/java
        -Xbootclasspath/p:/home/jwilkiew/temp/relock.jar -server Serialize 32
        120000 1000
        Warming up with : [32] threads, for : [30000] ms, [1000] requests at a
        time
        Running with : [32] threads, for : [120000] ms, [1000] requests at a
        time
        Throghuput : [21.0] (ser-deSer / ms)
        workaround: To help with troubleshooting the problem, we have created a
        reproducer (attached) and a patch to ReferenceQueue.java (attached).
        While we are not proposing the patch as a final solution, it helps
        illustrate the problem.

          Attachments

            Issue Links

              Activity

                People

                Assignee:
                vchoudhaorcl Vaibhav Choudhary (Inactive)
                Reporter:
                mmma Marvin Ma (Inactive)
                Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                  Dates

                  Created:
                  Updated:
                  Resolved:
                  Imported:
                  Indexed: