Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8013809

deadlock in SSLSocketImpl between between write and close

    Details

    • Subcomponent:
    • Resolved In Build:
      b103
    • Verification:
      Not verified

      Backports

        Description

        FULL PRODUCT VERSION :
        java version " 1.6.0_38 "
        Java(TM) SE Runtime Environment (build 1.6.0_38-b05)
        Java HotSpot(TM) 64-Bit Server VM (build 20.13-b02, mixed mode)

        ADDITIONAL OS VERSION INFORMATION :
        Linux magnifico.aimsv.com 2.6.18-308.24.1.el5xen #1 SMP Tue Dec 4 18:33:26 EST 2012 x86_64 x86_64 x86_64 GNU/Linux


        EXTRA RELEVANT SYSTEM CONFIGURATION :
        Our application runs in JBOSS-6.1 server and one of its components uses JBOSS MQ (i know that MQ is deprecated but this component is legacy that we need to carry on for some more time).

        Several messages get transferred between various components of the system via JMS (asynchronous communication), perhaps tens of messages per second at a peak load.







        A DESCRIPTION OF THE PROBLEM :
        We have been running this application for several years without any known issues. We have regularly upgrading our JDK/JRE version at regular intervals. We tested our application with JRE-1.6.33 and did not face this deadlock issue.

        However, after upgrading to JRE-1.6.38 and running our application for about a week (roughly 5-7 days), we are certainly seeing deadlocks appearing in JDK core classes like SSLScoketImpl.java.

        On searching on the internet for this particular " deadlock in SSLSocketImpl.readDataRecord() or SSLSocketImpl.writeRecord() " , we found several references to this exact same issue in various open source products, and they claim that this issue is in core JRE and that this issue is still present in JRE-1.7 as well, and is not yet fixed in any release of JRE.

        Here is one reference to the above claim:
        https://jira.atlassian.com/browse/BAM-12699

        This thread from Oracle Forums claims that this issue is still present in JRE-1.7.x
        https://forums.oracle.com/forums/thread.jspa?messageID=10748121

        We found that this issue seems to have been reported on JRE-1.4 and/or JRE-1.5 and was fixed (http://bugs.sun.com/view_bug.do?bug_id=6569529)
        but looks like it reappeared in JDK-1.6.38+ or JDK-1.7.x

        If we continue to run the test even after seeing the deadlocks, the number of deadlocks are increasing at a rate of 5 per week


        Here is the extract from thread-dump
        ----------------------------------------------------------------------
         " UIL2.SocketManager.WriteTask#204296 client=10.64.2.123:8193 " :
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.getConnectionState(SSLSocketImpl.java:601)
                - waiting to lock <0x000000076ab4ca18> (a com.sun.net.ssl.internal.ssl.SSLSocketImpl)
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.isClosed(SSLSocketImpl.java:1328)
                at java.net.Socket.getTcpNoDelay(Socket.java:877)
                at com.sun.net.ssl.internal.ssl.BaseSSLSocketImpl.getTcpNoDelay(BaseSSLSocketImpl.java:325)
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:770)
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:753)
                at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:100)
                - locked <0x000000076ab43a18> (a com.sun.net.ssl.internal.ssl.AppOutputStream)
                at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
                at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
                - locked <0x000000076ab5afb0> (a org.jboss.util.stream.NotifyingBufferedOutputStream)
                at java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputStream.java:1793)
                at java.io.ObjectOutputStream.flush(ObjectOutputStream.java:699)
                at org.jboss.mq.il.uil2.SocketManager$WriteTask.run(SocketManager.java:558)
                at java.lang.Thread.run(Thread.java:662)
         " UIL2.SocketManager.ReadTask#204295 client=10.64.2.123:8193 " :
                at sun.misc.Unsafe.park(Native Method)
                - parking to wait for <0x000000076ab40898> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
                at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
                at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
                at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
                at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:751)
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:624)
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.sendAlert(SSLSocketImpl.java:1875)
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.warning(SSLSocketImpl.java:1713)
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.closeInternal(SSLSocketImpl.java:1481)
                - locked <0x000000076ab4ca18> (a com.sun.net.ssl.internal.ssl.SSLSocketImpl)
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1819)
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1019)
                - locked <0x000000076ab4ca18> (a com.sun.net.ssl.internal.ssl.SSLSocketImpl)
                - locked <0x000000076ab41738> (a java.lang.Object)
                at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
                at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
        - locked <0x000000076ab43988> (a com.sun.net.ssl.internal.ssl.AppInputStream)
                at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
                at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
                - locked <0x000000076ab55098> (a org.jboss.util.stream.NotifyingBufferedInputStream)
                at org.jboss.util.stream.NotifyingBufferedInputStream.read(NotifyingBufferedInputStream.java:79)
                at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2248)
                at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2428)
                at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2498)
                at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2570)
                at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2719)
                at java.io.ObjectInputStream.readByte(ObjectInputStream.java:894)
                at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:340)
                at java.lang.Thread.run(Thread.java:662)


        REGRESSION. Last worked in version 6u31

        STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
        Heavy JMS load with Publisher and Receiver both on the same host, running for a one or two weeks.

        EXPECTED VERSUS ACTUAL BEHAVIOR :
        EXPECTED -
        NO Deadlocks in the Thread dump taken by JSTACK
        ACTUAL -
        these deadlocks continue to increase over the time and is slowing down the system below acceptable levels.

        REPRODUCIBILITY :
        This bug can be reproduced always.

          Issue Links

            Activity

            Hide
            xuelei Xue-Lei Fan added a comment -
            The problem is that the getTcpNoDelay() (in write lock) calls into isClosed() (in "this" lock) in SSLSocketImpl.
            Show
            xuelei Xue-Lei Fan added a comment - The problem is that the getTcpNoDelay() (in write lock) calls into isClosed() (in "this" lock) in SSLSocketImpl.
            Hide
            xuelei Xue-Lei Fan added a comment -
            No new regression test, simple fix and hard to reproduce the issue. Customer (sustaining team) had confirmed that the fix works.
            Show
            xuelei Xue-Lei Fan added a comment - No new regression test, simple fix and hard to reproduce the issue. Customer (sustaining team) had confirmed that the fix works.
            Hide
            hgupdate HG Updates added a comment -
            URL: http://hg.openjdk.java.net/jdk8/tl/jdk/rev/8c7cf4926157
            User: xuelei
            Date: 2013-08-07 13:43:25 +0000
            Show
            hgupdate HG Updates added a comment - URL: http://hg.openjdk.java.net/jdk8/tl/jdk/rev/8c7cf4926157 User: xuelei Date: 2013-08-07 13:43:25 +0000
            Hide
            hgupdate HG Updates added a comment -
            URL: http://hg.openjdk.java.net/jdk8/jdk8/jdk/rev/8c7cf4926157
            User: lana
            Date: 2013-08-13 18:14:45 +0000
            Show
            hgupdate HG Updates added a comment - URL: http://hg.openjdk.java.net/jdk8/jdk8/jdk/rev/8c7cf4926157 User: lana Date: 2013-08-13 18:14:45 +0000

              People

              • Assignee:
                xuelei Xue-Lei Fan
                Reporter:
                shadowbug Shadow Bug
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: