Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8258575

JVM crash with C 0x0000000000000002 j bea.jolt.NwHdlr.open_socket()

    Details

    • Subcomponent:
    • CPU:
      x86_64
    • OS:
      linux_suse_slec

      Description

                                      
      ADDITIONAL SYSTEM INFORMATION :
      SUSE linux 11 sp4 x86_64 jdk 8.0_111-b14

      A DESCRIPTION OF THE PROBLEM :
      My application has been running for mor than 7 year,Recently it crash twice.most recently is 2020-12-11,and the earlier one happens in 2017-12-22,i think it maybe the same bug.

      ERROR MESSAGES/STACK TRACES THAT OCCUR :
      *****************************************crash log in 2020-12-11*****************************************
      #
      # A fatal error has been detected by the Java Runtime Environment:
      #
      # SIGSEGV (0xb) at pc=0x0000000000000002, pid=9503, tid=0x00007f3518747700
      #
      # JRE version: Java(TM) SE Runtime Environment (8.0_111-b14) (build 1.8.0_111-b14)
      # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.111-b14 mixed mode linux-amd64 compressed oops)
      # Problematic frame:
      # C 0x0000000000000002
      #
      # Core dump written. Default location: /esb/app/SmartESB/bin/core or core.9503 (max size 25000 kB). To ensure a full core dump, try "ulimit -c unlimited" before starting Java again
      #
      # If you would like to submit a bug report, please visit:
      # http://bugreport.java.com/bugreport/crash.jsp
      #

      --------------- T H R E A D ---------------

      Current thread (0x00007f358d317000): JavaThread "Thread-316" [_thread_in_Java, id=12533, stack(0x00007f3518647000,0x00007f3518748000)]

      siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000002

      Registers:
      RAX=0x0000000749fb5cc0, RBX=0x00007f35187457e0, RCX=0x00007f35fa6af5e1, RDX=0x00007f35fa6af6b4
      RSP=0x00007f3518745800, RBP=0x00007f3518745e80, RSI=0x00000003e875f958, RDI=0x00007f351810cf90
      R8 =0x00000003e87606e8, R9 =0x00007f35f9de7000, R10=0x0000000000000009, R11=0x0000000000000001
      R12=0x0000000000000000, R13=0x000000000000000c, R14=0x00007f3518745850, R15=0x00007f358d317000
      RIP=0x0000000000000002, EFLAGS=0x0000000000010206, CSGSFS=0x0000000000000033, ERR=0x0000000000000014
        TRAPNO=0x000000000000000e

      Top of Stack: (sp=0x00007f3518745800)
      0x00007f3518745800: 00007f3518745850 00007f358d317000
      0x00007f3518745810: 00007f3518745860 00007f35fa1a0f3f
      0x00007f3518745820: 00000006d45cd0e0 0000000000000000
      0x00007f3518745830: 0000000000000000 0000000000000000
      0x00007f3518745840: 00000003c005f7b0 00007f35fa3e4ac0
      0x00007f3518745850: 00000003c00004b8 0000000000000000
      0x00007f3518745860: 00007f3518745e80 00007f35fadb7024
      0x00007f3518745870: 0000000000000014 0000000000000000
      0x00007f3518745880: 00000003e875f958 00000006d45cca18
      0x00007f3518745890: 00000000000010cc 00000006d45ccb40
      0x00007f35187458a0: 00000006d45ccc38 00007f3500000007
      0x00007f35187458b0: 0000000749fb5cc0 00007f350000001b
      0x00007f35187458c0: 0000000000000009 00000003e875f958
      0x00007f35187458d0: 0000000000000001 00000006d45ccb40
      0x00007f35187458e0: 00007f3518745968 00007f35f9def2bd
      0x00007f35187458f0: 00007f3518745e80 00007f35fa6b1344
      0x00007f3518745900: 0000000000000004 0000000000000000
      0x00007f3518745910: 00007f3518745e80 00007f35fa99b1d4
      0x00007f3518745920: 00000006d45cce88 00000003c221fc60
      0x00007f3518745930: 00007f35cd874d9c 00007f3518745990
      0x00007f3518745940: 00000006d45ccf80 00000006d45cd0b0
      0x00007f3518745950: 00000003c22204b0 00000003e875f958
      0x00007f3518745960: 00000000000010cc 00000006d45cca18
      0x00007f3518745970: 00000006d45cd0b0 000000000000001b
      0x00007f3518745980: 0000000000000004 00000000000010cc
      0x00007f3518745990: 00007f3518745e80 00007f35fb5e8374
      0x00007f35187459a0: 0000003000000010 00007f3518740130
      0x00007f35187459b0: 00007f3518745c40 00007f3518745968
      0x00007f35187459c0: 00007f35187459a9 0000000800000001
      0x00007f35187459d0: 00007f358ccada00 00007f358ccade98
      0x00007f35187459e0: 00007f35cd82a0a8 00007f358d317000
      0x00007f35187459f0: 00007f350000000e 00007f3518745750

      Instructions: (pc=0x0000000000000002)
      0xffffffffffffffe2:

      Register to memory mapping:

      RAX=0x0000000749fb5cc0 is an oop
      java.net.ConnectException
       - klass: 'java/net/ConnectException'
      RBX=0x00007f35187457e0 is pointing into the stack for thread: 0x00007f358d317000
      RCX=0x00007f35fa6af5e1 is at entry_point+3297 in (nmethod*)0x00007f35fa6ae690
      RDX=0x00007f35fa6af6b4 is at entry_point+3508 in (nmethod*)0x00007f35fa6ae690
      RSP=0x00007f3518745800 is pointing into the stack for thread: 0x00007f358d317000
      RBP=0x00007f3518745e80 is pointing into the stack for thread: 0x00007f358d317000
      RSI=0x00000003e875f958 is an oop
      java.net.SocksSocketImpl
       - klass: 'java/net/SocksSocketImpl'
      RDI=0x00007f351810cf90 is pointing into metadata
      R8 =0x00000003e87606e8 is an oop
      java.lang.Object
       - klass: 'java/lang/Object'
      R9 =0x00007f35f9de7000 is an unknown value
      R10=0x0000000000000009 is an unknown value
      R11=0x0000000000000001 is an unknown value
      R12=0x0000000000000000 is an unknown value
      R13=0x000000000000000c is an unknown value
      R14=0x00007f3518745850 is pointing into the stack for thread: 0x00007f358d317000
      R15=0x00007f358d317000 is a thread


      Stack: [0x00007f3518647000,0x00007f3518748000], sp=0x00007f3518745800, free space=1018k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      C 0x0000000000000002
      j bea.jolt.NwHdlr.open_socket()V+286
      j bea.jolt.NwHdlr.checkAuth(Z)[Ljava/lang/Object;+27
      j bea.jolt.CMgr.getInfo(IZ)[Ljava/lang/Object;+11
      j bea.jolt.JoltSessionAttributes.getDomainInfo(Z)I+218
      j bea.jolt.JoltSessionAttributes.getIntDef(Ljava/lang/String;I)I+15
      j bea.jolt.pool.Connection.init(Ljava/lang/String;Lbea/jolt/pool/UserInfo;)I+313
      j bea.jolt.pool.SessionPool.estConnect(Lbea/jolt/pool/Connection;)Lbea/jolt/pool/Connection;+191
      j bea.jolt.pool.SessionPool.sessionPoolCreate(Ljava/lang/String;[Ljava/lang/String;[Ljava/lang/String;IILbea/jolt/pool/UserInfo;ZZLbea/jolt/SecurityContext;)V+137
      j bea.jolt.pool.SessionPool.<init>(Ljava/lang/String;[Ljava/lang/String;[Ljava/lang/String;IILbea/jolt/pool/UserInfo;ZZLbea/jolt/SecurityContext;)V+42
      j bea.jolt.pool.SessionPoolManager.newSessionPool(Ljava/lang/String;[Ljava/lang/String;[Ljava/lang/String;IILbea/jolt/pool/UserInfo;ZZLbea/jolt/SecurityContext;)Lbea/jolt/pool/SessionPool;+19
      j bea.jolt.pool.SessionPoolManager.createSessionPool([Ljava/lang/String;[Ljava/lang/String;IILbea/jolt/pool/UserInfo;Ljava/lang/String;ZLbea/jolt/SecurityContext;)I+36
      j bea.jolt.pool.SessionPoolManager.createSessionPool([Ljava/lang/String;[Ljava/lang/String;IILbea/jolt/pool/UserInfo;Ljava/lang/String;)I+16
      j com.xx.xx.xxx.protocol.jolt.client.JOLTLongConsumerProxy$InitPoolThread.run()V+39
      J 5865 C1 java.lang.Thread.run()V (17 bytes) @ 0x00007f35fad8d8c4 [0x00007f35fad8d780+0x144]
      v ~StubRoutines::call_stub
      V [libjvm.so+0x690206] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
      V [libjvm.so+0x690711] JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321
      V [libjvm.so+0x690bb7] JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47
      V [libjvm.so+0x7278f0] thread_entry(JavaThread*, Thread*)+0xa0
      V [libjvm.so+0xa742b3] JavaThread::thread_main_inner()+0x103
      V [libjvm.so+0xa743fc] JavaThread::run()+0x11c
      V [libjvm.so+0x924ea8] java_start(Thread*)+0x108
      C [libpthread.so.0+0x7806] start_thread+0xe6


      and i found some text in core file,i'm not sure it may be a clue

      Exception <a 'java/lang/NullPointerException'> (0x00000007263ab4f8) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u111/7883/hotspot/src/share/vm/runtime/sharedRuntime.cpp, line 1085]
      /systemDictionary.cpp, line 210]
      Exception <a 'java/net/ConnectException'> (0x0000000749fc8180) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u111/7883/hotspot/src/share/vm/prims/jni.cpp, line 709]



      *****************************************crash log in 2017-12-22*****************************************
      #
      # A fatal error has been detected by the Java Runtime Environment:
      #
      # SIGSEGV (0xb) at pc=0x00007ff08bdee860, pid=7308, tid=0x00007fef5689f700
      #
      # JRE version: Java(TM) SE Runtime Environment (8.0_111-b14) (build 1.8.0_111-b14)
      # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.111-b14 mixed mode linux-amd64 compressed oops)
      # Problematic frame:
      # V [libjvm.so+0xfb5860]
      #
      # Core dump written. Default location: /esb/app/SmartESB/bin/core or core.7308 (max size 25000 kB). To ensure a full core dump, try "ulimit -c unlimited" before starting Java again
      #
      # If you would like to submit a bug report, please visit:
      # http://bugreport.java.com/bugreport/crash.jsp
      #

      --------------- T H R E A D ---------------

      Current thread (0x00000000041e9000): JavaThread "pool-RouterConsumerListenerHandler-local_out299" [_thread_in_Java, id=15917, stack(0x00007fef5679f000,0x00007fef568a0000)]

      siginfo: si_signo: 11 (SIGSEGV), si_code: 2 (SEGV_ACCERR), si_addr: 0x00007ff08bdee860

      Registers:
      RAX=0x00000006d72ba2c8, RBX=0x00007fef5689d0c0, RCX=0x00000000041eacc0, RDX=0x00007ff07c89b6d4
      RSP=0x00007fef5689d190, RBP=0x00007fef5689d2f0, RSI=0x00000006afa297b8, RDI=0x00000000041e91f8
      R8 =0x00007fef5689d048, R9 =0x00000006afa29b90, R10=0x0000000000000000, R11=0x00007ff08bdfcf88
      R12=0x0000000000000000, R13=0x00000000f4b2f2b6, R14=0x0000000000000000, R15=0x00000000041e9000
      RIP=0x00007ff08bdee860, EFLAGS=0x0000000000010202, CSGSFS=0x0000000000000033, ERR=0x0000000000000015
        TRAPNO=0x000000000000000e

      Top of Stack: (sp=0x00007fef5689d190)
      0x00007fef5689d190: 000000079f0a0720 00000006d72ba228
      0x00007fef5689d1a0: 00000006d72ba240 00000006d72b7b38
      0x00007fef5689d1b0: 00007fef5689d238 00007ff07b36a2bd
      0x00007fef5689d1c0: 00007fef5689d230 00007ff07b36a040
      0x00007fef5689d1d0: 0000000000000007 00007ff07b36a040
      0x00007fef5689d1e0: 000000079f0a0720 00000006d72b6880
      0x00007fef5689d1f0: 00000007a5979598 00007fefc28c064d
      0x00007fef5689d200: 00007fef5689d248 00007fefc28c9440
      0x00007fef5689d210: 0000000000000009 00007fefc28c0670
      0x00007fef5689d220: 0000000000000000 00007ff07c579cd4
      0x00007fef5689d230: 0000000000000007 00007ff07b36a114
      0x00007fef5689d240: 00000006af94e370 0000000000000000
      0x00007fef5689d250: 00000006af94e3b8 00000006afa297b8
      0x00007fef5689d260: 00000006d72ba2c8 00007fefc28ee810
      0x00007fef5689d270: 0000000000000001 00007fefc28eb3f0
      0x00007fef5689d280: 00007fef5689d348 00007ff07c01857c
      0x00007fef5689d290: 00007fef5689d2f0 00007ff07b369a90
      0x00007fef5689d2a0: 00007ff07b369a90 00000006af94e370
      0x00007fef5689d2b0: 00007fef5689d2b0 00007fefea3d65d0
      0x00007fef5689d2c0: 00007fef5689d300 00007fefc28f1ed0
      0x00007fef5689d2d0: 0000000000000000 00007fefea3d6608
      0x00007fef5689d2e0: 00007fef5689d2a8 00007fef5689d300
      0x00007fef5689d2f0: 00007fef5689d348 00007ff07b369a90
      0x00007fef5689d300: 00000006af94e370 00007fef5689d308
      0x00007fef5689d310: 00007fefea3da6d3 00007fef5689d358
      0x00007fef5689d320: 00007fefc28f3e50 0000000000000000
      0x00007fef5689d330: 00007fefea3da6f8 00007fef5689d300
      0x00007fef5689d340: 00007fef5689d358 00007fef5689d3a0
      0x00007fef5689d350: 00007ff07b369a90 00000006d72ba1c8
      0x00007fef5689d360: 00007fef5689d360 00007fefc28eb9b9
      0x00007fef5689d370: 00007fef5689d3d0 00007fefc28ee810
      0x00007fef5689d380: 0000000000000000 00007fefc28ebc38

      Instructions: (pc=0x00007ff08bdee860)
      0x00007ff08bdee840: 00 00 00 00 00 00 00 00 60 20 36 7b f0 7f 00 00
      0x00007ff08bdee850: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
      0x00007ff08bdee860: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
      0x00007ff08bdee870: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

      Register to memory mapping:

      RAX=0x00000006d72ba2c8 is an oop
      java.net.SocketTimeoutException
       - klass: 'java/net/SocketTimeoutException'
      RBX=0x00007fef5689d0c0 is pointing into the stack for thread: 0x00000000041e9000
      RCX=0x00000000041eacc0 is an unknown value
      RDX=0x00007ff07c89b6d4 is at entry_point+3764 in (nmethod*)0x00007ff07c89a550
      RSP=0x00007fef5689d190 is pointing into the stack for thread: 0x00000000041e9000
      RBP=0x00007fef5689d2f0 is pointing into the stack for thread: 0x00000000041e9000
      RSI=0x00000006afa297b8 is an oop
      java.net.Socket
       - klass: 'java/net/Socket'
      RDI=0x00000000041e91f8 is an unknown value
      R8 =0x00007fef5689d048 is pointing into the stack for thread: 0x00000000041e9000
      R9 =0x00000006afa29b90 is an oop
      java.io.FileDescriptor
       - klass: 'java/io/FileDescriptor'
      R10=0x0000000000000000 is an unknown value
      R11=0x00007ff08bdfcf88: <offset 0xfc3f88> in /usr/java/jdk1.8.0_111/jre/lib/amd64/server/libjvm.so at 0x00007ff08ae39000
      R12=0x0000000000000000 is an unknown value
      R13=0x00000000f4b2f2b6 is an unknown value
      R14=0x0000000000000000 is an unknown value
      R15=0x00000000041e9000 is a thread


      Stack: [0x00007fef5679f000,0x00007fef568a0000], sp=0x00007fef5689d190, free space=1016k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      V [libjvm.so+0xfb5860]
      j org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.closeIfStale()Z+11
      j org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(Lorg/apache/commons/httpclient/HttpMethod;)V+65
      j org.apache.commons.httpclient.HttpMethodDirector.executeMethod(Lorg/apache/commons/httpclient/HttpMethod;)V+288
      j org.apache.commons.httpclient.HttpClient.executeMethod(Lorg/apache/commons/httpclient/HostConfiguration;Lorg/apache/commons/httpclient/HttpMethod;Lorg/apache/commons/httpclient/HttpState;)I+114
      J 10555 C1 com.xx.xx.xx.protocol.ws.WSDataHandler.writeHttpRequest(Lorg/apache/commons/httpclient/HttpClient;Lorg/apache/commons/httpclient/methods/PostMethod;Ljava/lang/Object;Lcom/dc/esb/container/protocol/ws/WSProtocolConfig;Lcom/dc/esb/container/core/sclite/IBaseContext;)Ljava/lang/Object; (785 bytes) @ 0x00007ff07da25044 [0x00007ff07da22cc0+0x2384]
      J 10151 C1 com.xx.xx.xx.protocol.ws.client.WebServiceConsumer.send(Lcom/dc/esb/container/core/sclite/IBaseContext;)Ljava/lang/Object; (1916 bytes) @ 0x00007ff07d7e80fc [0x00007ff07d7dab20+0xd5dc]
      J 10669 C1 com.xx.xx.xx.protocol.ws.client.WSClientConnector.send(Lcom/dc/esb/container/core/sclite/IBaseContext;)Ljava/lang/Object; (69 bytes) @ 0x00007ff07caba15c [0x00007ff07cab9e40+0x31c]
      J 9636 C1 com.xx.xx.xx.service.DefaultBusinessService.invoke(Lcom/dc/esb/container/core/data/IServiceDataObject;Lcom/dc/esb/container/core/sclite/IBaseContext;)Lcom/dc/esb/container/core/data/IServiceDataObject; (577 bytes) @ 0x00007ff07d3d40c4 [0x00007ff07d3d12c0+0x2e04]
      J 10290 C1 com.xx.xx.xx.adaptor.frame.SynFrameRunnable.processFlow([Ljava/lang/String;Lcom/dc/esb/container/core/frame/IAdaptorFlow;ZI)V (1141 bytes) @ 0x00007ff07d8a289c [0x00007ff07d89e340+0x455c]
      J 10402 C1 com.xx.xx.xx.adaptor.frame.SynFrameRunnable.invoke()V (307 bytes) @ 0x00007ff07d92e754 [0x00007ff07d92c620+0x2134]
      J 10399 C1 com.xx.xx.xx.adaptor.frame.SynFrameRunnable.run()V (511 bytes) @ 0x00007ff07d914924 [0x00007ff07d911e40+0x2ae4]
      J 5496 C1 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007ff07b95e60c [0x00007ff07b95d5e0+0x102c]
      J 5495 C1 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (9 bytes) @ 0x00007ff07b889bc4 [0x00007ff07b889ac0+0x104]
      J 3565 C1 java.lang.Thread.run()V (17 bytes) @ 0x00007ff07bff7bc4 [0x00007ff07bff7a80+0x144]
      v ~StubRoutines::call_stub
      V [libjvm.so+0x690206]
      V [libjvm.so+0x690711]
      V [libjvm.so+0x690bb7]
      V [libjvm.so+0x7278f0]
      V [libjvm.so+0xa742b3]
      V [libjvm.so+0xa743fc]
      V [libjvm.so+0x924ea8]
      C [libpthread.so.0+0x7806] start_thread+0xe6

      and i found some text in core file,i'm not sure it may be a clue

      Exception <a 'java/lang/NullPointerException'> (0x000000070c3ee5f8) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u111/7883/hotspot/src/share/vm/runtime/sharedRuntime.cpp, line 605]
      systemDictionary.cpp, line 210]
      Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x00000006d72ba2c8) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u111/7883/hotspot/src/share/vm/prims/jni.cpp, line 735]


      FREQUENCY : rarely



       

        Attachments

          Activity

            People

            • Assignee:
              sswsharm swati sharma
              Reporter:
              pnarayanaswa Praveen Narayanaswamy
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: