Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8273019

Crash with native code: [libc.so.6+0x81a8c] cfree+0x1c

    XMLWordPrintable

    Details

    • Subcomponent:
    • CPU:
      x86_64
    • OS:
      linux

      Description

      ADDITIONAL SYSTEM INFORMATION :
      netty-all-4.1.50.Final.jar
      netty-tcnative-2.0.30.Final-linux-x86_64.jar
      libc:glibc 2.28 NPTL 2.28
      OpenSSL: OpenSSL 1.0.2v-fips 5 May 2020
      JRE: OpenJDK Runtime Environment (Zulu11.39+16-SA) (11.0.7+10) (build 11.0.7+10-LTS)

      A DESCRIPTION OF THE PROBLEM :
      We have our application deployed at customer premises. Out of multiple installations, one instance is constantly crashing while executing the native code as mentioned in the title.

      The crashes are observed at very uneven intervals ranging from 4 hours to even more than 5 days. All the occurrences show that the crash happens while executing the Java frame: io.netty.internal.tcnative.SSL.readFromSSL and native library and offset of [libc.so.6+0x81a8c]. Java frames look similar in most of the cases.

      Providing below relevant sections of the crash dump file. Any inputs would be of great help.

      #
      # A fatal error has been detected by the Java Runtime Environment:
      #
      # SIGSEGV (0xb) at pc=0x00007fdb64c26a8c, pid=8969, tid=9065
      #
      # JRE version: OpenJDK Runtime Environment (Zulu11.39+16-SA) (11.0.7+10) (build 11.0.7+10-LTS)
      # Java VM: OpenJDK 64-Bit Server VM (11.0.7+10-LTS, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
      # Problematic frame:
      # C [libc.so.6+0x81a8c] cfree+0x1c
      #



      ost: Intel(R) Xeon(R) Gold 6142 CPU @ 2.60GHz, 4 cores, 15G, VMware Photon OS 3.0

      --------------- T H R E A D ---------------

      Current thread (0x00007fdaf8018000): JavaThread "nioEventLoopGroup-10-4" [_thread_in_native, id=9065, stack(0x00007fdab94f1000,0x00007fdab95f2000)]

      Stack: [0x00007fdab94f1000,0x00007fdab95f2000], sp=0x00007fdab95efde8, free space=1019k
      Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
      C [libc.so.6+0x81a8c] cfree+0x1c

      Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
      J 3701 io.netty.internal.tcnative.SSL.readFromSSL(JJI)I (0 bytes) @ 0x00007fdb4fbaa177 [0x00007fdb4fbaa120+0x0000000000000057]
      J 10590 c2 io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap([Ljava/nio/ByteBuffer;II[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; (1438 bytes) @ 0x00007fdb5032c00c [0x00007fdb5032ba40+0x00000000000005cc]
      J 6066 c2 io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(Lio/netty/handler/ssl/SslHandler;Lio/netty/buffer/ByteBuf;IILio/netty/buffer/ByteBuf;)Ljavax/net/ssl/SSLEngineResult; (134 bytes) @ 0x00007fdb4fd49d90 [0x00007fdb4fd499e0+0x00000000000003b0]
      J 10485 c2 io.netty.handler.ssl.SslHandler.unwrap(Lio/netty/channel/ChannelHandlerContext;Lio/netty/buffer/ByteBuf;II)I (629 bytes) @ 0x00007fdb5034d364 [0x00007fdb5034d080+0x00000000000002e4]
      J 10202 c2 io.netty.handler.ssl.SslHandler.decode(Lio/netty/channel/ChannelHandlerContext;Lio/netty/buffer/ByteBuf;Ljava/util/List;)V (31 bytes) @ 0x00007fdb4fee5bc0 [0x00007fdb4fee58a0+0x0000000000000320]
      J 10198 c2 io.netty.handler.codec.ByteToMessageDecoder.callDecode(Lio/netty/channel/ChannelHandlerContext;Lio/netty/buffer/ByteBuf;Ljava/util/List;)V (172 bytes) @ 0x00007fdb4fee92c8 [0x00007fdb4fee91e0+0x00000000000000e8]
      J 6825 c2 io.netty.handler.codec.ByteToMessageDecoder.channelRead(Lio/netty/channel/ChannelHandlerContext;Ljava/lang/Object;)V (328 bytes) @ 0x00007fdb4fe2a200 [0x00007fdb4fe29d80+0x0000000000000480]
      J 8672 c2 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(Ljava/lang/Object;)Lio/netty/channel/ChannelHandlerContext; (12 bytes) @ 0x00007fdb500e9f28 [0x00007fdb500e9cc0+0x0000000000000268]
      J 7050 c2 io.netty.handler.timeout.IdleStateHandler.channelRead(Lio/netty/channel/ChannelHandlerContext;Ljava/lang/Object;)V (42 bytes) @ 0x00007fdb4fe5eafc [0x00007fdb4fe5eaa0+0x000000000000005c]
      J 8672 c2 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(Ljava/lang/Object;)Lio/netty/channel/ChannelHandlerContext; (12 bytes) @ 0x00007fdb500e9f28 [0x00007fdb500e9cc0+0x0000000000000268]
      J 6504 c2 io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(Lio/netty/channel/ChannelHandlerContext;Ljava/lang/Object;)V (9 bytes) @ 0x00007fdb4fdd6964 [0x00007fdb4fdd6920+0x0000000000000044]
      J 9935 c2 io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read()V (298 bytes) @ 0x00007fdb4fea0678 [0x00007fdb4fea0180+0x00000000000004f8]
      J 9969 c2 io.netty.channel.nio.NioEventLoop.processSelectedKey(Ljava/nio/channels/SelectionKey;Lio/netty/channel/nio/AbstractNioChannel;)V (136 bytes) @ 0x00007fdb4fea9300 [0x00007fdb4fea9220+0x00000000000000e0]
      J 10373% c2 io.netty.channel.nio.NioEventLoop.run()V (419 bytes) @ 0x00007fdb5033cef4 [0x00007fdb5033ca20+0x00000000000004d4]
      j io.netty.util.concurrent.SingleThreadEventExecutor$4.run()V+44
      j io.netty.util.internal.ThreadExecutorMap$2.run()V+11
      j io.netty.util.concurrent.FastThreadLocalRunnable.run()V+4
      j java.lang.Thread.run()V+11 java.base@11.0.7
      v ~StubRoutines::call_stub

      siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000288

      Top of Stack: (sp=0x00007fdab95efde8)
      0x00007fdab95efde8: 00007fdaea98389a 00007fdb142ab520
      0x00007fdab95efdf8: 00007fdaea7ed9fc 0000000000000001
      0x00007fdab95efe08: 00007fdb5cf82540 0000000000000100
      0x00007fdab95efe18: 00007fdab95effd0 00007fdb5cf82540

      Instructions: (pc=0x00007fdb64c26a8c)
      0x00007fdb64c2698c: 85 c0 48 89 c2 75 2c 4d 85 e4 74 27 0f 1f 84 00
      0x00007fdb64c2699c: 00 00 00 00 4c 89 e7 48 89 ee e8 b5 c5 ff ff 48
      0x00007fdb64c269ac: 89 ea 48 89 de 48 89 c7 49 89 c4 e8 24 ed ff ff
      0x00007fdb64c269bc: 48 89 c2 4d 85 e4 0f 84 25 ff ff ff e9 f1 fe ff
      0x00007fdb64c269cc: ff 0f 1f 00 48 8b 05 a1 04 13 00 31 d2 64 c7 00
      0x00007fdb64c269dc: 16 00 00 00 e9 e9 fd ff ff 48 8d 0d 6c bd 0f 00
      0x00007fdb64c269ec: 48 8d 35 13 7a 0f 00 48 8d 3d 4e bb 0f 00 ba e9
      0x00007fdb64c269fc: 0c 00 00 e8 ec b4 ff ff 48 89 fb e9 6a fd ff ff
      0x00007fdb64c26a0c: 0f 1f 40 00 48 83 ec 18 48 8b 05 cd 04 13 00 48
      0x00007fdb64c26a1c: c7 00 00 00 00 00 8b 05 3c 08 13 00 85 c0 78 14
      0x00007fdb64c26a2c: 48 8b 54 24 18 48 83 c4 18 e9 06 fd ff ff 66 0f
      0x00007fdb64c26a3c: 1f 44 00 00 48 89 74 24 08 48 89 3c 24 e8 c2 bb
      0x00007fdb64c26a4c: ff ff 48 8b 74 24 08 48 8b 3c 24 48 8b 54 24 18
      0x00007fdb64c26a5c: 48 83 c4 18 e9 db fc ff ff 66 66 2e 0f 1f 84 00
      0x00007fdb64c26a6c: 00 00 00 00 48 8b 05 89 04 13 00 48 8b 00 48 85
      0x00007fdb64c26a7c: c0 0f 85 dd 00 00 00 48 85 ff 0f 84 ac 00 00 00
      0x00007fdb64c26a8c: 48 8b 47 f8 48 8d 77 f0 a8 02 74 48 8b 15 1e 08
      0x00007fdb64c26a9c: 13 00 85 d2 75 33 48 3b 05 e7 07 13 00 76 2a 48
      0x00007fdb64c26aac: 3d 00 00 00 02 77 22 48 3b 35 7e 2e 13 00 0f 83
      0x00007fdb64c26abc: b0 00 00 00 48 83 e0 f8 48 89 05 c5 07 13 00 48
      0x00007fdb64c26acc: 01 c0 48 89 05 ab 07 13 00 48 89 f7 e9 43 b9 ff
      0x00007fdb64c26adc: ff 0f 1f 00 48 89 c2 83 e2 04 48 83 3d ba 07 13
      0x00007fdb64c26aec: 00 00 74 2b 48 8b 0d 91 02 13 00 48 85 d2 64 4c
      0x00007fdb64c26afc: 8b 01 74 40 48 89 f7 48 81 e7 00 00 00 fc 48 8b
      0x00007fdb64c26b0c: 3f 49 39 f8 75 1a 48 83 e0 f8 64 48 29 41 08 48
      0x00007fdb64c26b1c: 85 d2 74 2c 48 89 f0 48 25 00 00 00 fc 48 8b 38
      0x00007fdb64c26b2c: 31 d2 e9 0d cb ff ff 0f 1f 44 00 00 f3 c3 66 0f
      0x00007fdb64c26b3c: 1f 44 00 00 48 8d 3d f9 10 13 00 49 39 f8 74 c6
      0x00007fdb64c26b4c: 48 8d 3d ed 10 13 00 31 d2 e9 e6 ca ff ff 66 0f
      0x00007fdb64c26b5c: 1f 44 00 00 48 8b 34 24 ff e0 66 2e 0f 1f 84 00
      0x00007fdb64c26b6c: 00 00 00 00 48 3b 35 b9 2d 13 00 0f 83 43 ff ff
      0x00007fdb64c26b7c: ff e9 53 ff ff ff 66 66 2e 0f 1f 84 00 00 00 00


      Stack slot to memory mapping:
      stack at sp + 0 slots: 0x00007fdaea98389a: SSL_SESSION_free+0x000000000000012a in /lib/libssl.so.1.1 at 0x00007fdaea942000
      stack at sp + 1 slots: 0x00007fdb142ab520 points into unknown readable memory: 70 f3 3f 14 db 7f 00 00
      stack at sp + 2 slots: 0x00007fdaea7ed9fc: OPENSSL_LH_doall_arg+0x000000000000004c in /lib/libcrypto.so.1.1 at 0x00007fdaea67c000
      stack at sp + 3 slots: 0x0000000000000001 is an unknown value
      stack at sp + 4 slots: 0x00007fdb5cf82540 points into unknown readable memory: c0 cc 9c ea da 7f 00 00
      stack at sp + 5 slots: 0x0000000000000100 is an unknown value
      stack at sp + 6 slots: 0x00007fdab95effd0 is pointing into the stack for thread: 0x00007fdaf8018000
      stack at sp + 7 slots: 0x00007fdb5cf82540 points into unknown readable memory: c0 cc 9c ea da 7f 00 00

      VM state:not at safepoint (normal execution)

      VM Mutex/Monitor currently owned by a thread: None

      Heap address: 0x0000000705400000, size: 4012 MB, Compressed Oops mode: Zero based, Oop shift amount: 3
      Narrow klass base: 0x0000000800000000, Narrow klass shift: 0
      Compressed class space size: 1073741824 Address: 0x0000000800000000

      Heap:
       garbage-first heap total 311296K, used 166454K [0x0000000705400000, 0x0000000800000000)
        region size 1024K, 132 young (135168K), 2 survivors (2048K)
       Metaspace used 47712K, capacity 48362K, committed 48668K, reserved 1091584K
        class space used 5432K, capacity 5678K, committed 5760K, reserved 1048576K

      Metaspace:

      Usage:
        Non-class: 41.68 MB capacity, 41.29 MB (>99%) used, 317.92 KB ( <1%) free+waste, 86.06 KB ( <1%) overhead.
            Class: 5.54 MB capacity, 5.31 MB ( 96%) used, 209.49 KB ( 4%) free+waste, 35.69 KB ( <1%) overhead.
             Both: 47.23 MB capacity, 46.59 MB ( 99%) used, 527.41 KB ( 1%) free+waste, 121.75 KB ( <1%) overhead.

      Virtual space:
        Non-class space: 42.00 MB reserved, 41.90 MB (>99%) committed
            Class space: 1.00 GB reserved, 5.62 MB ( <1%) committed
                   Both: 1.04 GB reserved, 47.53 MB ( 4%) committed

      Chunk freelists:
         Non-Class: 140.00 KB
             Class: 0 bytes
              Both: 140.00 KB

      MaxMetaspaceSize: unlimited
      CompressedClassSpaceSize: 1.00 GB

      CodeHeap 'non-profiled nmethods': size=120032Kb used=10300Kb max_used=10300Kb free=109731Kb
       bounds [0x00007fdb4fa32000, 0x00007fdb50452000, 0x00007fdb56f6a000]
      CodeHeap 'profiled nmethods': size=120032Kb used=12721Kb max_used=19635Kb free=107310Kb
       bounds [0x00007fdb484fa000, 0x00007fdb4985a000, 0x00007fdb4fa32000]
      CodeHeap 'non-nmethods': size=5696Kb used=1238Kb max_used=1298Kb free=4457Kb
       bounds [0x00007fdb47f6a000, 0x00007fdb481da000, 0x00007fdb484fa000]
       total_blobs=17519 nmethods=8176 adapters=533
       compilation: enabled
                    stopped_count=0, restarted_count=0
       full_count=0


      VM Arguments:
      jvm_args: -Dlog4j.configuration=file:/............/conf/log4j.properties
      java_command: /......./application-version.jar
      java_class_path (initial): /......./application-version.jar
      Launcher Type: SUN_STANDARD

      [Global flags]
           intx CICompilerCount = 3 {product} {ergonomic}
           uint ConcGCThreads = 1 {product} {ergonomic}
           uint G1ConcRefinementThreads = 4 {product} {ergonomic}
         size_t G1HeapRegionSize = 1048576 {product} {ergonomic}
          uintx GCDrainStackTargetSize = 64 {product} {ergonomic}
         size_t InitialHeapSize = 264241152 {product} {ergonomic}
         size_t MarkStackSize = 4194304 {product} {ergonomic}
         size_t MaxHeapSize = 4206886912 {product} {ergonomic}
         size_t MaxNewSize = 2523922432 {product} {ergonomic}
         size_t MinHeapDeltaBytes = 1048576 {product} {ergonomic}
          uintx NonNMethodCodeHeapSize = 5830732 {pd product} {ergonomic}
          uintx NonProfiledCodeHeapSize = 122913754 {pd product} {ergonomic}
          uintx ProfiledCodeHeapSize = 122913754 {pd product} {ergonomic}
          uintx ReservedCodeCacheSize = 251658240 {pd product} {ergonomic}
           bool SegmentedCodeCache = true {product} {ergonomic}
           bool UseCompressedClassPointers = true {lp64_product} {ergonomic}
           bool UseCompressedOops = true {lp64_product} {ergonomic}
           bool UseG1GC = true {product} {ergonomic}

      Logging:
      Log output configuration:
       #0: stdout all=warning uptime,level,tags
       #1: stderr all=off uptime,level,tags

      Environment Variables:
      PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin


      libc:glibc 2.28 NPTL 2.28





      STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
      We are trying to reproduce this but have not been able to do so.

      EXPECTED VERSUS ACTUAL BEHAVIOR :
      EXPECTED -
      No crash should occur
      ACTUAL -
      Crash occurs and the Java process gets restarted

      FREQUENCY : occasionally


        Attachments

          Activity

            People

            Assignee:
            tongwan Andrew Wang
            Reporter:
            webbuggrp Webbug Group
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: