Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8251183

JVM Crash During Code Generation on Apache Mesos with Problematic frame: # V [libjvm.so+0x3e7274] ciEnv::get_method_from_handle(Method*)

    Details

    • Subcomponent:
    • CPU:
      x86_64
    • OS:
      linux

      Description

      ADDITIONAL SYSTEM INFORMATION :
      Java HotSpot(TM) 64-Bit Server VM (25.201-b09) for linux-amd64 JRE (1.8.0_201-b09), built on Dec 15 2018 11:35:59 by "java_re" with gcc 7.3.0

      A DESCRIPTION OF THE PROBLEM :
      We are launching tasks on Apache Mesos on JVM running inside Docker container. We have encountered a lot of JVM crashes and we managed to get a full dump. I pasted the first line of my hs_err_pid29683.log file. I also have a replay_pid29683.log and a core.29683 file.

      ERROR MESSAGES/STACK TRACES THAT OCCUR :
      #
      # A fatal error has been detected by the Java Runtime Environment:
      #
      # SIGSEGV (0xb) at pc=0x00007f1c73927274, pid=29683, tid=0x00007f1c576f6700
      #
      # JRE version: Java(TM) SE Runtime Environment (8.0_201-b09) (build 1.8.0_201-b09)
      # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.201-b09 mixed mode linux-amd64 compressed oops)
      # Problematic frame:
      # V [libjvm.so+0x3e7274] ciEnv::get_method_from_handle(Method*)+0x154
      #
      # Core dump written. Default location: /tmp/mesos_work_dir/slave/slaves/cb551b8f-9af5-4f68-bafb-4ead10bc80b4-S0/frameworks/0230d043-6dae-428e-9bbe-d04ee3c66769/executors/sg-mesos-pl-exec-2.4-fgornes@1596615947000-58/runs/f16fc839-752c-4e3e-a030-4947635e39fd/core or core.29683
      #
      # If you would like to submit a bug report, please visit:
      # http://bugreport.java.com/bugreport/crash.jsp
      #

      --------------- T H R E A D ---------------

      Current thread (0x00007f1c6c20c800): JavaThread "C2 CompilerThread2" daemon [_thread_in_vm_trans, id=29703, stack(0x00007f1c575f6000,0x00007f1c576f7000)]

      siginfo: si_signo: 11 (SIGSEGV), si_code: 0 (SI_USER)

      Registers:
      RAX=0x00007f1c14198a90, RBX=0x00007f1c6c20c800, RCX=0x00007f1c74507f98, RDX=0x00007f1c75113c80
      RSP=0x00007f1c576f58f0, RBP=0x00007f1c576f5930, RSI=0x0000000000000069, RDI=0x0000000000000080
      R8 =0x0000000000000067, R9 =0x00000000000000ce, R10=0x0000000000000006, R11=0x0000000000000038
      R12=0x00007f1c576f5cf0, R13=0x00007f1c6c20d2c0, R14=0x00007f1c7450cf44, R15=0x00007f1c74507f8c
      RIP=0x00007f1c73927274, EFLAGS=0x0000000000010202, CSGSFS=0x0000000000000033, ERR=0x0000000000000007
        TRAPNO=0x000000000000000e

      Top of Stack: (sp=0x00007f1c576f58f0)
      0x00007f1c576f58f0: 00007f1c576f5930 00007f1c7392625f
      0x00007f1c576f5900: 00007f1c576f5a40 00007f1c6c20c800
      0x00007f1c576f5910: 00007f1bdc04fbf0 00007f1c6c007ff0
      0x00007f1c576f5920: 00007f1c576f59e0 00007f1c576f5a40
      0x00007f1c576f5930: 00007f1c576f5c60 00007f1c739cb79b
      0x00007f1c576f5940: 00007f1c576f59d0 00007f1c73b6421b
      0x00007f1c576f5950: 00000000000003d8 00007f1c00000000
      0x00007f1c576f5960: 00007f1c6c20cee0 00007f1c6c20d2b8
      0x00007f1c576f5970: 00000000000021ff 00000e0f5d1af810
      0x00007f1c576f5980: 00007f1c6c20ce90 00007f1c6c20ced0
      0x00007f1c576f5990: 00007f00ffffffff 00007f1c576f5900
      0x00007f1c576f59a0: 0000000000000004 00007f1c576f5a40
      0x00007f1c576f59b0: 00007f1c576f59d0 00007f1c739cf9bc
      0x00007f1c576f59c0: 0000000000000000 00007f1c6c20cee0
      0x00007f1c576f59d0: 00007f1c576f5a00 00007f1c73e16e88
      0x00007f1c576f59e0: 0000000000000001 00000000004cb2d9
      0x00007f1c576f59f0: 00000004ce7a1380 00007f1c6c20ce00
      0x00007f1c576f5a00: 00007f1c576f59c0 00007f1c5d121250
      0x00007f1c576f5a10: 0000000000000000 0000000000000000
      0x00007f1c576f5a20: 0000000000000000 00007f1c576f5b30
      0x00007f1c576f5a30: 00007f1c74cea0fd 00007f1c6c200060
      0x00007f1c576f5a40: 00007f1c576f5a48 00007f1c00000006
      0x00007f1c576f5a50: 00007f1c1c0baf60 00007f1c14197fd0
      0x00007f1c576f5a60: 00007f1c14198e30 00007f1c1419ffb8
      0x00007f1c576f5a70: 00000000000083b0 00000000000021ff
      0x00007f1c576f5a80: 00007f1c1c0baf70 0000000000000000
      0x00007f1c576f5a90: 0000000000000000 0000000000000000
      0x00007f1c576f5aa0: 0000000000000000 00007f0000000000
      0x00007f1c576f5ab0: 00007f1c00000000 00007f1bdc04fbf0
      0x00007f1c576f5ac0: 0000000000000000 0000000000000000
      0x00007f1c576f5ad0: 0000000000000000 0000000000000000
      0x00007f1c576f5ae0: 0000000000000000 00007f1c141988f0

      Instructions: (pc=0x00007f1c73927274)
      0x00007f1c73927254: 41 5e 41 5f 5d c3 48 8d 0d 2f 0d be 00 48 89 da
      0x00007f1c73927264: 48 c1 ea 04 23 11 48 8d 0d 27 0d be 00 48 03 11
      0x00007f1c73927274: c7 02 01 00 00 00 eb ae 48 8d 15 0d 0d be 00 48
      0x00007f1c73927284: 89 d8 48 c1 e8 04 23 02 48 8d 15 05 0d be 00 48

      Register to memory mapping:

      RAX=0x00007f1c14198a90 is an unknown value
      RBX=0x00007f1c6c20c800 is a thread
      RCX=0x00007f1c74507f98: <offset 0xfc7f98> in /usr/java/jdk1.8.0_201-amd64/jre/lib/amd64/server/libjvm.so at 0x00007f1c73540000
      RDX=0x00007f1c75113c80 is an unknown value
      RSP=0x00007f1c576f58f0 is pointing into the stack for thread: 0x00007f1c6c20c800
      RBP=0x00007f1c576f5930 is pointing into the stack for thread: 0x00007f1c6c20c800
      RSI=0x0000000000000069 is an unknown value
      RDI=0x0000000000000080 is an unknown value
      R8 =0x0000000000000067 is an unknown value
      R9 =0x00000000000000ce is an unknown value
      R10=0x0000000000000006 is an unknown value
      R11=0x0000000000000038 is an unknown value
      R12=0x00007f1c576f5cf0 is pointing into the stack for thread: 0x00007f1c6c20c800
      R13=0x00007f1c6c20d2c0 is an unknown value
      R14=0x00007f1c7450cf44: <offset 0xfccf44> in /usr/java/jdk1.8.0_201-amd64/jre/lib/amd64/server/libjvm.so at 0x00007f1c73540000
      R15=0x00007f1c74507f8c: <offset 0xfc7f8c> in /usr/java/jdk1.8.0_201-amd64/jre/lib/amd64/server/libjvm.so at 0x00007f1c73540000


      Stack: [0x00007f1c575f6000,0x00007f1c576f7000], sp=0x00007f1c576f58f0, free space=1022k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      V [libjvm.so+0x3e7274] ciEnv::get_method_from_handle(Method*)+0x154
      V [libjvm.so+0x48b79b] CompileBroker::invoke_compiler_on_method(CompileTask*)+0x68b
      V [libjvm.so+0x48d488] CompileBroker::compiler_thread_loop()+0x6d8
      V [libjvm.so+0xa7bb6b] JavaThread::thread_main_inner()+0xdb
      V [libjvm.so+0xa7be71] JavaThread::run()+0x2d1
      V [libjvm.so+0x90d922] java_start(Thread*)+0x102
      C [libpthread.so.0+0x7aa1] start_thread+0xd1


      Current CompileTask:
      C2: 20705 3599 4 sun.reflect.annotation.AnnotationParser::toArray (18 bytes)


      --------------- P R O C E S S ---------------

      Java Threads: ( => current thread )
        0x00007f1bdc001800 JavaThread "Thread-16" [_thread_blocked, id=29894, stack(0x00007f1be775b000,0x00007f1be7f5c000)]
        0x00007f1bd057d800 JavaThread "Okio Watchdog" daemon [_thread_blocked, id=29969, stack(0x00007f1c55d7f000,0x00007f1c55e80000)]
        0x00007f1bd0579000 JavaThread "OkHttp ConnectionPool" daemon [_thread_blocked, id=29968, stack(0x00007f1c5537b000,0x00007f1c5547c000)]
        0x00007f1bd02c6000 JavaThread "SGMesosFramework-stable_sg_mesos_sa:
      bar fw:58 (/sg/noBackup/sg-sg-exec-integration-tests/pipeline/test/pipeline_tests/Executor_testing->retryable_task_once)-thread" [_thread_in_native, id=29955, stack(0x00007f1c5587c000,0x00007f1c5597d000)]
        0x00007f1bbc001800 JavaThread "I/O dispatcher 1" [_thread_in_native, id=29923, stack(0x00007f1c3c0b4000,0x00007f1c3c1b5000)]
        0x00007f1bf4150000 JavaThread "pool-11-thread-1" [_thread_in_native, id=29922, stack(0x00007f1c54007000,0x00007f1c54108000)]
        0x00007f1bd0049000 JavaThread "process reaper" daemon [_thread_blocked, id=29906, stack(0x00007f1c549d8000,0x00007f1c54a11000)]
        0x00007f1c000c1000 JavaThread "pool-6-thread-1" [_thread_blocked, id=29904, stack(0x00007f1c55a7e000,0x00007f1c55b7f000)]
        0x00007f1c6d0d8800 JavaThread "BatchNotifier" [_thread_blocked, id=29858, stack(0x00007f1c5527a000,0x00007f1c5537b000)]
        0x00007f1c6c22c000 JavaThread "Service Thread" daemon [_thread_blocked, id=29705, stack(0x00007f1c573f4000,0x00007f1c574f5000)]
        0x00007f1c6c20e800 JavaThread "C1 CompilerThread3" daemon [_thread_blocked, id=29704, stack(0x00007f1c574f5000,0x00007f1c575f6000)]
      =>0x00007f1c6c20c800 JavaThread "C2 CompilerThread2" daemon [_thread_in_vm_trans, id=29703, stack(0x00007f1c575f6000,0x00007f1c576f7000)]
        0x00007f1c6c20a800 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=29702, stack(0x00007f1c576f7000,0x00007f1c577f8000)]
        0x00007f1c6c207800 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=29701, stack(0x00007f1c577f8000,0x00007f1c578f9000)]
        0x00007f1c6c206000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=29700, stack(0x00007f1c578f9000,0x00007f1c579fa000)]
        0x00007f1c6c1d4000 JavaThread "Finalizer" daemon [_thread_blocked, id=29699, stack(0x00007f1c579fa000,0x00007f1c57afb000)]
        0x00007f1c6c1d1800 JavaThread "Reference Handler" daemon [_thread_blocked, id=29698, stack(0x00007f1c57afb000,0x00007f1c57bfc000)]
        0x00007f1c6c00a000 JavaThread "main" [_thread_in_native, id=29688, stack(0x00007f1c7500b000,0x00007f1c7510c000)]

      Other Threads:
        0x00007f1c6c1c7800 VMThread [stack: 0x00007f1c57bfc000,0x00007f1c57cfd000] [id=29697]
        0x00007f1c6c22e800 WatcherThread [stack: 0x00007f1c572f3000,0x00007f1c573f4000] [id=29706]

      VM state:synchronizing (normal execution)

      VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
      [0x00007f1c6c007570] Safepoint_lock - owner thread: 0x00007f1c6c1c7800
      [0x00007f1c6c0075f0] Threads_lock - owner thread: 0x00007f1c6c1c7800

      heap address: 0x00000000f0000000, size: 256 MB, Compressed Oops mode: 32-bit
      Narrow klass base: 0x0000000000000000, Narrow klass shift: 3
      Compressed class space size: 1073741824 Address: 0x0000000100000000

      Heap:
       PSYoungGen total 74752K, used 19392K [0x00000000fab00000, 0x0000000100000000, 0x0000000100000000)
        eden space 63488K, 15% used [0x00000000fab00000,0x00000000fb453678,0x00000000fe900000)
        from space 11264K, 87% used [0x00000000ff500000,0x00000000ffe9cba0,0x0000000100000000)
        to space 11776K, 0% used [0x00000000fe900000,0x00000000fe900000,0x00000000ff480000)
       ParOldGen total 175104K, used 13097K [0x00000000f0000000, 0x00000000fab00000, 0x00000000fab00000)
        object space 175104K, 7% used [0x00000000f0000000,0x00000000f0cca508,0x00000000fab00000)
       Metaspace used 46141K, capacity 47088K, committed 47232K, reserved 1091584K
        class space used 5658K, capacity 5920K, committed 6016K, reserved 1048576K

      Card table byte_map: [0x00007f1c74f82000,0x00007f1c75003000] byte_map_base: 0x00007f1c74802000

      Marking Bits: (ParMarkBitMap*) 0x00007f1c7450bd80
       Begin Bits: [0x00007f1c71484000, 0x00007f1c71884000)
       End Bits: [0x00007f1c71884000, 0x00007f1c71c84000)

      Polling page: 0x00007f1c75114000

      CodeCache: size=245760Kb used=10476Kb max_used=10476Kb free=235283Kb
       bounds [0x00007f1c5d000000, 0x00007f1c5da50000, 0x00007f1c6c000000]
       total_blobs=3810 nmethods=3260 adapters=463
       compilation: enabled

      Compilation events (10 events):
      Event: 20.663 Thread 0x00007f1c6c20e800 3591 3 java.util.TreeMap::<init> (20 bytes)
      Event: 20.663 Thread 0x00007f1c6c20e800 nmethod 3591 0x00007f1c5da43650 code [0x00007f1c5da437c0, 0x00007f1c5da439f0]
      Event: 20.663 Thread 0x00007f1c6c20e800 3587 3 com.fasterxml.jackson.databind.introspect.POJOPropertyBuilder$5::withMember (12 bytes)
      Event: 20.663 Thread 0x00007f1c6c20e800 nmethod 3587 0x00007f1c5da43ad0 code [0x00007f1c5da43c40, 0x00007f1c5da43e68]
      Event: 20.663 Thread 0x00007f1c6c20e800 3588 3 com.fasterxml.jackson.databind.introspect.JacksonAnnotationIntrospector::findPropertyDescription (27 bytes)
      Event: 20.663 Thread 0x00007f1c6c20e800 nmethod 3588 0x00007f1c5da43f50 code [0x00007f1c5da440e0, 0x00007f1c5da445f8]
      Event: 20.663 Thread 0x00007f1c6c20e800 3595 3 com.fasterxml.jackson.databind.type.TypeFactory::_resolveSuperInterfaces (72 bytes)
      Event: 20.664 Thread 0x00007f1c6c20e800 nmethod 3595 0x00007f1c5da44790 code [0x00007f1c5da44960, 0x00007f1c5da45198]
      Event: 20.664 Thread 0x00007f1c6c20e800 3594 3 com.fasterxml.jackson.databind.type.ClassStack::resolveSelfReferences (43 bytes)
      Event: 20.664 Thread 0x00007f1c6c20c800 3599 4 sun.reflect.annotation.AnnotationParser::toArray (18 bytes)

      GC Heap History (10 events):
      Event: 11.639 GC heap before
      {Heap before GC invocations=6 (full 1):
       PSYoungGen total 76288K, used 72130K [0x00000000fab00000, 0x0000000100000000, 0x0000000100000000)
        eden space 65536K, 100% used [0x00000000fab00000,0x00000000feb00000,0x00000000feb00000)
        from space 10752K, 61% used [0x00000000ff580000,0x00000000ffbf0b88,0x0000000100000000)
        to space 10752K, 0% used [0x00000000feb00000,0x00000000feb00000,0x00000000ff580000)
       ParOldGen total 175104K, used 7545K [0x00000000f0000000, 0x00000000fab00000, 0x00000000fab00000)
        object space 175104K, 4% used [0x00000000f0000000,0x00000000f075e5c8,0x00000000fab00000)
       Metaspace used 31326K, capacity 31892K, committed 32256K, reserved 1077248K
        class space used 4078K, capacity 4240K, committed 4352K, reserved 1048576K
      Event: 11.657 GC heap after
      Heap after GC invocations=6 (full 1):
       PSYoungGen total 74240K, used 8600K [0x00000000fab00000, 0x0000000100000000, 0x0000000100000000)
        eden space 65536K, 0% used [0x00000000fab00000,0x00000000fab00000,0x00000000feb00000)
        from space 8704K, 98% used [0x00000000feb00000,0x00000000ff366188,0x00000000ff380000)
        to space 11264K, 0% used [0x00000000ff500000,0x00000000ff500000,0x0000000100000000)
       ParOldGen total 175104K, used 7553K [0x00000000f0000000, 0x00000000fab00000, 0x00000000fab00000)
        object space 175104K, 4% used [0x00000000f0000000,0x00000000f07605c8,0x00000000fab00000)
       Metaspace used 31326K, capacity 31892K, committed 32256K, reserved 1077248K
        class space used 4078K, capacity 4240K, committed 4352K, reserved 1048576K
      }
      Event: 12.111 GC heap before
      {Heap before GC invocations=7 (full 1):
       PSYoungGen total 74240K, used 71695K [0x00000000fab00000, 0x0000000100000000, 0x0000000100000000)
        eden space 65536K, 96% used [0x00000000fab00000,0x00000000fe89daa8,0x00000000feb00000)
        from space 8704K, 98% used [0x00000000feb00000,0x00000000ff366188,0x00000000ff380000)
        to space 11264K, 0% used [0x00000000ff500000,0x00000000ff500000,0x0000000100000000)
       ParOldGen total 175104K, used 7553K [0x00000000f0000000, 0x00000000fab00000, 0x00000000fab00000)
        object space 175104K, 4% used [0x00000000f0000000,0x00000000f07605c8,0x00000000fab00000)
       Metaspace used 34759K, capacity 35358K, committed 35456K, reserved 1079296K
        class space used 4516K, capacity 4697K, committed 4736K, reserved 1048576K
      Event: 12.120 GC heap after
      Heap after GC invocations=7 (full 1):
       PSYoungGen total 75776K, used 8889K [0x00000000fab00000, 0x0000000100000000, 0x0000000100000000)
        eden space 64512K, 0% used [0x00000000fab00000,0x00000000fab00000,0x00000000fea00000)
        from space 11264K, 78% used [0x00000000ff500000,0x00000000ffdae648,0x0000000100000000)
        to space 11264K, 0% used [0x00000000fea00000,0x00000000fea00000,0x00000000ff500000)
       ParOldGen total 175104K, used 7561K [0x00000000f0000000, 0x00000000fab00000, 0x00000000fab00000)
        object space 175104K, 4% used [0x00000000f0000000,0x00000000f07625c8,0x00000000fab00000)
       Metaspace used 34759K, capacity 35358K, committed 35456K, reserved 1079296K
        class space used 4516K, capacity 4697K, committed 4736K, reserved 1048576K
      }
      Event: 12.120 GC heap before
      {Heap before GC invocations=8 (full 2):
       PSYoungGen total 75776K, used 8889K [0x00000000fab00000, 0x0000000100000000, 0x0000000100000000)
        eden space 64512K, 0% used [0x00000000fab00000,0x00000000fab00000,0x00000000fea00000)
        from space 11264K, 78% used [0x00000000ff500000,0x00000000ffdae648,0x0000000100000000)
        to space 11264K, 0% used [0x00000000fea00000,0x00000000fea00000,0x00000000ff500000)
       ParOldGen total 175104K, used 7561K [0x00000000f0000000, 0x00000000fab00000, 0x00000000fab00000)
        object space 175104K, 4% used [0x00000000f0000000,0x00000000f07625c8,0x00000000fab00000)
       Metaspace used 34759K, capacity 35358K, committed 35456K, reserved 1079296K
        class space used 4516K, capacity 4697K, committed 4736K, reserved 1048576K
      Event: 12.165 GC heap after
      Heap after GC invocations=8 (full 2):
       PSYoungGen total 75776K, used 0K [0x00000000fab00000, 0x0000000100000000, 0x0000000100000000)
        eden space 64512K, 0% used [0x00000000fab00000,0x00000000fab00000,0x00000000fea00000)
        from space 11264K, 0% used [0x00000000ff500000,0x00000000ff500000,0x0000000100000000)
        to space 11264K, 0% used [0x00000000fea00000,0x00000000fea00000,0x00000000ff500000)
       ParOldGen total 175104K, used 13081K [0x00000000f0000000, 0x00000000fab00000, 0x00000000fab00000)
        object space 175104K, 7% used [0x00000000f0000000,0x00000000f0cc6508,0x00000000fab00000)
       Metaspace used 34759K, capacity 35358K, committed 35456K, reserved 1079296K
        class space used 4516K, capacity 4697K, committed 4736K, reserved 1048576K
      }
      Event: 12.496 GC heap before
      {Heap before GC invocations=9 (full 2):
       PSYoungGen total 75776K, used 64512K [0x00000000fab00000, 0x0000000100000000, 0x0000000100000000)
        eden space 64512K, 100% used [0x00000000fab00000,0x00000000fea00000,0x00000000fea00000)
        from space 11264K, 0% used [0x00000000ff500000,0x00000000ff500000,0x0000000100000000)
        to space 11264K, 0% used [0x00000000fea00000,0x00000000fea00000,0x00000000ff500000)
       ParOldGen total 175104K, used 13081K [0x00000000f0000000, 0x00000000fab00000, 0x00000000fab00000)
        object space 175104K, 7% used [0x00000000f0000000,0x00000000f0cc6508,0x00000000fab00000)
       Metaspace used 37802K, capacity 38664K, committed 38912K, reserved 1083392K

      STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
      I do believe this is a memory issue and I can re-run my software layers to reproduce the bug but since this is happening in my Mesos environment, I do not have specific steps to reproduce the bug. I'll be happy to test a lot of stuff for you guys.


      FREQUENCY : occasionally


        Attachments

          Activity

            People

            • Assignee:
              sswsharm swati sharma
              Reporter:
              webbuggrp Webbug Group
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: