Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8218961

Crash in C2 compiler at Node::rematerialize


    • Subcomponent:
    • CPU:
    • OS:


      CentOS Linux release 7.6.1810
      OpenJDK 64-Bit Server VM (11.0.2+9) for linux-amd64 JRE (11.0.2+9)

      Our application is using large heap (ms/mx=272GB) on Java 11.0.2 with ZGC and -XX:+UseCountedLoopSafepoints (to avoid long time to safepoints).
      Every 3 days in average, the JVM is crashing in C2 compiler at Node::rematerialize() const+0x0

      I found 2 similar issues from 2010 and 2014 that are supposed to be fixed a long time ago :

      I also found a very old article (2012) that says that "-XX:-UseLoopPredicate -XX:-LoopLimitCheck" can resolve the crash.

      Here is an extract of the hs_err crash file :
      # A fatal error has been detected by the Java Runtime Environment:
      # SIGSEGV (0xb) at pc=0x00007fcab30272a0, pid=1, tid=240
      # JRE version: OpenJDK Runtime Environment (11.0.2+9) (build 11.0.2+9)
      # Java VM: OpenJDK 64-Bit Server VM (11.0.2+9, mixed mode, tiered, z gc, linux-amd64)
      # Problematic frame:
      # V [libjvm.so+0xbd02a0] Node::rematerialize() const+0x0
      # Core dump will be written. Default location: /opt/shell/core.1
      # If you would like to submit a bug report, please visit:
      # http://bugreport.java.com/bugreport/crash.jsp

      --------------- S U M M A R Y ------------

      Command Line: -Djdk.serialFilter=* -Dsun.rmi.registry.registryFilter=* -Dsun.rmi.server.disableIncomingHttp=false -Dsun.rmi.dgc.checkInterval=2000 -Djava.awt.headless=true -Xms292057776128 -Xmx292057776128 -XX:+UnlockExperimentalVMOptions -XX:+UseZGC -XX:ConcGCThreads=8 -XX:ParallelGCThreads=8 -javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent.jar=5556:/opt/jmx_exporter/jmx_java_config.yml -Xlog:gc*,gc+heap=debug,gc+age=trace,safepoint:file=/opt/logs/mdds_gc_201902071430.log:time,level,tags -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/data/heapdump -XX:+PrintCommandLineFlags -Dfile.encoding=UTF8 -XX:+UseCountedLoopSafepoints -XX:ErrorFile=/opt/logs/java_error%p.log nims.mddb.clispro.MDDBServer
      Host: Intel Core Processor (Broadwell), 70 cores, 320G, CentOS Linux release 7.6.1810 (Core)
      Time: Mon Feb 11 18:24:35 2019 IST elapsed time: 359687 seconds (4d 3h 54m 47s)

      --------------- T H R E A D ---------------

      Current thread (0x00007fcaac12f5f0): JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=240, stack(0x00007fca76bfc000,0x00007fca76cfd000)]

      Current CompileTask:
      C2:359687050 39433 4 nims.mddb.engine.DimPaneMapper::<init> (77 bytes)

      Stack: [0x00007fca76bfc000,0x00007fca76cfd000], sp=0x00007fca76cf82c8, free space=1008k
      Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
      V [libjvm.so+0xbd02a0] Node::rematerialize() const+0x0
      V [libjvm.so+0x56db1e] PhaseChaitin::Register_Allocate()+0x43e
      V [libjvm.so+0x63867a] Compile::Code_Gen()+0x28a
      V [libjvm.so+0x63bbf0] Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool, DirectiveSet*)+0xc80
      V [libjvm.so+0x547561] C2Compiler::compile_method(ciEnv*, ciMethod*, int, DirectiveSet*)+0xb1
      V [libjvm.so+0x645b4c] CompileBroker::invoke_compiler_on_method(CompileTask*)+0x4fc
      V [libjvm.so+0x6476c8] CompileBroker::compiler_thread_loop()+0x598
      V [libjvm.so+0xdc2b0d] JavaThread::thread_main_inner()+0x21d
      V [libjvm.so+0xdc2eb7] JavaThread::run()+0x377
      V [libjvm.so+0xc076a0] thread_native_entry(Thread*)+0xf0
      siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x000000000000002c


      VM Arguments:
      jvm_args: -Djdk.serialFilter=* -Dsun.rmi.registry.registryFilter=* -Dsun.rmi.server.disableIncomingHttp=false -Dsun.rmi.dgc.checkInterval=2000 -Djava.awt.headless=true -Xms292057776128 -Xmx292057776128 -XX:+UnlockExperimentalVMOptions -XX:+UseZGC -XX:ConcGCThreads=8 -XX:ParallelGCThreads=8 -javaagent:/opt/jmx_exporter/jmx_prometheus_javaagent.jar=5556:/opt/jmx_exporter/jmx_java_config.yml -Xlog:gc*,gc+heap=debug,gc+age=trace,safepoint:file=/opt/logs/mdds_gc_201902071430.log:time,level,tags -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/data/heapdump -XX:+PrintCommandLineFlags -Dfile.encoding=UTF8 -XX:+UseCountedLoopSafepoints -XX:ErrorFile=/opt/logs/java_error%p.log
      java_command: nims.mddb.clispro.MDDBServer
      java_class_path (initial): /opt/jar/admin-api.jar:/opt/jar/alarm-rest-api.jar:/opt/jar/commons-compression.jar:/opt/jar/commons-messaging-kafka.jar:/opt/jar/commons.jar:/opt/jar/core.jar:/opt/jar/dropin.jar:/opt/jar/eaa-proptima-ffrpc.jar:/opt/jar/etl-jms-api.jar:/opt/jar/etl.jar:/opt/jar/feature-enabler.jar:/opt/jar/fflayer.jar:/opt/jar/fm-ingestion-api.jar:/opt/jar/gis.jar:/opt/jar/help.jar:/opt/jar/icm-api.jar:/opt/jar/licmgr-acl.jar:/opt/jar/licmgr-api.jar:/opt/jar/mdds-roles.jar:/opt/jar/nims-auth-api.jar:/opt/jar/nims-nbi-client.jar:/opt/jar/nims.jar:/opt/jar/nims.nbi.api.jar:/opt/jar/nova.oauth-agent.jar:/opt/jar/ptma-thick-client.jar:/opt/jar/reallocation-mapping.jar:/opt/jar/sched.jar:/opt/jar/sys.jar:/opt/jar/tools.jar:/opt/jar/xlgeo-conf.jar:/opt/jar/lib/bcmail-jdk14.jar:/opt/jar/lib/bcprov-jdk14.jar:/opt/jar/lib/bctsp-jdk14.jar:/opt/jar/lib/bsh.jar:/opt/jar/lib/castor.jar:/opt/jar/lib/commons-compress.jar:/opt/jar/lib/commons-io.jar:/opt/jar/lib/commons-lang.jar:/opt/jar/lib/commons-logging.jar:/opt/jar/lib/commons-net.jar:/opt/jar/lib/compress-lzf.jar:/opt/jar/lib/dom4j.jar:/opt/jar/lib/eclipse-collections-api.jar:/opt/jar/lib/eclipse-collections-forkjoin.jar:/opt/jar/lib/eclipse-collections.jar:/opt/jar/lib/elsa.jar:/opt/jar/lib/geoapi-pending.jar:/opt/jar/lib/geotk-metadata.jar:/opt/jar/lib/geotk-referencing.jar:/opt/jar/lib/geotk-utility.jar:/opt/jar/lib/geotk-xml-base.jar:/opt/jar/lib/geronimo-jms_1.1_spec.jar:/opt/jar/lib/hk2-api.jar:/opt/jar/lib/hk2-locator.jar:/opt/jar/lib/hk2-utils.jar:/opt/jar/lib/httpclient.jar:/opt/jar/lib/httpcore.jar:/opt/jar/lib/itext.jar:/opt/jar/lib/jackson-annotations.ja
      Launcher Type: SUN_STANDARD

      [Global flags]
           intx CICompilerCount = 12 {product} {ergonomic}
           uint ConcGCThreads = 8 {product} {command line}
          ccstr ErrorFile = /opt/logs/java_error%p.log {product} {command line}
           bool HeapDumpOnOutOfMemoryError = true {manageable} {command line}
          ccstr HeapDumpPath = /opt/heapdump {manageable} {command line}
         size_t InitialHeapSize = 292057776128 {product} {command line}
         size_t MaxHeapSize = 292057776128 {product} {command line}
         size_t MinHeapDeltaBytes = 2097152 {product} {ergonomic}
          uintx NonNMethodCodeHeapSize = 7591728 {pd product} {ergonomic}
          uintx NonProfiledCodeHeapSize = 122033256 {pd product} {ergonomic}
           uint ParallelGCThreads = 8 {product} {command line}
           bool PrintCommandLineFlags = true {product} {command line}
          uintx ProfiledCodeHeapSize = 122033256 {pd product} {ergonomic}
          uintx ReservedCodeCacheSize = 251658240 {pd product} {ergonomic}
           bool SegmentedCodeCache = true {product} {ergonomic}
           bool UnlockExperimentalVMOptions = true {experimental} {command line}
           bool UseCountedLoopSafepoints = true {C2 product} {command line}
           bool UseZGC = true {experimental} {command line}

      Log output configuration:
       #0: stdout all=warning uptime,level,tags
       #1: stderr all=off uptime,level,tags
       #2: file=/opt/logs/mdds_gc_201902071430.log all=off,gc*=info,age*=trace,safepoint=info,gc+heap=debug time,level,tags filecount=5,filesize=20M

      --------------- S Y S T E M ---------------

      OS:CentOS Linux release 7.6.1810 (Core)
      uname:Linux 3.10.0-862.26.1.el7.bz1657714.x86_64 #1 SMP Mon Dec 10 16:04:19 EST 2018 x86_64
      libc:glibc 2.17 NPTL 2.17
      rlimit: STACK 8192k, CORE infinity, NPROC 1048576, NOFILE 1048576, AS infinity, DATA infinity, FSIZE infinity
      load average:22.58 20.42 19.06

      MemTotal: 722232576 kB
      MemFree: 10751424 kB
      MemAvailable: 293819008 kB
      Buffers: 4420220 kB
      Cached: 555092772 kB
      SwapCached: 0 kB
      Active: 435939340 kB
      Inactive: 240657932 kB
      Active(anon): 306961060 kB
      Inactive(anon): 103989228 kB
      Active(file): 128978280 kB
      Inactive(file): 136668704 kB
      Unevictable: 24796 kB
      Mlocked: 24796 kB
      SwapTotal: 0 kB
      SwapFree: 0 kB
      Dirty: 80284 kB
      Writeback: 0 kB
      AnonPages: 117109716 kB
      Mapped: 287838684 kB
      Shmem: 293857076 kB
      Slab: 26610448 kB
      SReclaimable: 20516812 kB
      SUnreclaim: 6093636 kB
      KernelStack: 311152 kB
      PageTables: 1914916 kB
      NFS_Unstable: 0 kB
      Bounce: 0 kB
      WritebackTmp: 0 kB
      CommitLimit: 361116288 kB
      Committed_AS: 643991644 kB
      VmallocTotal: 34359738367 kB
      VmallocUsed: 2019456 kB
      VmallocChunk: 34357485568 kB
      HardwareCorrupted: 0 kB
      AnonHugePages: 11704320 kB
      CmaTotal: 0 kB
      CmaFree: 0 kB
      HugePages_Total: 0
      HugePages_Free: 0
      HugePages_Rsvd: 0
      HugePages_Surp: 0
      Hugepagesize: 2048 kB
      DirectMap4k: 917320 kB
      DirectMap2M: 123863040 kB
      DirectMap1G: 611319808 kB

      /proc/sys/kernel/threads-max (system-wide limit on the number of threads):

      /proc/sys/vm/max_map_count (maximum number of memory map areas a process may have):

      /proc/sys/kernel/pid_max (system-wide limit on number of process identifiers):

      container (cgroup) information:
      container_type: cgroupv1
      cpu_cpuset_cpus: 0-69
      cpu_memory_nodes: 0
      active_processor_count: 25
      cpu_quota: 2500000
      cpu_period: 100000
      cpu_shares: 10240
      memory_limit_in_bytes: 343597383680
      memory_and_swap_limit_in_bytes: -1
      memory_soft_limit_in_bytes: -1
      memory_usage_in_bytes: 342271852544
      memory_max_usage_in_bytes: 343597383680

      CPU:total 70 (initial active 25) (1 cores per cpu, 1 threads per core) family 6 model 61 stepping 2, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, rtm, 3dnowpref, lzcnt, tsc, bmi1, bmi2, adx, fma
      CPU Model and flags from /proc/cpuinfo:
      model name : Intel Core Processor (Broadwell)
      flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm rdseed adx smap xsaveopt arat

      Memory: 4k page, physical 335544320k(1294484k free), swap 0k(0k free)

      vm_info: OpenJDK 64-Bit Server VM (11.0.2+9) for linux-amd64 JRE (11.0.2+9), built on Jan 17 2019 21:18:23 by "mach5one" with gcc 7.3.0

      FREQUENCY : occasionally




            • Assignee:
              neliasso Nils Eliasson
              webbuggrp Webbug Group
            • Votes:
              0 Vote for this issue
              7 Start watching this issue


              • Created: