Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8244611

Launching JVM with large '@' argument file causes random JVM crashes

    Details

    • Subcomponent:
    • CPU:
      x86_64
    • OS:
      windows_10

      Description

      ADDITIONAL SYSTEM INFORMATION :
      Using AdoptOpenJdk 11.06

      A DESCRIPTION OF THE PROBLEM :
      For a detailed history see here: https://github.com/spring-projects/sts4/issues/434

      I will summarise our findings here:

      - affects users on Windows
      - affects Java 11 specifically, may affect Java 9 but this has not been confirmed / tested.
      - java is launched with a fairly long commandline, some of which (the -classpath argument) is provided as a `@file` argument.
      - behavior observed is random JVM crashes. Even when launching the same app, with the exact same commandline repeatedly, it sometimes starts up and sometimes crashes, either with some crash message in the console, or without any output at all.

      STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
      This is a 'sample' project that should allow to reproduce the issue.
      https://drive.google.com/file/d/1iS9U-lAkZk7buciYJzBCbBzoBpBu0Q4x/view?usp=sharing

      - Find a windows 10 VM or machine
      - Create a folder called `C:\workspace`.
      - Download the sample zip
      - unpack it with `jar xvf <your-zip>` into the workspace folder.
      - when you are done... you will have a folder called 'C:\workspace\demo'.

      This project contains:
      - a 'run.sh' script. It can be run from 'git bash'. Or you can just look at it to see what command to
        type in a terminal to make the bug happen.
      - You probably need to change the first line of the script to point to wherever you have the 'java' executable for JDK 11.06
      - If you have all the files in C:\workspace\demo the script should launch a spring boot applications.

      If you repeatedly run this script a few times, you should eventually
      see a JVM crash. I'd say that chance is a little better than
      fifty fifty for the app to launch *normally* and the bug to
      not manifest (in this case it will crash with
      some error printed by spring boot, this is the expected
      behavior.

      When the bug manifest you will instead get something like this:

      #
      # A fatal error has been detected by the Java Runtime Environment:
      #
      # Internal Error (c:/cygwin64/tmp/openjdk-jdk11u-windows-x86-32-hotspot/workspace/build/src/src/hotspot/share/runtime/signature.cpp:53), pid=9476, tid=3428
      # fatal error: expecting (
      #
      # JRE version: OpenJDK Runtime Environment (11.0.6+10) (build 11.0.6+10)
      # Java VM: OpenJDK Client VM (11.0.6+10, mixed mode, serial gc, windows-x86)
      # No core dump will be written. Minidumps are not enabled by default on client versions of Windows
      #
      # An error report file with more information is saved as:
      # C:\workspace\demo\hs_err_pid9476.log
      #
      # If you would like to submit a bug report, please visit:
      # https://github.com/AdoptOpenJDK/openjdk-support/issues
      #

      You can also find a number of 'hs_err_pid' files in the 'demo' folder that
      were the result of these crashes during my own experiments.

      The bug probably has something to do with the fact that the commandline itself, or the '@file' argument has
      a fairly large amount of data. Though we are not exactly sure what the precise pattern is for triggering it.
      Several varations have been tried but it's hard to be sure of anything because of how the bug manifest only
      randomly.
       
      One theory is that maybe it just comes down to exceeding a certain amount of data rather than the specific details
      of the argument data. Maybe there is a buffer overflow somewhere during argument processing causing memory corruption?
      This is just a theory, but I think it fits with the facts:

      - adding more arguments / dependencies seems to cause this to happen more.
      - random behavior could be the result of memory corruption from a buffer overflow.

      EXPECTED VERSUS ACTUAL BEHAVIOR :
      EXPECTED -
      THe app launches normally and you see output that starts with:


        . ____ _ __ _ _
       /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
      ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
       \\/ ___)| |_)| | | | | || (_| | ) ) ) )
        ' |____| .__|_| |_|_| |_\__, | / / / /
       =========|_|==============|___/=/_/_/_/
       :: Spring Boot :: (v2.2.6.RELEASE)

      2020-04-09 17:01:49.182 INFO 7944 --- [ main] com.example.demo.DemoAp...
      ACTUAL -
      This is a typical failure for me:

      #
      # A fatal error has been detected by the Java Runtime Environment:
      #
      # Internal Error (c:/cygwin64/tmp/openjdk-jdk11u-windows-x86-32-hotspot/workspace/build/src/src/hotspot/share/runtime/signature.cpp:53), pid=9476, tid=3428
      # fatal error: expecting (
      #
      # JRE version: OpenJDK Runtime Environment (11.0.6+10) (build 11.0.6+10)
      # Java VM: OpenJDK Client VM (11.0.6+10, mixed mode, serial gc, windows-x86)
      # No core dump will be written. Minidumps are not enabled by default on client versions of Windows
      #
      # An error report file with more information is saved as:
      # C:\workspace\demo\hs_err_pid9476.log
      #
      # If you would like to submit a bug report, please visit:
      # https://github.com/AdoptOpenJDK/openjdk-support/issues
      #

      The error varies somewhat between runs. Sometimes it crashes without printing any output at all.

      You can also find a whole series of hs_err_pid files in the sample project.

      ---------- BEGIN SOURCE ----------
      See the attached zip.
      ---------- END SOURCE ----------

      CUSTOMER SUBMITTED WORKAROUND :
      Avoid making the commandline large. Unfortunately, it doesn't seem like using '@file' arguments really helps though, since large '@file' seems to also cause the problem.

      In our case, the problem can be avoided by creating a temporary jar file to replace all the individual classpath elements with a single jar. Thereby we can make the -classpath argument a short string, and the problem is avoided.

      FREQUENCY : often


        Attachments

          Activity

            People

            • Assignee:
              tongwan Andrew Wang
              Reporter:
              webbuggrp Webbug Group
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: