Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-7013634

jvmti resexhausted001 can timeout or fail due to excessive thread creation

    Details

    • Subcomponent:
    • CPU:
      generic
    • OS:
      generic

      Description

      The out of swap space hs_err output looks like:

      #
      # There is insufficient memory for the Java Runtime Environment to continue.
      # Out of swap space to map in thread stack.
      # Possible reasons:
      # The system is out of physical RAM or swap space
      # In 32 bit mode, the process size limit was hit
      # Possible solutions:
      # Reduce memory load on the system
      # Increase physical memory or swap space
      # Check if swap backing store is full
      # Use 64 bit Java on a 64 bit OS
      # Decrease Java heap size (-Xmx/-Xms)
      # Decrease number of Java threads
      # Decrease Java thread stack sizes (-Xss)
      # Set larger code cache with -XX:ReservedCodeCacheSize=
      # This output file may be truncated or incomplete.
      #
      # Out of Memory Error (/tmp/jprt/P1/B/215741.dcubed/source/src/os_cpu/solaris_sparc/vm/os_solaris_sparc.cpp:592), pid=9203, tid=2
      #
      # JRE version: 7.0-b125
      # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.0-b07-internal-201101192157.dcubed.7012505_for_hsx_20-fastdebug mixed mode solaris-sparc )

      --------------- T H R E A D ---------------

      Current thread (0x0000000100130000): JavaThread "main" [_thread_in_vm, id=2, stack(0xffffffff7b000000,0xffffffff7b100000)]

      Stack: [0xffffffff7b000000,0xffffffff7b100000], sp=0xffffffff7b0fe030, free space=1016k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      V [libjvm.so+0x126c7f0]
      [error occurred during error reporting (printing native stack), id 0xe0000000]

      Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
      J java.lang.Thread.start0()V
      J java.lang.Thread.start()V
      J nsk.jvmti.ResourceExhausted.resexhausted001.run([Ljava/lang/String;Ljava/io/PrintStream;)I
      j nsk.jvmti.ResourceExhausted.resexhausted001.main([Ljava/lang/String;)V+9
      v ~StubRoutines::call_stub

      --------------- P R O C E S S ---------------

      Java Threads: ( => current thread )
        0x000000011574b800 JavaThread "fleece" [_thread_in_native, id=35416, stack(0xffffffee0e600000,0xffffffee0e700000)]

      <snip the huge number of "fleece" threads>
      The malloc failure hs_err output looks like:

      #
      # There is insufficient memory for the Java Runtime Environment to continue.
      # Native memory allocation (malloc) failed to allocate 32764 bytes for ChunkPool::allocate
      # Possible reasons:
      # The system is out of physical RAM or swap space
      # In 32 bit mode, the process size limit was hit
      # Possible solutions:
      # Reduce memory load on the system
      # Increase physical memory or swap space
      # Check if swap backing store is full
      # Use 64 bit Java on a 64 bit OS
      # Decrease Java heap size (-Xmx/-Xms)
      # Decrease number of Java threads
      # Decrease Java thread stack sizes (-Xss)
      # Set larger code cache with -XX:ReservedCodeCacheSize=
      # This output file may be truncated or incomplete.
      #
      # Out of Memory Error (/tmp/jprt/P1/B/215741.dcubed/source/src/share/vm/memory/allocation.cpp:211), pid=12781, tid=7
      #
      # JRE version: 7.0-b125
      # Java VM: Java HotSpot(TM) Server VM (20.0-b07-internal-201101192157.dcubed.7012505_for_hsx_20-fastdebug compiled mode solaris-sparc )

      --------------- T H R E A D ---------------

      Current thread (0x00183400): VMThread [stack: 0xd6700000,0xd6780000] [id=7]

      Stack: [0xd6700000,0xd6780000], sp=0xd677f480, free space=509k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      V [libjvm.so+0x11589bc]# Host info: SunOS vm-v440-01 5.10 Generic_139555-08 sun4u sparc SUNW,Sun-Fire-V440
      This bug is for the following VM/NSK test:

          nsk/jvmti/ResourceExhausted/resexhausted001

      The following note documents issues with resexhausted001 that
      were found in 2011.01.19 RT_Baseline nightly testing:

      6 resexhausted001 failed in
         linux-amd64_vm_server_mixed_nsk.jvmti.testlist
             - timeout on machine vm-v20z-2
         windows-amd64_vm_server_comp_nsk.jvmti.testlist
             - timeout on machine jck-amd2
         solaris-sparc_vm_server_comp_nsk.jvmti.testlist
             - malloc failed on machine vm-v440-01
         linux-i586_vm_client_mixed_nsk.jvmti.testlist
             - exit code 139 on machine vm-v20z-34
         windows-amd64_vm_server_mixed_nsk.jvmti.testlist
             - timeout on machine jck-amd2
         solaris-sparcv9_vm_server_mixed_nsk.jvmti.testlist
             - out of swap space on machine soda

         resexhausted001 was previous covered by the following bug:
             6302804 3/1 Hotspot VM dies ungraceful death when C heap is
                         exhausted in various places.

         Coleen used 6302804 to improve the way these types of resource
         exhaustion failures are reported. The two Solaris failures
         above are good examples of the new diagnostic output.

         Coleen, Nicolay and I are discussing the resexhausted00[1-4]
         tests and are working out the type of bug to file to cover
         the failure modes that we are observing.

      Here are the URLs for the failing configs:

      timeout on machine vm-v20z-2:

      http://sqeweb.sfbay/net/sqenfs-2/export2/results/vm/gtee/JDK7/NIGHTLY/VM/2011-01-19/RT_Baseline/vm/linux-amd64/server/mixed/linux-amd64_vm_server_mixed_nsk.jvmti.testlist/analysis.html

      timeout on machine jck-amd2:

      http://sqeweb.sfbay/net/sqenfs-2/export2/results/vm/gtee/JDK7/NIGHTLY/VM/2011-01-19/RT_Baseline/vm/windows-amd64/server/comp/windows-amd64_vm_server_comp_nsk.jvmti.testlist/analysis.html

      malloc failed on machine vm-v440-01:

      http://sqeweb.sfbay/net/sqenfs-2/export2/results/vm/gtee/JDK7/NIGHTLY/VM/2011-01-19/RT_Baseline/vm/solaris-sparc/server/comp/solaris-sparc_vm_server_comp_nsk.jvmti.testlist/analysis.html

      exit code 139 on machine vm-v20z-34:

      http://sqeweb.sfbay/net/sqenfs-2/export2/results/vm/gtee/JDK7/NIGHTLY/VM/2011-01-19/RT_Baseline/vm/linux-i586/client/mixed/linux-i586_vm_client_mixed_nsk.jvmti.testlist/analysis.html

      timeout on machine jck-amd2

      http://sqeweb.sfbay/net/sqenfs-2/export2/results/vm/gtee/JDK7/NIGHTLY/VM/2011-01-19/RT_Baseline/vm/windows-amd64/server/mixed/windows-amd64_vm_server_mixed_nsk.jvmti.testlist/analysis.html

      out of swap space on machine soda:

      http://sqeweb.sfbay/net/sqenfs-2/export2/results/vm/gtee/JDK7/NIGHTLY/VM/2011-01-19/RT_Baseline/vm/solaris-sparcv9/server/mixed/solaris-sparcv9_vm_server_mixed_nsk.jvmti.testlist/analysis.html
      The following VM/NSK test also failed in the 2011.01.19 RT_Baseline
      nightly with a malloc failure:

          nsk/jvmti/ResourceExhausted/resexhausted004

      This particular failure didn't stand out in my initial analysis
      because it is on the known fail_list for a different bug (6606767).

      Here is the URL for the failing config:

      http://sqeweb.sfbay/net/sqenfs-2/export2/results/vm/gtee/JDK7/NIGHTLY/VM/2011-01-19/RT_Baseline/vm/solaris-sparc/server/comp/solaris-sparc_vm_server_comp_nsk.jvmti.testlist/analysis.html

      And here is the failing hs_err output:

      #
      # There is insufficient memory for the Java Runtime Environment to continue.
      # Native memory allocation (malloc) failed to allocate 32764 bytes for ChunkPool::allocate
      # Possible reasons:
      # The system is out of physical RAM or swap space
      # In 32 bit mode, the process size limit was hit
      # Possible solutions:
      # Reduce memory load on the system
      # Increase physical memory or swap space
      # Check if swap backing store is full
      # Use 64 bit Java on a 64 bit OS
      # Decrease Java heap size (-Xmx/-Xms)
      # Decrease number of Java threads
      # Decrease Java thread stack sizes (-Xss)
      # Set larger code cache with -XX:ReservedCodeCacheSize=
      # This output file may be truncated or incomplete.
      #
      # Out of Memory Error (/tmp/jprt/P1/B/215741.dcubed/source/src/share/vm/memory/allocation.cpp:211), pid=13230, tid=7
      #
      # JRE version: 7.0-b125
      # Java VM: Java HotSpot(TM) Server VM (20.0-b07-internal-201101192157.dcubed.7012505_for_hsx_20-fastdebug compiled mode solaris-sparc )

      --------------- T H R E A D ---------------

      Current thread (0x00183400): VMThread [stack: 0xf8900000,0xf8980000] [id=7]

      Stack: [0xf8900000,0xf8980000], sp=0xf897ee80, free space=507k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      V [libjvm.so+0x11589bc]# Host info: SunOS vm-v440-01 5.10 Generic_139555-08 sun4u sparc SUNW,Sun-Fire-V440
      This bug is primarily used to track malloc failures due to excessive thread creation not general malloc failures. The new output tells you not to file a bug for those because it is generally a system problem that is the cause. The case with excessive thread creation should be handled so it doesn't fail with malloc failures if possible.

      Timeouts may not be able to be handled, only error return from thread create. On some systems, the os tries to wait for memory to be available or tries to swap out other processes if there's sufficient swap space (ie I can't get this to fail on some systems).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                cjplummer Chris Plummer
                Reporter:
                dcubed Daniel Daugherty
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Imported:
                  Indexed: