Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8164293

HotSpot leaking memory in long-running requests

    Details

    • Subcomponent:
    • Resolved In Build:
      b01
    • CPU:
      x86_64
    • OS:
      linux_ubuntu

      Description

      FULL PRODUCT VERSION :
      java version "1.8.0_101"
      Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
      Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)

      FULL OS VERSION :
      Ubuntu 14.04
      Linux andrew-VirtualBox 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

      EXTRA RELEVANT SYSTEM CONFIGURATION :
      Ubuntu 14.04 Guest running in VirtualBox on OSX 10.11.5 host.
      No other changes to standard Ubuntu configuration.

      A DESCRIPTION OF THE PROBLEM :
      When running a web application for a long period of time, and that application makes a large number of long-runing requests (> 1s), it will begin to leak memory.

      Looking at the application with JConsole, all the following pools are limited, and none are growing:

      * Heap Space
      * Meta Space
      * Code Cache
      * Compressed Class Space

      As such, I'm not entirely sure where all the extra memory is going. Looking at the application with JCmd and NMT, I see that over time, the Class/Malloc area grows. That is, given the following partial output from jcmd <pid> VM.native_memory summary:

      - Class (reserved=23075KB, committed=15651KB)
                                  (classes #2182)
                                  (malloc=547KB #6210)
                                  (mmap: reserved=22528KB, committed=15104KB)

      The "classes" number remains static, the "mmap" reserved & committed numbers remain static, but the "malloc" number grows over time, and does not decrease.

      I have also observed that, after setting MALLOC_ARENAS_MAX to 2, when running on a system with glibc >= 2.16, that the malloc arenas seem to fragment over time, and that, after a couple of days, the amount of native heap space reported by pmap grows. For the sample application I have provided below, I have seen the *native* (*not* Java!) heap space grow to over 200MB used. This seems... incorrect.

      THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: No

      THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes

      REGRESSION. Last worked in version 7u80

      STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
      Check out the sample project from GitHub onto a machine with JRE 1.8.0_101.
      cd into nacroleptic
      Run "mvn clean package"
      Run java -jar target/narcoleptic.jar
      cd into ../devourer
      Run "mvn clean package"
      Run ./run.sh

      Download Apache JMeter 3.0 from http://jmeter.apache.org/download_jmeter.cgi
      Extract the archive and start with the "bin/jmeter.sh" script.

      Open the file "load_test.jmx" with JMeter.

      Start the JMeter tests with CTRL+R (CMD+R on OSX)

      Leave for *at least* 30 minutes, then come back and observe the amount of memory used by the process. It will have increased beyond what would be expected. Leave for several hours, maybe a couple of days. Memory usage will continue to increase.

      EXPECTED VERSUS ACTUAL BEHAVIOR :
      Expected behaviour: Memory usage of the application remains stable.

      Actual behaviour: Memory usage of the application increases over time, without stopping. It may slow, and *appear* to stop, but given a large enough time span, it will *alwasy* continue to increase.
      ERROR MESSAGES/STACK TRACES THAT OCCUR :
      N/A - application does not crash unless placed inside a cgroup (for example) with a hard memory limit.

      REPRODUCIBILITY :
      This bug can be reproduced always.

      ---------- BEGIN SOURCE ----------
      Please find the source code at: https://github.com/ipsi/memory-testing

      This is the smallest test case I have been able to find, and I've been looking quite hard.
      ---------- END SOURCE ----------

      CUSTOMER SUBMITTED WORKAROUND :
      Revert to Java 7
      Disable HotSpot completely with -Xint
      Set the MALLOC_ARENAS_MAX environment variable to 2, which slows the memory growth, but does not halt it.

        Issue Links

          Activity

          Hide
          jcm Jamsheed C M added a comment - - edited
           this is jdk8 only issue, as sweeper implementation has gone considerable changes in jdk9.
          bug id :https://bugs.openjdk.java.net/browse/JDK-8046809
          Show
          jcm Jamsheed C M added a comment - - edited  this is jdk8 only issue, as sweeper implementation has gone considerable changes in jdk9. bug id : https://bugs.openjdk.java.net/browse/JDK-8046809
          Hide
          jcm Jamsheed C M added a comment -
          after sweeper memory leak fix, i observe one class module leak. yet to figure out what is causing it.
          Show
          jcm Jamsheed C M added a comment - after sweeper memory leak fix, i observe one class module leak. yet to figure out what is causing it.
          Hide
          jcm Jamsheed C M added a comment - - edited
          i strongly assume there is no leak in mtClass, only steady growth of OopMapCache. This should stabilize after a while. waiting for sometime to confirm this.
          Show
          jcm Jamsheed C M added a comment - - edited i strongly assume there is no leak in mtClass, only steady growth of OopMapCache. This should stabilize after a while. waiting for sometime to confirm this.
          Hide
          hgupdate HG Updates added a comment -
          URL: http://hg.openjdk.java.net/jdk8u/jdk8u-dev/hotspot/rev/8dfbb002197a
          User: thartmann
          Date: 2017-01-11 14:37:06 +0000
          Show
          hgupdate HG Updates added a comment - URL: http://hg.openjdk.java.net/jdk8u/jdk8u-dev/hotspot/rev/8dfbb002197a User: thartmann Date: 2017-01-11 14:37:06 +0000
          Hide
          hgupdate HG Updates added a comment -
          URL: http://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/rev/8dfbb002197a
          User: robm
          Date: 2017-01-20 23:38:54 +0000
          Show
          hgupdate HG Updates added a comment - URL: http://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/rev/8dfbb002197a User: robm Date: 2017-01-20 23:38:54 +0000

            People

            • Assignee:
              jcm Jamsheed C M
              Reporter:
              webbuggrp Webbug Group
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: