Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8164293

HotSpot leaking memory in long-running requests

    Details

    • Subcomponent:
    • Resolved In Build:
      b01
    • CPU:
      x86_64
    • OS:
      linux_ubuntu

      Backports

        Description

        FULL PRODUCT VERSION :
        java version "1.8.0_101"
        Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
        Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)

        FULL OS VERSION :
        Ubuntu 14.04
        Linux andrew-VirtualBox 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

        EXTRA RELEVANT SYSTEM CONFIGURATION :
        Ubuntu 14.04 Guest running in VirtualBox on OSX 10.11.5 host.
        No other changes to standard Ubuntu configuration.

        A DESCRIPTION OF THE PROBLEM :
        When running a web application for a long period of time, and that application makes a large number of long-runing requests (> 1s), it will begin to leak memory.

        Looking at the application with JConsole, all the following pools are limited, and none are growing:

        * Heap Space
        * Meta Space
        * Code Cache
        * Compressed Class Space

        As such, I'm not entirely sure where all the extra memory is going. Looking at the application with JCmd and NMT, I see that over time, the Class/Malloc area grows. That is, given the following partial output from jcmd <pid> VM.native_memory summary:

        - Class (reserved=23075KB, committed=15651KB)
                                    (classes #2182)
                                    (malloc=547KB #6210)
                                    (mmap: reserved=22528KB, committed=15104KB)

        The "classes" number remains static, the "mmap" reserved & committed numbers remain static, but the "malloc" number grows over time, and does not decrease.

        I have also observed that, after setting MALLOC_ARENAS_MAX to 2, when running on a system with glibc >= 2.16, that the malloc arenas seem to fragment over time, and that, after a couple of days, the amount of native heap space reported by pmap grows. For the sample application I have provided below, I have seen the *native* (*not* Java!) heap space grow to over 200MB used. This seems... incorrect.

        THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: No

        THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes

        REGRESSION. Last worked in version 7u80

        STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
        Check out the sample project from GitHub onto a machine with JRE 1.8.0_101.
        cd into nacroleptic
        Run "mvn clean package"
        Run java -jar target/narcoleptic.jar
        cd into ../devourer
        Run "mvn clean package"
        Run ./run.sh

        Download Apache JMeter 3.0 from http://jmeter.apache.org/download_jmeter.cgi
        Extract the archive and start with the "bin/jmeter.sh" script.

        Open the file "load_test.jmx" with JMeter.

        Start the JMeter tests with CTRL+R (CMD+R on OSX)

        Leave for *at least* 30 minutes, then come back and observe the amount of memory used by the process. It will have increased beyond what would be expected. Leave for several hours, maybe a couple of days. Memory usage will continue to increase.

        EXPECTED VERSUS ACTUAL BEHAVIOR :
        Expected behaviour: Memory usage of the application remains stable.

        Actual behaviour: Memory usage of the application increases over time, without stopping. It may slow, and *appear* to stop, but given a large enough time span, it will *alwasy* continue to increase.
        ERROR MESSAGES/STACK TRACES THAT OCCUR :
        N/A - application does not crash unless placed inside a cgroup (for example) with a hard memory limit.

        REPRODUCIBILITY :
        This bug can be reproduced always.

        ---------- BEGIN SOURCE ----------
        Please find the source code at: https://github.com/ipsi/memory-testing

        This is the smallest test case I have been able to find, and I've been looking quite hard.
        ---------- END SOURCE ----------

        CUSTOMER SUBMITTED WORKAROUND :
        Revert to Java 7
        Disable HotSpot completely with -Xint
        Set the MALLOC_ARENAS_MAX environment variable to 2, which slows the memory growth, but does not halt it.

          Issue Links

            Activity

            Hide
            jcm Jamsheed C M added a comment - - edited
             this is jdk8 only issue, as sweeper implementation has gone considerable changes in jdk9.
            bug id :https://bugs.openjdk.java.net/browse/JDK-8046809
            Show
            jcm Jamsheed C M added a comment - - edited  this is jdk8 only issue, as sweeper implementation has gone considerable changes in jdk9. bug id : https://bugs.openjdk.java.net/browse/JDK-8046809
            Hide
            jcm Jamsheed C M added a comment -
            after sweeper memory leak fix, i observe one class module leak. yet to figure out what is causing it.
            Show
            jcm Jamsheed C M added a comment - after sweeper memory leak fix, i observe one class module leak. yet to figure out what is causing it.
            Hide
            jcm Jamsheed C M added a comment - - edited
            i strongly assume there is no leak in mtClass, only steady growth of OopMapCache. This should stabilize after a while. waiting for sometime to confirm this.
            Show
            jcm Jamsheed C M added a comment - - edited i strongly assume there is no leak in mtClass, only steady growth of OopMapCache. This should stabilize after a while. waiting for sometime to confirm this.
            Hide
            hgupdate HG Updates added a comment -
            URL: http://hg.openjdk.java.net/jdk8u/jdk8u-dev/hotspot/rev/8dfbb002197a
            User: thartmann
            Date: 2017-01-11 14:37:06 +0000
            Show
            hgupdate HG Updates added a comment - URL: http://hg.openjdk.java.net/jdk8u/jdk8u-dev/hotspot/rev/8dfbb002197a User: thartmann Date: 2017-01-11 14:37:06 +0000
            Hide
            hgupdate HG Updates added a comment -
            URL: http://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/rev/8dfbb002197a
            User: robm
            Date: 2017-01-20 23:38:54 +0000
            Show
            hgupdate HG Updates added a comment - URL: http://hg.openjdk.java.net/jdk8u/jdk8u/hotspot/rev/8dfbb002197a User: robm Date: 2017-01-20 23:38:54 +0000

              People

              • Assignee:
                jcm Jamsheed C M
                Reporter:
                webbuggrp Webbug Group
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: