Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8164293

HotSpot leaking memory in long-running requests

    Details

    • Subcomponent:
    • Resolved In Build:
      b01
    • CPU:
      x86_64
    • OS:
      linux_ubuntu

      Backports

        Description

        FULL PRODUCT VERSION :
        java version "1.8.0_101"
        Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
        Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)

        FULL OS VERSION :
        Ubuntu 14.04
        Linux andrew-VirtualBox 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

        EXTRA RELEVANT SYSTEM CONFIGURATION :
        Ubuntu 14.04 Guest running in VirtualBox on OSX 10.11.5 host.
        No other changes to standard Ubuntu configuration.

        A DESCRIPTION OF THE PROBLEM :
        When running a web application for a long period of time, and that application makes a large number of long-runing requests (> 1s), it will begin to leak memory.

        Looking at the application with JConsole, all the following pools are limited, and none are growing:

        * Heap Space
        * Meta Space
        * Code Cache
        * Compressed Class Space

        As such, I'm not entirely sure where all the extra memory is going. Looking at the application with JCmd and NMT, I see that over time, the Class/Malloc area grows. That is, given the following partial output from jcmd <pid> VM.native_memory summary:

        - Class (reserved=23075KB, committed=15651KB)
                                    (classes #2182)
                                    (malloc=547KB #6210)
                                    (mmap: reserved=22528KB, committed=15104KB)

        The "classes" number remains static, the "mmap" reserved & committed numbers remain static, but the "malloc" number grows over time, and does not decrease.

        I have also observed that, after setting MALLOC_ARENAS_MAX to 2, when running on a system with glibc >= 2.16, that the malloc arenas seem to fragment over time, and that, after a couple of days, the amount of native heap space reported by pmap grows. For the sample application I have provided below, I have seen the *native* (*not* Java!) heap space grow to over 200MB used. This seems... incorrect.

        THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: No

        THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes

        REGRESSION. Last worked in version 7u80

        STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
        Check out the sample project from GitHub onto a machine with JRE 1.8.0_101.
        cd into nacroleptic
        Run "mvn clean package"
        Run java -jar target/narcoleptic.jar
        cd into ../devourer
        Run "mvn clean package"
        Run ./run.sh

        Download Apache JMeter 3.0 from http://jmeter.apache.org/download_jmeter.cgi
        Extract the archive and start with the "bin/jmeter.sh" script.

        Open the file "load_test.jmx" with JMeter.

        Start the JMeter tests with CTRL+R (CMD+R on OSX)

        Leave for *at least* 30 minutes, then come back and observe the amount of memory used by the process. It will have increased beyond what would be expected. Leave for several hours, maybe a couple of days. Memory usage will continue to increase.

        EXPECTED VERSUS ACTUAL BEHAVIOR :
        Expected behaviour: Memory usage of the application remains stable.

        Actual behaviour: Memory usage of the application increases over time, without stopping. It may slow, and *appear* to stop, but given a large enough time span, it will *alwasy* continue to increase.
        ERROR MESSAGES/STACK TRACES THAT OCCUR :
        N/A - application does not crash unless placed inside a cgroup (for example) with a hard memory limit.

        REPRODUCIBILITY :
        This bug can be reproduced always.

        ---------- BEGIN SOURCE ----------
        Please find the source code at: https://github.com/ipsi/memory-testing

        This is the smallest test case I have been able to find, and I've been looking quite hard.
        ---------- END SOURCE ----------

        CUSTOMER SUBMITTED WORKAROUND :
        Revert to Java 7
        Disable HotSpot completely with -Xint
        Set the MALLOC_ARENAS_MAX environment variable to 2, which slows the memory growth, but does not halt it.

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  jcm Jamsheed C M
                  Reporter:
                  webbuggrp Webbug Group
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  8 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: