Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-6857566

(bf) DirectByteBuffer garbage creation can outpace reclamation

    Details

    • Subcomponent:
    • Resolved In Build:
      b04
    • CPU:
      x86
    • OS:
      linux, windows_7
    • Verification:
      Verified

      Backports

        Description

        FULL PRODUCT VERSION :
        1.6.0_12-b04, 1.6.0_14-ea-b03 & 1.7.0-ea-b44

        A DESCRIPTION OF THE PROBLEM :
        Before throwing an OutOfMemory error for direct memory (allocated using ByteBuffer.allocateDirect), I've noticed that JVM does a full GC to reclaim the direct memory. However, if I have multiple threads trying to allocate direct memory, I get a "java.lang.OutOfMemoryError: Direct buffer memory," even though I know that the garbage collection should have freed up enough direct memory.

        In the test case I've attached I have 2 threads which loop, creating 1MB direct byte buffers and then immediately dereferencing them. So I should need a maximum of 2MB of direct memory. However with a 32 MB direct memory size I still get an OutOfMemory error.

        When I run the same test case with only 1 thread it passes.

        I suspect that maybe the direct memory is being freed up in a java thread, rather than during the garbage collection. I noticed that the JVM creates a DirectByteBuffer$Deallocator object for each of these buffers, which is run by a sun.misc.Cleaner, which I think is run in the reference handler thread.

        STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
        Run "java -Xmx32m -XX:MaxDirectMemorySize=32m test.DirectBufferTest"

        ACTUAL -
        Thread Test thread 0 got an OOM on iteration 223
        java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:633)
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:95)
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
        at test.DirectBufferTest$1.run(DirectBufferTest.java:24)

        REPRODUCIBILITY :
        This bug can be reproduced always.

        ---------- BEGIN SOURCE ----------
        package test;

        import java.nio.ByteBuffer;

        public class DirectBufferTest {
          private static int NUM_THREADS = 2;
          
          public static void main(String[] args) throws InterruptedException {
            for(int i = 0; i < NUM_THREADS; i++) {
              Thread thread = new Thread("Test thread " + i) {
                public void run() {
                  int i = 0;
                  try {
                    while(true) {
                      ByteBuffer bb = ByteBuffer.allocateDirect(1024 * 1024);
                      i++;
                    }
                  } catch(Throwable t) {
                    System.err.println("Thread " + Thread.currentThread().getName() + " got an OOM on iteration " + i);
                    t.printStackTrace();
                    System.exit(1);
                  }
                }
              };
              thread.start();
            }
            
            Thread.sleep(60 * 1000);
            System.out.println("No errors after 60 seconds.");
            System.exit(0);
          }
        }
        ---------- END SOURCE ----------

          Issue Links

            Activity

            Hide
            plevart Peter Levart added a comment -
            Here's the updated patch which incorporates review suggestions from Aleksey Shipilev and Alan Bateman:

            http://cr.openjdk.java.net/~plevart/jdk8-tl/DirectBufferAlloc/webrev.02/

            Compared to previous patch it incorporates cosmetic changes to code and the test now runs as a jtreg test for at most 5 seconds. It can be run with arguments to increase running time and/or threads to stress-test the code.

            I have not been able to push the torture tests mentioned by Leonid above to fail on my machine (Linux i7 4-core CPU) even without the patch. Perhaps some special parameters are needed to make them fail with OOME?
            Show
            plevart Peter Levart added a comment - Here's the updated patch which incorporates review suggestions from Aleksey Shipilev and Alan Bateman: http://cr.openjdk.java.net/~plevart/jdk8-tl/DirectBufferAlloc/webrev.02/ Compared to previous patch it incorporates cosmetic changes to code and the test now runs as a jtreg test for at most 5 seconds. It can be run with arguments to increase running time and/or threads to stress-test the code. I have not been able to push the torture tests mentioned by Leonid above to fail on my machine (Linux i7 4-core CPU) even without the patch. Perhaps some special parameters are needed to make them fail with OOME?
            Hide
            ppunegov Pavel Punegov added a comment -
            RULE gc/memory/Nio Exception gc.memory.Nio.Nio$Fault: Nop, OOM is unexpected again: java.lang.OutOfMemoryError: Direct buffer memory

            OpenJDK RI 8b126
            http://aurora.ru.oracle.com/functional/faces/RunDetails.xhtml?names=355022.VMSQE.promotion.openjdk-25
            Show
            ppunegov Pavel Punegov added a comment - RULE gc/memory/Nio Exception gc.memory.Nio.Nio$Fault: Nop, OOM is unexpected again: java.lang.OutOfMemoryError: Direct buffer memory OpenJDK RI 8b126 http://aurora.ru.oracle.com/functional/faces/RunDetails.xhtml?names=355022.VMSQE.promotion.openjdk-25
            Hide
            plevart Peter Levart added a comment - - edited
            Here's the patch, rebased to current jdk9/dev tip:

            http://cr.openjdk.java.net/~plevart/jdk9-dev/DirectBufferAlloc/webrev.01/

            ...Request for Review follows shortly...
            Show
            plevart Peter Levart added a comment - - edited Here's the patch, rebased to current jdk9/dev tip: http://cr.openjdk.java.net/~plevart/jdk9-dev/DirectBufferAlloc/webrev.01/ ...Request for Review follows shortly...
            Hide
            hgupdate HG Updates added a comment -
            URL: http://hg.openjdk.java.net/jdk9/dev/jdk/rev/9934d34ed3c0
            User: plevart
            Date: 2014-02-24 14:38:37 +0000
            Show
            hgupdate HG Updates added a comment - URL: http://hg.openjdk.java.net/jdk9/dev/jdk/rev/9934d34ed3c0 User: plevart Date: 2014-02-24 14:38:37 +0000
            Hide
            hgupdate HG Updates added a comment -
            URL: http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/9934d34ed3c0
            User: lana
            Date: 2014-03-01 00:14:01 +0000
            Show
            hgupdate HG Updates added a comment - URL: http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/9934d34ed3c0 User: lana Date: 2014-03-01 00:14:01 +0000

              People

              • Assignee:
                plevart Peter Levart
                Reporter:
                ndcosta Nelson Dcosta
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Imported:
                  Indexed: