Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-4469299

(bf) Excessive native memory growth with NIO due to finalization delay

    Details

    • Subcomponent:
    • CPU:
      generic, x86, sparc
    • OS:
      generic, linux, solaris_8, solaris_10

      Backports

        Description

        The java.nio package added DirectByteBuffer and MappedByteBuffer classes
        that allocate native memory on the native C heap and via a platform specific
        memory mapping function (mmap()). The current implementation provides no
        api to allow Java applications to deallocate these memory regions, but
        instead relys upon finalizers to perform the deallocation. In some cases,
        particularly with applications with large heaps and light to moderate
        loads where collections happen infrequently, the Java process can consume
        memory to the point of process address space exhaustion.

        This problem is not unique to nio, as any native resources that rely upon
        finalizers for cleanup can also exhibit similar issues. However, nio exposes
        this issue in new ways.

        This problem is observable by running a nio file copy program that uses
        a small (8K) MappedByteBuffer to repeatedly copy a large (~7m) file within the
        same VM with a 50m heap. 100 iterations result in the process size growing
        from 84m to 564m, with pmap reporting large numbers, >60K , of 8K
        mapped pages. After the only minor GC, finialization kicks in and reduces
        the number of mapped pages.

           java -server -Xms50m -Xmx50m CopyFile 7 100 <src_file> <dest_file> 8192

        This problem is also reproducible with the same copy program using
        ByteBuffer objects. With the same heap, buffer, and file sizes, the native
        C heap grows to nearly 2G before a minor GC event occurs. On Solaris,
        the C heap size does not retract. Note the collection takes 9.6s and
        collects 4.4K of Java heap space. The large native heap is the result
        of nio allocating a new DirectByteBuffer for each read and write call
        that uses a heap allocated ByteBuffer, creating many DirectByteBuffer
        objects.

           java -server -Xms50m -Xmx50m 3 100 <src_file> <dst_file> 8192

        In a more realistic scenario, the nio HttpServer also exhibits native
        C heap growth when subjected to moderate loads with a 96mb Eden and 256mb
        heap. While ramping up load to 100 request/s, the process size grows to
        437mb (248mb RSS, 142mb native C heap size) before the first minor GC
        (5 min after start of load). Growth eventually returns as load ramps up.

        Although these examples may seem dubious, they do expose some real problems.
        Some communication between the native allocation mechanisms and the Java
        garbage collectors are needed to force a GC of the Java heap when native
        resources exceed some threshold, possibly set by a hueristic or tunable.
        Otherwise, java.nio will need to provide deallocation api's so native
        resources can be freed long before finalization.

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  bpb Brian Burkhalter
                  Reporter:
                  briand Brian Doherty (Inactive)
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  6 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:
                    Imported:
                    Indexed: