Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-4109263

HPROF: support needs improvement (misc feature requests)

    Details

    • Type: Enhancement
    • Status: Closed
    • Priority: P5
    • Resolution: Won't Fix
    • Affects Version/s: 1.2.0
    • Fix Version/s: None
    • Component/s: tools
    • Labels:
    • Subcomponent:
    • CPU:
      generic
    • OS:
      generic

      Description

      From achut@nairobi Thu Dec 11 18:40:50 1997
      Return-Path: <achut@nairobi>
      Received: from kindra.eng.sun.com by taller.eng.sun.com (SMI-8.6/SMI-SVR4)
      id SAA04266; Thu, 11 Dec 1997 18:40:50 -0800
      Received: from nairobi.eng.sun.com by kindra.eng.sun.com (SMI-8.6/SMI-SVR4)
      id SAA15161; Thu, 11 Dec 1997 18:40:46 -0800
      Received: from nairobi.eng.sun.com by nairobi.eng.sun.com (SMI-8.6/SMI-SVR4)
      id SAA09934; Thu, 11 Dec 1997 18:40:41 -0800
      Date: Thu, 11 Dec 1997 18:40:41 -0800 (PST)
      From: Achut Reddy <achut@nairobi>
      Reply-To: Achut Reddy <achut@nairobi>
      Subject: Proposal for Memory Profiling format changes
      To: sheng.liang@nairobi, tom.rodriguez@nairobi, anand.palaniswamy@nairobi
      Cc: gwhite@kindra
      Message-ID: <libSDtMail.9712111840.8129.achut@nairobi/nairobi>
      MIME-Version: 1.0
      Content-Type: TEXT/plain; charset=us-ascii
      Content-MD5: 618IcVG494uuiDzivMQ11g==
      X-Mailer: dtmail 1.1.0 CDE Version 1.1_50 SunOS 5.5.1 sun4m sparc
      Content-Length: 6407
      Status: RO
      X-Status: $$$$
      X-UID: 0000002352

      Sheng,

      We have been working on a version of the memory profiler that uses the
      binary format hprof data. One thing we noticed right away is that
      volume of data is much too high for us to be able to handle well. For
      anything beyond simple toy programs, the amount of data was *huge* (for
      one app we tried, over 1 GB of data was generated).

      This amount of data means we cannot even keep up with reading the data
      (a pure reader in Java which simply read the data and threw it away
      would barely keep up; even then, probably only on a multi-cpu machine,
      or two separate machines). Even if we could read it fast enough, there
      is no way we could store 1 GB or more of data in memory. Even if the
      VM and OS allowed it (which it probably won't), the performance of our
      tool would be so slow as to be useless.

      Also, it seems a waste to deal with so much data which our current tool
      does not even make use of in its displays. We considered "reducing"
      the data on the fly, but after some analysis we decided this would only
      help a little bit. In order to make a usable tool, we need a drastic
      reduction in the amount of data being sent by the VM. Below is our
      specific proposal. It is on the order of the amount of data that we
      got with the ASCII version. It also adds some additional info needed
      by our tool.

      Please review this and let us know what you think.

      Proposal for extending -hprof option:
      ------------------------------------

      1. We keep everything we have now, but add a new "short format"
      sub-option to -hprof:

      -hprof:format=short
      -hprof:format=long

      If format=long, everything behaves mostly as it does now (with some
      minor additions, see below).
      If format=short, then emit new record types (described below),
      and only on GC events, and on HPROF_CMD_ALLOCS.
      The default could be either long or short, but probably should
      be short.

      2. Add sub-options to limit amount of data generated:

      -hprof:depth=n

      Limits the max stack depth to n. Default is no limit.

      -hprof:top=n

      Limits the number of entries in the HPROF_ALLOCS and HPROF_ALLOCS_LIVE
      records to the top n (measured in bytes). Default is no limit.

      3. Add a new command:

      HPROF_CMD_ALLOCS emit a HPROF_ALLOCS record (see below)
      *without doing a GC*.

      4. Add the following record types:

      HPROF_ALLOCS_LIVE a set of heap allocations (as opposed to
      HPROF_ALLOC, which is a single allocation)
      aggregated by class/thread/stack combination,
      plus live data.

      [id class name id
      id thread id
      id stack trace id
      u4 number of instances allocated since last
      HPROF_ALLOCS or HPROF_ALLOCS_LIVE record
      for this class/thread/stack id.
      u4 total bytes allocated since last HPROF_ALLOCS
      or HPROF_ALLOCS_LIVE record for this
      class/thread/stack id.
      u4 live instances: total number of instances
      that remain (after GC).
      u4]* live bytes: total number of bytes
      that remain (after GC).

      HPROF_ALLOCS a set of heap allocations (as opposed to
      HPROF_ALLOC, which is a single allocation)
      aggregated by class/thread/stack combination

      [id class name id
      id thread id
      id stack trace id
      u4 number of instances allocated since last
      HPROF_ALLOCS or HPROF_ALLOCS_LIVE record
      for this class/thread/stack id.
      u4]* total bytes allocated since last HPROF_ALLOCS
      or HPROF_ALLOCS_LIVE record for this
      class/thread/stack id.

      HPROF_HEAP_SIZE size of heap changed

      u4 new heap size

      HPROF_HEADER one-time header info, always the first record
      UTF8 command line string (progname + args)
      UTF8 os.name
      UTF8 os.arch
      UTF8 os.version
      UTF8 CLASSPATH value
      u4 initial heap size (-ms)
      u4 max heap size (-mx)


      HPROF_GC garbage collection info
      u4 time to perform GC in microseconds.
      (As much as possible, includes only actual
      GC time, not overhead such as writing these
      records)
      u4 total allocated instances before GC
      u4 total allocated instances after GC
      u4 total allocated bytes before GC
      u4 total allocated bytes after GC

      HPROF_TRACE_FRAMES a Java stack trace of frame ids

      id stack trace ID
      [id]* frame ID

        HPROF_FRAME a Java stack frame

               id frame ID
                      id class name ID
                      id method name ID
                      id method signature ID
                      i4 line number. >0: normal
                                             -1: unknown
                                             -2: compiled method
                                             -3: native method
      (aligned on id boundary)

      These records would behave as follows:

      - One HPROF_HEADER record is always emitted at the beginning,
      regardless of format setting.

      - When the heap size changes emit HPROF_HEAP_SIZE record,
      regardless of format setting.

      - One HPROF_GC record is emitted on each GC, regardless of format
      setting.

      - HPROF_TRACE_FRAMES can be used in place of HPROF_TRACE, to reduce
      the size of the data.

      - If format=short:

      A HPROF_ALLOCS_LIVE record is emitted instead of HPROF_ALLOC and
      HPROF_FREE records, and only on GC.

      This would be similar to the type and amount of data previously
      emitted by the ASCII version, but it would be using the new style
      records and binary format.

      More precisely, on every GC event it would emit, in this order
      *after* the GC has ocurred:

      - one HPROF_ALLOCS_LIVE record containing a list of
      each allocation (aggregated by class/thread/stack
      combination), since the last HPROF_ALLOCS or HPROF_ALLOCS_LIVE
      record. If no allocations have been made since the last
      HPROF_ALLOCS or HPROF_ALLOCS_LIVE record, a "zero" record
      is still emitted.

      - one HPROF_GC record giving time to perform GC and totals.

      Upon receiving a HPROF_CMD_ALLOCS command, it would emit:

      - one HPROF_ALLOCS record containing a list of
      each allocation and free (aggregated by class/thread/stack
      combination) since the last HPROF_ALLOCS or HPROF_ALLOCS_LIVE
      record. If no allocations have been made since the last
      HPROF_ALLOCS or HPROF_ALLOCS_LIVE record, a "zero" record is
      still emitted.

      - if format=long, HPROF_ALLOC and HPROF_FREE records are emitted on
      every allocation (current behavior). A HPROF_GC record is emitted
      on every GC.

      Achut
      Suggestions on improving hprof support in JDK 1.2. See attachments for details.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ohair Kelly Ohair (Inactive)
                Reporter:
                sliangsunw Sheng Liang (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Imported:
                  Indexed: