Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8069004

Kitchensink hanged with 16Gb heap and GC pause >30 min

    Details

    • Type: Bug
    • Status: Closed
    • Priority: P1
    • Resolution: Fixed
    • Affects Version/s: 9
    • Fix Version/s: 9
    • Component/s: hotspot
    • Environment:
    • Subcomponent:
      gc
    • Introduced In Build:
      b42
    • Introduced In Version:
      9
    • Resolved In Build:
      b64
    • Verification:
      Verified

      Backports

        Description

        Kitchensink was considered hanged, but in fact it was too long GC. Heap was 16Gb, and pause was more than 30 minutes, after time limit application was killed.

        run time:16hours

        RULE bigapps/Kitchensink/stability Unknown Server is hang: no server activity noticed.

        ILW=M(can't say if it is hang or just very long pause) L(rest of batch jobs are running ok for 26+hours for the moment ) H(no workaround), => priority=P3


        pstack shows 1 active GC tread
        ----------------- lwp# 7 / thread# 7 --------------------
         ffff80ff3f82b5ef __1cOFreeRegionListLadd_ordered6Mp0_v_ () + 7f
         ffff80ff3f678ea6 __1cQG1ParNoteEndTaskEwork6MI_v_ () + 196
         ffff80ff40025844 __1cKGangWorkerEloop6M_v_ () + a4
         ffff80ff3fd5f5c0 java_start () + 230
         ffff80ffbcc41545 _thrp_setup () + a5
         ffff80ffbcc417f0 _lwp_start ()

        almost all other lwps are running lwp_cond_wait

        gc log ends up with unfinished "[GC cleanup":

        60093.184: #226056: [GC pause (G1 Evacuation Pause) (young) (initial-mark), 0.0504610 secs]
           [Parallel Time: 40.3 ms, GC Workers: 6]
              [GC Worker Start (ms): Min: 60093184.7, Avg: 60093184.8, Max: 60093184.8, Diff: 0.0]
              [Ext Root Scanning (ms): Min: 1.0, Avg: 1.2, Max: 1.2, Diff: 0.2, Sum: 7.0]
              [Update RS (ms): Min: 9.7, Avg: 10.1, Max: 10.7, Diff: 0.9, Sum: 60.3]
                 [Processed Buffers: Min: 8, Avg: 10.2, Max: 12, Diff: 4, Sum: 61]
              [Scan RS (ms): Min: 0.1, Avg: 0.2, Max: 0.4, Diff: 0.3, Sum: 1.4]
              [Code Root Scanning (ms): Min: 0.0, Avg: 0.0, Max: 0.2, Diff: 0.2, Sum: 0.3]
              [Object Copy (ms): Min: 28.0, Avg: 28.5, Max: 28.7, Diff: 0.6, Sum: 170.8]
              [Termination (ms): Min: 0.0, Avg: 0.0, Max: 0.1, Diff: 0.1, Sum: 0.2]
              [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.4]
              [GC Worker Total (ms): Min: 40.0, Avg: 40.1, Max: 40.1, Diff: 0.1, Sum: 240.4]
              [GC Worker End (ms): Min: 60093224.8, Avg: 60093224.8, Max: 60093224.9, Diff: 0.0]
           [Code Root Fixup: 0.1 ms]
           [Code Root Purge: 0.0 ms]
           [Clear CT: 0.8 ms]
           [Other: 9.2 ms]
              [Choose CSet: 0.0 ms]
              [Ref Proc: 6.8 ms]
              [Ref Enq: 0.3 ms]
              [Redirty Cards: 0.2 ms]
              [Humongous Reclaim: 0.0 ms]
              [Free CSet: 1.4 ms]
           [Eden: 3500.0M(3500.0M)->0.0B(3500.0M) Survivors: 108.0M->96.0M Heap: 7307.2M(8292.0M)->3819.0M(8292.0M)]
         [Times: user=0.25 sys=0.00 real=0.05 secs]
        60093.235: #226057: [GC concurrent-root-region-scan-start]
        60093.265: #226057: [GC concurrent-root-region-scan-end, 0.0303055 secs]
        60093.265: #226057: [GC concurrent-mark-start]
        60095.477: #226057: [GC concurrent-mark-end, 2.2117385 secs]
        60095.479: #226057: [GC remark 60095.479: #226057: [Finalize Marking, 0.0004639 secs] 60095.480: #226057: [GC ref-proc, 1.3226148 secs] 60096.803: #226057: [Unloading, 0.0389114 secs] 60096.841: #226057: [GC aggregate-data, 0.0042423 secs], 1.3672145 secs]
         [Times: user=1.54 sys=0.00 real=1.37 secs]
        60096.847: #226057: [GC cleanup

          Attachments

          1. jstack_l.out
            2 kB
          2. pmap.out
            32 kB
          3. pstack.out
            76 kB

            Issue Links

              Activity

                People

                • Assignee:
                  drwhite Derek White
                  Reporter:
                  aaivanov Anton Ivanov (Inactive)
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  9 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: