Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-6916527

14.9% s10u9_01 specjvm08_64bit performance regression x86

    Details

    • Type: Bug
    • Status: Closed
    • Priority: P2
    • Resolution: Won't Fix
    • Affects Version/s: solaris_10u9
    • Fix Version/s: None
    • Component/s: performance
    • Labels:
    • Subcomponent:
    • CPU:
      x86
    • OS:
      solaris_10

      Description

      14.9% regression in specjvm08_64bit in build s10u9_01
      specjvm08_64bit :

      --------------------------------------------------------------------------------
      Historical specjvm08_64bit results on oaf315_64
      --------------------------------------------------------------------------------
      Build Result Status Description Related Bugs
      --------------------------------------------------------------------------------
      s10u9_01 14.08 RED 14.9% regression
      s10u8_08a 16.54 GREEN unchanged
      s10u8_08
      s10u8_07 16.5 GREEN 1.8% improvement
      s10u8_06
      s10u8_05
      s10u8_04
      s10u8_03
      s10u8_02
      s10u8_01 16.21 RED 1.1% regression
      s10u7_08 16.39 GREEN unchanged
      s10u7_07 ~ YELLOW Comparison not statistically significant
      s10u7_06 ~ YELLOW Comparison not statistically significant
      s10u7_05 16.39 GREEN unchanged
      s10u7_04 16.38 GREEN unchanged
      s10u7_03 16.36 GREEN unchanged
      s10u7_02 16.38 GREEN unchanged
      s10u7_01 16.3 GREEN unchanged
      s10u6_07b 16.3 GREEN unchanged
      s10u6_07a 16.34 GREEN unchanged
      s10u6_07
      s10u6_06
      s10u6_05 16.27 GREEN unchanged
      s10u6_04 ~ YELLOW Comparison not statistically significant
      s10u6_03 16.3 First Result

      Using s10u8_08a as baseline
      9 baselines. Average is 16.539 PctStDevis 0.60%
      Plot_mean: 14.08 , overall change: 14.87 %
      Iteration Result Change Colour
      1 14.08 -14.87 % RED
      2 14.089 -14.81 % RED
      3 14.08 -14.87 % RED
      4 14.109 -14.69 % RED
      5 14.039 -15.11 % RED
      6 14.06 -14.99 % RED
      Number of runs: 6
      With all numbers shown, overall change will be -14.89%
      Previous build:s10u8_08a 16.544 0.03 % YELLOW
      The found results are: 1 YELLOW, 6 RED including the old result
      Std. Deviation as % from Mean is 0.17
      TTest against baseline: 0.0000000000

      Outliers
      =========
      Can't find the machine in /etc/bootparams.
      Benchmark res_range: 10:-1 (not ship)
      Driver Version changes...
        -scsi_vhci SCSI VHCI Driver 1.57 (1.57 -> 1.60)
        -sata SATA Module v1.25 (1.25 -> 1.26)
        -sd SCSI Disk Driver 1.547 (1.547 -> 1.550)
        -consconfig console configuration 1.35 (1.35 -> 1.36)
        -consconfig_dacf Consconfig DACF 1.36 (1.36 -> 1.37)
        -mpt MPT HBA Driver v1.92 (1.92 -> 1.96)
        -st SCSI tape Driver 1.255 (1.255 -> 1.256)


      Detailed statistical comparison of results is available at:
      http://perfwww.ireland/cgi-bin/detail.cgi?rig=oaf315_64&benchmark=specjvm08_64bit&build=s10u9_01&base=s10u8_08a&clean=FALSE

      Using s10_74L2a as baseline
      No Baselines
      Plot_mean: 14.08 , overall change: 100.00 %
      Warning: plot_per is 0.00, but overall change is 100.00. Will update later.
      Iteration Result Change Colour
      1 14.08
      2 14.089
      3 14.08
      4 14.109
      5 14.039
      6 14.06
      Number of runs: 6
      With all numbers shown, overall change will be 0.00%
      Previous build:s10u8_08a 16.544
      Std. Deviation as % from Mean is 0.17
      TTest against baseline: 1.0000000000

      Outliers
      =========
      Can't find the machine in /etc/bootparams.
      Benchmark res_range: 10:-1 (not ship)

      Detailed statistical comparison of results is available at:
      http://perfwww.ireland/cgi-bin/detail.cgi?rig=oaf315_64&benchmark=specjvm08_64bit&build=s10u9_01&base=s10_74L2a&clean=FALSE

      http://perfwww.ireland/cgi-bin/publictools/shipII.cgi?&BUILD=s10u9&BNAMES=specjvm08_64bit&tSysteM=oaf315_64

      The machine is a :Ultra 40 (64bit) 1 x 2010MHz 2048 MB ram
      The rerun is back and confirms the regression:

      /results9/s10u9_01-6916527-rerun.oaf315_64/specjvm08_64bitUsing s10u8_08a-6916527-rerun as baseline
      16 baselines. Average is 16.53 PctStDevis 0.00%
      Plot_mean: 14.06 , overall change: 14.94 %
      Iteration Result Change Colour
      1 14.13 -14.52 % RED
      2 14.06 -14.94 % RED
      3 14.009 -15.25 % RED
      4 14.089 -14.76 % RED
      5 14 -15.31 % RED
      6 14.07 -14.88 % RED
      Number of runs: 6
      With all numbers shown, overall change will be -14.94%
      No usable previous results for this benchmark
      The found results are: 6 RED
      Std. Deviation as % from Mean is 0.35
      TTest against baseline: 0.0000000000

      Outliers
      =========
      1 high outliers in base
      Can't find the machine in /etc/bootparams.
      Benchmark res_range: 10:-1 (not ship)
      No usable previous results available for driver comparison

      Detailed statistical comparison of results is available at:
      http://perfwww.ireland/cgi-bin/detail.cgi?rig=oaf315_64&benchmark=specjvm08_64bit&build=s10u9_01-6916527-rerun&base=s10u8_08a-6916527-rerun&clean=FALSE
      The generic gate did not track the regression. There are no feature gate putback, scheduling a driver swap.
      The regression has been introduced via the Java changes in s10u9_01.

      mean change% num stddev% build JVM
      16.54 0.03% 9 s10u8_08a 1.5.0_20-b02 mixed mode
      14.08 -14.89% 6 0.17 s10u9_01 1.5.0_23-b01 mixed mode
      14.11 -14.69% 3 0.14 s10u8_08a-6916527-java-s10u9_01 1.5.0_23-b01 mixed mode

      Baseline used: s10u8_08a

      The s10u8_08a-6916527-java-s10u9_01 run here is a fresh install of s10u8_08a with these packages
      from s10u9_01: SUNWj6rt SUNWj6rtx SUNWj6cfg SUNWj5rt SUNWj5rtx SUNWj5cfg
      As the results above show switching from 1.5.0_20-b02 to 1.5.0_23-b01 on s10u8_08a introduces the
      regression we are seeing in s10u9_01.

      s10u8_08a -> s10u8_08a with 1.5.0_23-b01

      Using s10u8_08a as baseline
      9 baselines. Average is 16.539 PctStDevis 0.60%
      Plot_mean: 14.11 , overall change: 14.69 %
      Iteration Result Change Colour
      1 14.109 -14.69 % RED
      2 14.089 -14.81 % RED
      3 14.13 -14.57 % RED
      Number of runs: 3
      With all numbers shown, overall change will be -14.69%
      No usable previous results for this benchmark
      The found results are: 3 RED
      Std. Deviation as % from Mean is 0.14
      TTest against baseline: 0.0000000000

      Outliers
      =========
      Can't find the machine in /etc/bootparams.
      Benchmark res_range: 10:-1 (not ship)
      No usable previous results available for driver comparison

      Detailed statistical comparison of results is available at:
      http://perfwww.ireland/cgi-bin/detail.cgi?rig=oaf315_64&benchmark=specjvm08_64bit&build=s10u8_08a-6916527-java-s10u9_01&base=s10u8_08a&clean=FALSE


      The rig seeing this regression is:
      Ultra 40 (64bit):oaf315_64:1 x 2010MHz:2048 MB:Nvidia Quadro NVS 285

      The regession was in specjvm08_64bit which can be found:
      http://perfwww.ireland/Benchmarks/Java/specjvm08.tar.gz

      Explorer:
      http://perfwww.ireland/cgi-bin/internaltools/machineStats.cgi?s10u9_01:oaf315_64

      Statis before regression:
      http://perfwww.ireland/Results9/s10u8_08a.oaf315_64/specjvm08_64bit/8/statit.bin.txt
      http://perfwww.ireland/Results9/s10u8_08a.oaf315_64/specjvm08_64bit/2/statit.bin.txt
      http://perfwww.ireland/Results9/s10u8_08a.oaf315_64/specjvm08_64bit/5/statit.bin.txt

      Statits after regression:
      http://perfwww.ireland/Results9/s10u8_08a-6916527-java-s10u9_01.oaf315_64/specjvm08_64bit/1/statit.bin.txt
      http://perfwww.ireland/Results9/s10u8_08a-6916527-java-s10u9_01.oaf315_64/specjvm08_64bit/2/statit.bin.txt
      http://perfwww.ireland/Results9/s10u8_08a-6916527-java-s10u9_01.oaf315_64/specjvm08_64bit/3/statit.bin.txt

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                coffeys Sean Coffey
                Reporter:
                gmurdoch Greg Murdoch (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Imported:
                  Indexed: