Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8076052

JEP 284: New HotSpot Build System

    Details

    • Author:
      Magnus Ihse Bursie
    • JEP Type:
      Infrastructure
    • Exposure:
      Open
    • Subcomponent:
    • Scope:
      Implementation
    • Discussion:
      build dash infra dash dev at openjdk dot java dot net
    • Effort:
      L
    • Duration:
      L
    • Alert Status:
       Green
    • JEP Number:
      284

      Description

      Summary

      Rewrite the HotSpot build system using the build-infra framework.

      Goals

      The goal of this project is to replace the current build system with a new, much simplified one, that is based on the build-infra framework. More specifically:

      • Leverage the functionality present in the build-infra framework, to minimize code duplication.
      • Simplify the HotSpot build system to provide a more maintainable code base and to lower the threshold for future improvement.

      Non-Goals

      We expect no specific performance improvement from this change, since the current HotSpot build system is quite heavily tuned for good performance.

      Motivation

      The current HotSpot build system contains lots of duplicated code and redundant functionality that is better handled by the overall build-infra framework. The structure and flow of information is hard to follow, even for experienced developers. This creates a reluctance to fix issues and add new features to the build system.

      The lack of full integration into build-infra is also blocking further improvements in the overall build process, which could possibly enable more performance gains in the overall JDK build process.

      Description

      Building HotSpot is, on a high level, not really different from building any other native component in the JDK. In essence, we need to call the build-infra function SetupNativeCompilation() for libjvm.so. In practice, the HotSpot build has evolved differently from the native libraries in the JDK repo, so there are several adaptations that need to be done.

      Some examples of technical differences:

      • HotSpot uses precompiled headers, which needs to be supported.
      • HotSpot uses its own set of compiler flags, which differs from the ones used in the rest of the product.
      • libjvm.so is possibly built several times for different variants, such as server and client.
      • HotSpot uses several different features that can be enabled or disabled, such as C1 and JVMCI.
      • For historical reasons, HotSpot uses different names for platforms and compilers.

      On top of compiling the libjvm.so native library, there are also other pre-/post-build things that needs to happen, such as:

      • Creating build tools like the adlc compiler.
      • Creating gensrc using adlc and the JVMTI tooling.
      • Dtrace support needs both pre- and post-processing.
      • Additional libraries such as libjsig.so need to be built.

      Testing

      The main testing method will be the same as when the JDK build was converted, that is to use the compare.sh script. By building the product twice, once with the old build system and once with the new, and then running the compare script, any differences in the resulting build will be pointed out.

      Native build artifacts will never achieve a 100% byte-by-byte equality, since technical differences will affect even rebuilds using the same build system. However, any remaining differences as pointed out by the compare script should be accounted for and explained to be within reason. (For instance, such changes could be debug data differences due to different build output directories.)

      If the build system re-creates the same binary, this should in theory be enough testing. For safeguarding, a suitable set of normal product tests will also be run before final integration.

        Issue Links

          Activity

          Hide
          erikj Erik Joelsson added a comment -
          Could you review this?
          Show
          erikj Erik Joelsson added a comment - Could you review this?
          Hide
          iklam Ioi Lam added a comment -
          Is there a goal for supporting a quick hotspot rebuild without involving unnecessary work of other parts of the JDK?

          Currently, if I change one C++ file in HotSpot (e.g., javaClasses.cpp), my hotspot-only rebuild takes about 11 seconds (done inside hotspot/make directory). This rebuilds libjvm.so, and I have some scripts to make the JDK use the new libjvm.so. Admittedly, this involves some black magic.

          In comparison, doing a "make images" in the "real" JDK build takes more than 1 minute.

          Ideally, I think we should switch to a clean build environment, AND to support super-quick rebuild of HotSpot C++ files. My 11 seconds above still includes unnecessary stuff (such as running adlc), so a cleaned up build system should be able to do it in even shorter time.
          Show
          iklam Ioi Lam added a comment - Is there a goal for supporting a quick hotspot rebuild without involving unnecessary work of other parts of the JDK? Currently, if I change one C++ file in HotSpot (e.g., javaClasses.cpp), my hotspot-only rebuild takes about 11 seconds (done inside hotspot/make directory). This rebuilds libjvm.so, and I have some scripts to make the JDK use the new libjvm.so. Admittedly, this involves some black magic. In comparison, doing a "make images" in the "real" JDK build takes more than 1 minute. Ideally, I think we should switch to a clean build environment, AND to support super-quick rebuild of HotSpot C++ files. My 11 seconds above still includes unnecessary stuff (such as running adlc), so a cleaned up build system should be able to do it in even shorter time.
          Hide
          ihse Magnus Ihse Bursie added a comment -
          One of the main goals for the new build system has always been that a rebuild with no changes should finish immediately (for resonable values of "immediately"), and that incremental rebuilds should only rebuild what is actually necessary due to the changes made. This goal applies to the new hotspot build as well.

          Unfortunately, rebuilding images today involves verifying module boundaries, which will take quite a long time, even without changes. This check will be eliminated when Jigsaw is integrated in mainline, and module consistency can be checked at compile time. I believe that is the main cause for your long delay in making images. (But without measurements, I can't know for sure.)
          Show
          ihse Magnus Ihse Bursie added a comment - One of the main goals for the new build system has always been that a rebuild with no changes should finish immediately (for resonable values of "immediately"), and that incremental rebuilds should only rebuild what is actually necessary due to the changes made. This goal applies to the new hotspot build as well. Unfortunately, rebuilding images today involves verifying module boundaries, which will take quite a long time, even without changes. This check will be eliminated when Jigsaw is integrated in mainline, and module consistency can be checked at compile time. I believe that is the main cause for your long delay in making images. (But without measurements, I can't know for sure.)
          Hide
          erikj Erik Joelsson added a comment -
          Please endorse this.
          Show
          erikj Erik Joelsson added a comment - Please endorse this.
          Hide
          acorn Karen Kinnear added a comment -
          First, thank you very much for doing this work.
          I assume you will continue to support all of our build targets. And that you will be getting code reviews from
          all the hotspot teams.
          Also, assuming that the binaries don't exactly match, can you run at least a before-vs-after set of nightly tests
          for ALL the platforms? I would appreciate the runtime set of nightly tests.
          Show
          acorn Karen Kinnear added a comment - First, thank you very much for doing this work. I assume you will continue to support all of our build targets. And that you will be getting code reviews from all the hotspot teams. Also, assuming that the binaries don't exactly match, can you run at least a before-vs-after set of nightly tests for ALL the platforms? I would appreciate the runtime set of nightly tests.

            People

            • Assignee:
              ihse Magnus Ihse Bursie
              Reporter:
              erikj Erik Joelsson
              Owner:
              Erik Joelsson
              Reviewed By:
              Iris Clark, Mikael Vidstedt
              Endorsed By:
              Mikael Vidstedt
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Due:
                Created:
                Updated:
                Resolved:
                Integration Due: