Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8167368

JEP 296: Consolidate the JDK Forest into a Single Repository

    Details

    • Author:
      Joseph D. Darcy
    • JEP Type:
      Infrastructure
    • Exposure:
      Open
    • Subcomponent:
    • Scope:
      Implementation
    • Discussion:
      jdk9 dash dev at openjdk dot java dot net
    • Effort:
      M
    • Duration:
      M
    • JEP Number:
      296

      Description

      Summary

      Combine the numerous repositories of the JDK forest into a single repository in order to simplify and streamline development.

      Non-Goals

      Adding the FX sources to the JDK forest is not part of the proposal.

      Motivation

      For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9 there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

      While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable source-code management operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the JDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs, since some engineers use separate bug ids to push to different repos.

      This mismatch between the divisions of the Mercurial repos and unity of the engineering dilutes one of the main benefits of modern source-code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools such as Mercurial bisect.

      The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds such as the "get source" script.

      Description

      To address these issues, a prototype of a consolidated forest has been developed. The prototype is available at:

      http://hg.openjdk.java.net/jdk10/consol-proto/

      Some of the supporting conversion scripts used to create the prototype are attached as unify.zip.

      In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per-file level, with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved.

      The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the JDK forest there are module-based directories like

      $ROOT/jdk/src/java.base
      ...
      $ROOT/langtools/src/java.compiler
      ...

      In the consolidated forest, this code is instead organized as

      $ROOT/src/java.base
      $ROOT/src/java.compiler
      ...

      As a consequence, from the root of the repository the relative path of a source file in a module is preserved after the consolidation and src directory combination.

      An analogous but less aggressive reorganization is done for the test directories to go from

      $ROOT/jdk/test/Foo.java
      $ROOT/langtools/test/Bar.java

      to

      $ROOT/test/jdk/Foo.java
      $ROOT/test/langtools/Bar.java

      Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

      While the regression tests will run with the current state of the prototype, further consolidations of the jtreg configuration files are possible and may be done in the future.

      Alternatives

      One alternative is to simply stay with the current set of repositories. The history of some or all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

      Testing

      To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

      Risks and Assumptions

      The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be finished before the consolidations is put into production. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward- and back- ports as opposed to exporting and importing changesets.

        Issue Links

          Activity

          darcy Joe Darcy created issue -
          darcy Joe Darcy made changes -
          Field Original Value New Value
          Description Summary
          -------
          An OpenJDK forest contains numerous repositories. To simply and streamline development, those numerous repositories should be combined into a single repository for JDK 10.

          Goals
          -----

          // What are the goals of this proposal? Omit this section if you have
          // nothing to say beyond what's already in the summary.

          Non-Goals
          ---------
          Combining the FX sources to the JDK forest is not part of the project.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9, on the OpenJDK side there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable SCM operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the OpenJDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs as some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Hg repos and unity of the engineering dilutes one of the main benefits of modern source code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools like Hg bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds like the "get source" script.

          Description
          -----------

          To address these issues, a prototype of a consolidated forest has been developed. The eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per file level with the consolidated forest being synchronized at the tags used to mark JDK promotions.
          The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the OpenJDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated OpenJDK forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          An analogous but less aggressive reorganization is planned (but not yet fully implemented) for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Supporting updates to the jtreg configuration files are in progress (JDK-8165187).

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. For example, there are still some files left in vestigial $ROOT/jdk, $ROOT/langtools, etc. directories, but these stray files should be relocated as part of follow-up work. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          Alternatives
          ------------

          // Did you consider any alternative approaches or technologies? If so
          // then please describe them here and explain why they were not chosen.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------

          // Describe any risks or assumptions that must be considered along with
          // this proposal. Could any plausible events derail this work, or even
          // render it unnecessary? If you have mitigation plans for the known
          // risks then please describe them.

          Dependencies
          -----------

          // Describe all dependencies that this JEP has on other JEPs, JBS issues,
          // components, products, or anything else. Dependencies upon JEPs or JBS
          // issues should also be recorded as links in the JEP issue itself.
          //
          // Describe any JEPs that depend upon this JEP, and likewise make sure
          // they are linked to this issue in JBS.
          darcy Joe Darcy made changes -
          Fix Version/s 10 [ 16302 ]
          darcy Joe Darcy made changes -
          Description Summary
          -------
          An OpenJDK forest contains numerous repositories. To simply and streamline development, those numerous repositories should be combined into a single repository for JDK 10.

          Goals
          -----

          // What are the goals of this proposal? Omit this section if you have
          // nothing to say beyond what's already in the summary.

          Non-Goals
          ---------
          Combining the FX sources to the JDK forest is not part of the project.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9, on the OpenJDK side there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable SCM operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the OpenJDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs as some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Hg repos and unity of the engineering dilutes one of the main benefits of modern source code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools like Hg bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds like the "get source" script.

          Description
          -----------

          To address these issues, a prototype of a consolidated forest has been developed. The eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per file level with the consolidated forest being synchronized at the tags used to mark JDK promotions.
          The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the OpenJDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated OpenJDK forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          An analogous but less aggressive reorganization is planned (but not yet fully implemented) for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Supporting updates to the jtreg configuration files are in progress (JDK-8165187).

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. For example, there are still some files left in vestigial $ROOT/jdk, $ROOT/langtools, etc. directories, but these stray files should be relocated as part of follow-up work. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          Alternatives
          ------------

          // Did you consider any alternative approaches or technologies? If so
          // then please describe them here and explain why they were not chosen.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------

          // Describe any risks or assumptions that must be considered along with
          // this proposal. Could any plausible events derail this work, or even
          // render it unnecessary? If you have mitigation plans for the known
          // risks then please describe them.

          Dependencies
          -----------

          // Describe all dependencies that this JEP has on other JEPs, JBS issues,
          // components, products, or anything else. Dependencies upon JEPs or JBS
          // issues should also be recorded as links in the JEP issue itself.
          //
          // Describe any JEPs that depend upon this JEP, and likewise make sure
          // they are linked to this issue in JBS.
          Summary
          -------
          An OpenJDK forest contains numerous repositories. To simply and streamline development, those numerous repositories should be combined into a single repository for JDK 10.

          Non-Goals
          ---------
          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9, on the OpenJDK side there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable SCM operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the OpenJDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs as some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Hg repos and unity of the engineering dilutes one of the main benefits of modern source code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools like Hg bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds like the "get source" script.

          Description
          -----------
          To address these issues, a prototype of a consolidated forest has been developed. The prototype and supporting scripts plan to be published in the near future.

          In the prototype. fhe eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per file level with the consolidated forest being synchronized at the tags used to mark JDK promotions. The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the OpenJDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated OpenJDK forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          An analogous but less aggressive reorganization is planned (but not yet fully implemented) for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. For example, there are still some files left in vestigial $ROOT/jdk, $ROOT/langtools, etc. directories, but these stray files should be relocated as part of follow-up work. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          Supporting updates to the jtreg configuration files are in progress (JDK-8165187).

          Alternatives
          ------------
          One alternative is to simply stay with the current set of repositories. The history of some of all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------
          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be ready by the time we want to open JDK 10 for development.
          darcy Joe Darcy made changes -
          Link This issue relates to JDK-8165178 [ JDK-8165178 ]
          darcy Joe Darcy made changes -
          Link This issue is blocked by JDK-8167373 [ JDK-8167373 ]
          darcy Joe Darcy made changes -
          Link This issue is blocked by JDK-8167374 [ JDK-8167374 ]
          darcy Joe Darcy made changes -
          Link This issue is blocked by JDK-8167375 [ JDK-8167375 ]
          darcy Joe Darcy made changes -
          Status Draft [ 10001 ] Submitted [ 10002 ]
          darcy Joe Darcy made changes -
          Description Summary
          -------
          An OpenJDK forest contains numerous repositories. To simply and streamline development, those numerous repositories should be combined into a single repository for JDK 10.

          Non-Goals
          ---------
          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9, on the OpenJDK side there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable SCM operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the OpenJDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs as some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Hg repos and unity of the engineering dilutes one of the main benefits of modern source code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools like Hg bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds like the "get source" script.

          Description
          -----------
          To address these issues, a prototype of a consolidated forest has been developed. The prototype and supporting scripts plan to be published in the near future.

          In the prototype. fhe eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per file level with the consolidated forest being synchronized at the tags used to mark JDK promotions. The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the OpenJDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated OpenJDK forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          An analogous but less aggressive reorganization is planned (but not yet fully implemented) for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. For example, there are still some files left in vestigial $ROOT/jdk, $ROOT/langtools, etc. directories, but these stray files should be relocated as part of follow-up work. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          Supporting updates to the jtreg configuration files are in progress (JDK-8165187).

          Alternatives
          ------------
          One alternative is to simply stay with the current set of repositories. The history of some of all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------
          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be ready by the time we want to open JDK 10 for development.
          Summary
          -------
          An OpenJDK forest contains numerous repositories. To simply and streamline development, those numerous repositories should be combined into a single repository for JDK 10.

          Non-Goals
          ---------
          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9, on the OpenJDK side there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable SCM operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the OpenJDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs as some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Hg repos and unity of the engineering dilutes one of the main benefits of modern source code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools like Hg bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds like the "get source" script.

          Description
          -----------
          To address these issues, a prototype of a consolidated forest has been developed. The prototype and supporting conversion scripts are planned to be published in the near future.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per file level with the consolidated forest being synchronized at the tags used to mark JDK promotions. The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the OpenJDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated OpenJDK forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          An analogous but less aggressive reorganization is planned (but not yet fully implemented) for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. For example, there are still some files left in vestigial $ROOT/jdk, $ROOT/langtools, etc. directories, but these stray files should be relocated as part of follow-up work. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          Supporting updates to the jtreg configuration files are in progress (JDK-8165187).

          Alternatives
          ------------
          One alternative is to simply stay with the current set of repositories. The history of some of all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------
          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be ready by the time we want to open JDK 10 for development. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward and back ports as opposed to exporting changesets.
          briangoetz Brian Goetz made changes -
          Reviewed By Brian Goetz [ briangoetz ]
          mikael Mikael Vidstedt made changes -
          Reviewed By Brian Goetz [ briangoetz ] Brian Goetz, Mikael Vidstedt [ briangoetz, mikael ]
          Hide
          ehelin Erik Helin added a comment -
          Attached the scripts used for consolidating a forest.
          Show
          ehelin Erik Helin added a comment - Attached the scripts used for consolidating a forest.
          ehelin Erik Helin made changes -
          Attachment unify.zip [ 63891 ]
          darcy Joe Darcy made changes -
          Description Summary
          -------
          An OpenJDK forest contains numerous repositories. To simply and streamline development, those numerous repositories should be combined into a single repository for JDK 10.

          Non-Goals
          ---------
          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9, on the OpenJDK side there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable SCM operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the OpenJDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs as some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Hg repos and unity of the engineering dilutes one of the main benefits of modern source code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools like Hg bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds like the "get source" script.

          Description
          -----------
          To address these issues, a prototype of a consolidated forest has been developed. The prototype and supporting conversion scripts are planned to be published in the near future.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per file level with the consolidated forest being synchronized at the tags used to mark JDK promotions. The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the OpenJDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated OpenJDK forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          An analogous but less aggressive reorganization is planned (but not yet fully implemented) for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. For example, there are still some files left in vestigial $ROOT/jdk, $ROOT/langtools, etc. directories, but these stray files should be relocated as part of follow-up work. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          Supporting updates to the jtreg configuration files are in progress (JDK-8165187).

          Alternatives
          ------------
          One alternative is to simply stay with the current set of repositories. The history of some of all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------
          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be ready by the time we want to open JDK 10 for development. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward and back ports as opposed to exporting changesets.
          Summary
          -------
          An OpenJDK forest contains numerous repositories. To simply and streamline development, those numerous repositories should be combined into a single repository for JDK 10.

          Non-Goals
          ---------
          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9, on the OpenJDK side there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable SCM operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the OpenJDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs as some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Hg repos and unity of the engineering dilutes one of the main benefits of modern source code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools like Hg bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds like the "get source" script.

          Description
          -----------
          To address these issues, a prototype of a consolidated forest has been developed. The prototype is planned to be published in the near future; the supporting conversion scripts are attached as unify.zip.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per file level with the consolidated forest being synchronized at the tags used to mark JDK promotions. The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the OpenJDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated OpenJDK forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          An analogous but less aggressive reorganization is planned (but not yet fully implemented) for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. For example, there are still some files left in vestigial $ROOT/jdk, $ROOT/langtools, etc. directories, but these stray files should be relocated as part of follow-up work. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          Supporting updates to the jtreg configuration files are in progress (JDK-8165187).

          Alternatives
          ------------
          One alternative is to simply stay with the current set of repositories. The history of some of all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------
          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be ready by the time we want to open JDK 10 for development. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward and back ports as opposed to exporting changesets.
          darcy Joe Darcy made changes -
          Description Summary
          -------
          An OpenJDK forest contains numerous repositories. To simply and streamline development, those numerous repositories should be combined into a single repository for JDK 10.

          Non-Goals
          ---------
          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9, on the OpenJDK side there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable SCM operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the OpenJDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs as some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Hg repos and unity of the engineering dilutes one of the main benefits of modern source code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools like Hg bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds like the "get source" script.

          Description
          -----------
          To address these issues, a prototype of a consolidated forest has been developed. The prototype is planned to be published in the near future; the supporting conversion scripts are attached as unify.zip.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per file level with the consolidated forest being synchronized at the tags used to mark JDK promotions. The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the OpenJDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated OpenJDK forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          An analogous but less aggressive reorganization is planned (but not yet fully implemented) for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. For example, there are still some files left in vestigial $ROOT/jdk, $ROOT/langtools, etc. directories, but these stray files should be relocated as part of follow-up work. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          Supporting updates to the jtreg configuration files are in progress (JDK-8165187).

          Alternatives
          ------------
          One alternative is to simply stay with the current set of repositories. The history of some of all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------
          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be ready by the time we want to open JDK 10 for development. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward and back ports as opposed to exporting changesets.
          Summary
          -------
          An OpenJDK forest contains numerous repositories. To simply and streamline development, those numerous repositories should be combined into a single repository for JDK 10.

          Non-Goals
          ---------
          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9, on the OpenJDK side there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable SCM operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the OpenJDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs as some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Hg repos and unity of the engineering dilutes one of the main benefits of modern source code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools like Hg bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds like the "get source" script.

          Description
          -----------
          To address these issues, a prototype of a consolidated forest has been developed. The prototype is planned to be published in the near future; the supporting conversion scripts are attached as unify.zip.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per file level with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved. The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the OpenJDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated OpenJDK forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          An analogous but less aggressive reorganization is planned (but not yet fully implemented) for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. For example, there are still some files left in vestigial $ROOT/jdk, $ROOT/langtools, etc. directories, but these stray files should be relocated as part of follow-up work. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          Supporting updates to the jtreg configuration files are in progress (JDK-8165187).

          Alternatives
          ------------
          One alternative is to simply stay with the current set of repositories. The history of some of all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------
          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be ready by the time we want to open JDK 10 for development. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward and back ports as opposed to exporting changesets.
          agarciar Aurelio Garcia-Ribeyro made changes -
          Link This issue is blocked by JDK-8167463 [ JDK-8167463 ]
          darcy Joe Darcy made changes -
          Description Summary
          -------
          An OpenJDK forest contains numerous repositories. To simply and streamline development, those numerous repositories should be combined into a single repository for JDK 10.

          Non-Goals
          ---------
          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9, on the OpenJDK side there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable SCM operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the OpenJDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs as some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Hg repos and unity of the engineering dilutes one of the main benefits of modern source code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools like Hg bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds like the "get source" script.

          Description
          -----------
          To address these issues, a prototype of a consolidated forest has been developed. The prototype is planned to be published in the near future; the supporting conversion scripts are attached as unify.zip.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per file level with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved. The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the OpenJDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated OpenJDK forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          An analogous but less aggressive reorganization is planned (but not yet fully implemented) for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. For example, there are still some files left in vestigial $ROOT/jdk, $ROOT/langtools, etc. directories, but these stray files should be relocated as part of follow-up work. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          Supporting updates to the jtreg configuration files are in progress (JDK-8165187).

          Alternatives
          ------------
          One alternative is to simply stay with the current set of repositories. The history of some of all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------
          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be ready by the time we want to open JDK 10 for development. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward and back ports as opposed to exporting changesets.
          Summary
          -------
          An OpenJDK forest contains numerous repositories. To simply and streamline development, those numerous repositories should be combined into a single repository for JDK 10.

          Non-Goals
          ---------
          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9, on the OpenJDK side there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable SCM operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the OpenJDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs as some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Hg repos and unity of the engineering dilutes one of the main benefits of modern source code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools like Hg bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds like the "get source" script.

          Description
          -----------
          To address these issues, a prototype of a consolidated forest has been developed. The prototype is planned to be published in the near future; the supporting conversion scripts are attached as unify.zip.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per file level with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved. The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the OpenJDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated OpenJDK forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          An analogous but less aggressive reorganization is planned (but not yet fully implemented) for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. For example, there are still some files left in vestigial $ROOT/jdk, $ROOT/langtools, etc. directories, but these stray files should be relocated as part of follow-up work. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          Supporting updates to the jtreg configuration files are in progress (JDK-8165187).

          Alternatives
          ------------
          One alternative is to simply stay with the current set of repositories. The history of some of all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------
          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be ready by the time we want to open JDK 10 for development. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward- and back- ports as opposed to exporting and importing changesets.
          darcy Joe Darcy made changes -
          Security Confidential [ 10000 ]
          darcy Joe Darcy made changes -
          Description Summary
          -------
          An OpenJDK forest contains numerous repositories. To simply and streamline development, those numerous repositories should be combined into a single repository for JDK 10.

          Non-Goals
          ---------
          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9, on the OpenJDK side there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable SCM operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the OpenJDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs as some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Hg repos and unity of the engineering dilutes one of the main benefits of modern source code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools like Hg bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds like the "get source" script.

          Description
          -----------
          To address these issues, a prototype of a consolidated forest has been developed. The prototype is planned to be published in the near future; the supporting conversion scripts are attached as unify.zip.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per file level with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved. The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the OpenJDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated OpenJDK forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          An analogous but less aggressive reorganization is planned (but not yet fully implemented) for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. For example, there are still some files left in vestigial $ROOT/jdk, $ROOT/langtools, etc. directories, but these stray files should be relocated as part of follow-up work. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          Supporting updates to the jtreg configuration files are in progress (JDK-8165187).

          Alternatives
          ------------
          One alternative is to simply stay with the current set of repositories. The history of some of all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------
          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be ready by the time we want to open JDK 10 for development. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward- and back- ports as opposed to exporting and importing changesets.
          Summary
          -------
          An OpenJDK forest contains numerous repositories. To simply and streamline development, those numerous repositories should be combined into a single repository for JDK 10.

          Non-Goals
          ---------
          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9, on the OpenJDK side there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable SCM operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the OpenJDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs as some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Hg repos and unity of the engineering dilutes one of the main benefits of modern source code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools like Hg bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds like the "get source" script.

          Description
          -----------
          To address these issues, a prototype of a consolidated forest has been developed. The prototype is available at:



          Some of the supporting conversion scripts used to create the prototype are attached as unify.zip.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per file level with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved. The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the OpenJDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated OpenJDK forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          An analogous but less aggressive reorganization is planned (but not yet fully implemented) for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. For example, there are still some files left in vestigial $ROOT/jdk, $ROOT/langtools, etc. directories, but these stray files should be relocated as part of follow-up work. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          Supporting updates to the jtreg configuration files are in progress (JDK-8165187).

          Alternatives
          ------------
          One alternative is to simply stay with the current set of repositories. The history of some of all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------
          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be ready by the time we want to open JDK 10 for development. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward- and back- ports as opposed to exporting and importing changesets.
          darcy Joe Darcy made changes -
          Description Summary
          -------
          An OpenJDK forest contains numerous repositories. To simply and streamline development, those numerous repositories should be combined into a single repository for JDK 10.

          Non-Goals
          ---------
          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9, on the OpenJDK side there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable SCM operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the OpenJDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs as some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Hg repos and unity of the engineering dilutes one of the main benefits of modern source code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools like Hg bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds like the "get source" script.

          Description
          -----------
          To address these issues, a prototype of a consolidated forest has been developed. The prototype is available at:



          Some of the supporting conversion scripts used to create the prototype are attached as unify.zip.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per file level with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved. The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the OpenJDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated OpenJDK forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          An analogous but less aggressive reorganization is planned (but not yet fully implemented) for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. For example, there are still some files left in vestigial $ROOT/jdk, $ROOT/langtools, etc. directories, but these stray files should be relocated as part of follow-up work. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          Supporting updates to the jtreg configuration files are in progress (JDK-8165187).

          Alternatives
          ------------
          One alternative is to simply stay with the current set of repositories. The history of some of all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------
          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be ready by the time we want to open JDK 10 for development. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward- and back- ports as opposed to exporting and importing changesets.
          Summary
          -------
          An OpenJDK forest contains numerous repositories. To simply and streamline development, those numerous repositories should be combined into a single repository for JDK 10.

          Non-Goals
          ---------
          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9, on the OpenJDK side there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable SCM operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the OpenJDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs as some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Hg repos and unity of the engineering dilutes one of the main benefits of modern source code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools like Hg bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds like the "get source" script.

          Description
          -----------
          To address these issues, a prototype of a consolidated forest has been developed. The prototype is available at:

              http://hg.openjdk.java.net/jdk9/consol-proto/

          Some of the supporting conversion scripts used to create the prototype are attached as unify.zip.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per file level with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved. The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the OpenJDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated OpenJDK forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          An analogous but less aggressive reorganization is planned (but not yet fully implemented) for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. For example, there are still some files left in vestigial $ROOT/jdk, $ROOT/langtools, etc. directories, but these stray files should be relocated as part of follow-up work. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          Supporting updates to the jtreg configuration files are in progress (JDK-8165187).

          Alternatives
          ------------
          One alternative is to simply stay with the current set of repositories. The history of some of all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------
          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be ready by the time we want to open JDK 10 for development. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward- and back- ports as opposed to exporting and importing changesets.
          darcy Joe Darcy made changes -
          Description Summary
          -------
          An OpenJDK forest contains numerous repositories. To simply and streamline development, those numerous repositories should be combined into a single repository for JDK 10.

          Non-Goals
          ---------
          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9, on the OpenJDK side there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable SCM operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the OpenJDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs as some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Hg repos and unity of the engineering dilutes one of the main benefits of modern source code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools like Hg bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds like the "get source" script.

          Description
          -----------
          To address these issues, a prototype of a consolidated forest has been developed. The prototype is available at:

              http://hg.openjdk.java.net/jdk9/consol-proto/

          Some of the supporting conversion scripts used to create the prototype are attached as unify.zip.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per file level with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved. The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the OpenJDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated OpenJDK forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          An analogous but less aggressive reorganization is planned (but not yet fully implemented) for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. For example, there are still some files left in vestigial $ROOT/jdk, $ROOT/langtools, etc. directories, but these stray files should be relocated as part of follow-up work. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          Supporting updates to the jtreg configuration files are in progress (JDK-8165187).

          Alternatives
          ------------
          One alternative is to simply stay with the current set of repositories. The history of some of all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------
          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be ready by the time we want to open JDK 10 for development. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward- and back- ports as opposed to exporting and importing changesets.
          Summary
          -------
          An OpenJDK forest contains numerous repositories. To simplify and streamline development, those numerous repositories should be combined into a single repository for JDK 10.

          Non-Goals
          ---------
          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9, on the OpenJDK side there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable SCM operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the OpenJDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs as some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Hg repos and unity of the engineering dilutes one of the main benefits of modern source code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools like Hg bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds like the "get source" script.

          Description
          -----------
          To address these issues, a prototype of a consolidated forest has been developed. The prototype is available at:

              http://hg.openjdk.java.net/jdk9/consol-proto/

          Some of the supporting conversion scripts used to create the prototype are attached as unify.zip.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per file level with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved. The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the OpenJDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated OpenJDK forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          An analogous but less aggressive reorganization is planned (but not yet fully implemented) for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. For example, there are still some files left in vestigial $ROOT/jdk, $ROOT/langtools, etc. directories, but these stray files should be relocated as part of follow-up work. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          Supporting updates to the jtreg configuration files are in progress (JDK-8165187).

          Alternatives
          ------------
          One alternative is to simply stay with the current set of repositories. The history of some of all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------
          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be ready by the time we want to open JDK 10 for development. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward- and back- ports as opposed to exporting and importing changesets.
          mr Mark Reinhold made changes -
          Description Summary
          -------
          An OpenJDK forest contains numerous repositories. To simplify and streamline development, those numerous repositories should be combined into a single repository for JDK 10.

          Non-Goals
          ---------
          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9, on the OpenJDK side there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable SCM operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the OpenJDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs as some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Hg repos and unity of the engineering dilutes one of the main benefits of modern source code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools like Hg bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds like the "get source" script.

          Description
          -----------
          To address these issues, a prototype of a consolidated forest has been developed. The prototype is available at:

              http://hg.openjdk.java.net/jdk9/consol-proto/

          Some of the supporting conversion scripts used to create the prototype are attached as unify.zip.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per file level with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved. The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the OpenJDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated OpenJDK forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          An analogous but less aggressive reorganization is planned (but not yet fully implemented) for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. For example, there are still some files left in vestigial $ROOT/jdk, $ROOT/langtools, etc. directories, but these stray files should be relocated as part of follow-up work. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          Supporting updates to the jtreg configuration files are in progress (JDK-8165187).

          Alternatives
          ------------
          One alternative is to simply stay with the current set of repositories. The history of some of all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------
          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be ready by the time we want to open JDK 10 for development. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward- and back- ports as opposed to exporting and importing changesets.
          Summary
          -------

          Combine the numerous repositories of the JDK forest into a single repository in order to simplify and streamline development.

          Non-Goals
          ---------

          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9 there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable source-code management operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the JDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs, since some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Mercurial repos and unity of the engineering dilutes one of the main benefits of modern source-code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools such as Mercurial bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds such as the "get source" script.

          Description
          -----------

          To address these issues, a prototype of a consolidated forest has been developed. The prototype is available at:

              http://hg.openjdk.java.net/jdk9/consol-proto/

          Some of the supporting conversion scripts used to create the prototype are attached as `unify.zip`.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per-file level, with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved.

          The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the JDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          An analogous but less aggressive reorganization is planned (but not yet fully implemented) for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. For example, there are still some files left in vestigial `$ROOT/jdk`, `$ROOT/langtools`, etc., directories, but these stray files should be relocated as part of follow-up work. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          Supporting updates to the jtreg configuration files are in progress ([JDK-8165187](JDK-8165187)).

          Alternatives
          ------------

          One alternative is to simply stay with the current set of repositories. The history of some or all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------

          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be ready by the time we want to open JDK 10 for development. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward- and back- ports as opposed to exporting and importing changesets.
          mr Mark Reinhold made changes -
          Status Submitted [ 10002 ] Candidate [ 10003 ]
          JEP Number 296
          mr Mark Reinhold made changes -
          Summary Consolidate JDK 10 OpenJDK repositories to a single repository JEP 296: Consolidate the JDK Forest into a Single Repository
          mr Mark Reinhold made changes -
          darcy Joe Darcy made changes -
          Description Summary
          -------

          Combine the numerous repositories of the JDK forest into a single repository in order to simplify and streamline development.

          Non-Goals
          ---------

          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9 there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable source-code management operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the JDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs, since some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Mercurial repos and unity of the engineering dilutes one of the main benefits of modern source-code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools such as Mercurial bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds such as the "get source" script.

          Description
          -----------

          To address these issues, a prototype of a consolidated forest has been developed. The prototype is available at:

              http://hg.openjdk.java.net/jdk9/consol-proto/

          Some of the supporting conversion scripts used to create the prototype are attached as `unify.zip`.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per-file level, with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved.

          The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the JDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          An analogous but less aggressive reorganization is planned (but not yet fully implemented) for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. For example, there are still some files left in vestigial `$ROOT/jdk`, `$ROOT/langtools`, etc., directories, but these stray files should be relocated as part of follow-up work. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          Supporting updates to the jtreg configuration files are in progress ([JDK-8165187](JDK-8165187)).

          Alternatives
          ------------

          One alternative is to simply stay with the current set of repositories. The history of some or all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------

          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be ready by the time we want to open JDK 10 for development. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward- and back- ports as opposed to exporting and importing changesets.
          Summary
          -------

          Combine the numerous repositories of the JDK forest into a single repository in order to simplify and streamline development.

          Non-Goals
          ---------

          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9 there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable source-code management operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the JDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs, since some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Mercurial repos and unity of the engineering dilutes one of the main benefits of modern source-code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools such as Mercurial bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds such as the "get source" script.

          Description
          -----------

          To address these issues, a prototype of a consolidated forest has been developed. The prototype is available at:

              http://hg.openjdk.java.net/jdk9/consol-proto/

          Some of the supporting conversion scripts used to create the prototype are attached as `unify.zip`.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per-file level, with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved.

          The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the JDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          As a consequence, from the root of the repository the relative path of a source file in a module preserved after the consolidation and src directory combination.

          An analogous but less aggressive reorganization is planned (but not yet fully implemented) for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. For example, there are still some files left in vestigial `$ROOT/jdk`, `$ROOT/langtools`, etc., directories, but these stray files should be relocated as part of follow-up work. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          Supporting updates to the jtreg configuration files are in progress ([JDK-8165187](JDK-8165187)).

          Alternatives
          ------------

          One alternative is to simply stay with the current set of repositories. The history of some or all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------

          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be ready by the time we want to open JDK 10 for development. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward- and back- ports as opposed to exporting and importing changesets.
          Hide
          simonis Volker Simonis added a comment -
          This is a relatively big change so I wonder if it was considered to not only consolidate the JDK forest but instead change the version control system from Mercurial to Git. I understand that such a change is even more disruptive and it would require a lot of infrastructure work on Oracle side paired with the support of two VCSs for quite some time. Nevertheless Git is nowadays the standard open-source, distributed revision control system. It has a lot more traction and support compared to Mercurial and it may simply solve some of the currently visible problems like for example performance for big repos.

          I think there will be probably no better opportunity for such a change in the foreseeable future. I know Joe has posted on the discussion thread for this JEP that "a git migration is well outside the scope of this project" but I wonder if such a change has already been discussed or evaluated?

          As a matter of fact, with Graal at least one major OpenJDK project is already using Git as version control system (https://github.com/graalvm/graal-core) so there must be already some experience and evidence about the pros and cons of the two systems.
          Show
          simonis Volker Simonis added a comment - This is a relatively big change so I wonder if it was considered to not only consolidate the JDK forest but instead change the version control system from Mercurial to Git. I understand that such a change is even more disruptive and it would require a lot of infrastructure work on Oracle side paired with the support of two VCSs for quite some time. Nevertheless Git is nowadays the standard open-source, distributed revision control system. It has a lot more traction and support compared to Mercurial and it may simply solve some of the currently visible problems like for example performance for big repos. I think there will be probably no better opportunity for such a change in the foreseeable future. I know Joe has posted on the discussion thread for this JEP that "a git migration is well outside the scope of this project" but I wonder if such a change has already been discussed or evaluated? As a matter of fact, with Graal at least one major OpenJDK project is already using Git as version control system ( https://github.com/graalvm/graal-core ) so there must be already some experience and evidence about the pros and cons of the two systems.
          Hide
          ihse Magnus Ihse Bursie added a comment -
          [~simonis], any kind of transition to git would be effectively hampered by the current forest structure. If, at any future point, we should consider doing a transition to git, having a single repo will be a prerequisite. Trust me, the consolidation's gonna be disruptive enough, so that you wouldn't want to change all tooling at the same time as well.
          Show
          ihse Magnus Ihse Bursie added a comment - [~simonis], any kind of transition to git would be effectively hampered by the current forest structure. If, at any future point, we should consider doing a transition to git, having a single repo will be a prerequisite. Trust me, the consolidation's gonna be disruptive enough, so that you wouldn't want to change all tooling at the same time as well.
          darcy Joe Darcy made changes -
          Description Summary
          -------

          Combine the numerous repositories of the JDK forest into a single repository in order to simplify and streamline development.

          Non-Goals
          ---------

          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9 there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable source-code management operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the JDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs, since some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Mercurial repos and unity of the engineering dilutes one of the main benefits of modern source-code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools such as Mercurial bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds such as the "get source" script.

          Description
          -----------

          To address these issues, a prototype of a consolidated forest has been developed. The prototype is available at:

              http://hg.openjdk.java.net/jdk9/consol-proto/

          Some of the supporting conversion scripts used to create the prototype are attached as `unify.zip`.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per-file level, with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved.

          The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the JDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          As a consequence, from the root of the repository the relative path of a source file in a module preserved after the consolidation and src directory combination.

          An analogous but less aggressive reorganization is planned (but not yet fully implemented) for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. For example, there are still some files left in vestigial `$ROOT/jdk`, `$ROOT/langtools`, etc., directories, but these stray files should be relocated as part of follow-up work. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          Supporting updates to the jtreg configuration files are in progress ([JDK-8165187](JDK-8165187)).

          Alternatives
          ------------

          One alternative is to simply stay with the current set of repositories. The history of some or all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------

          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be ready by the time we want to open JDK 10 for development. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward- and back- ports as opposed to exporting and importing changesets.
          Summary
          -------

          Combine the numerous repositories of the JDK forest into a single repository in order to simplify and streamline development.

          Non-Goals
          ---------

          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9 there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable source-code management operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the JDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs, since some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Mercurial repos and unity of the engineering dilutes one of the main benefits of modern source-code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools such as Mercurial bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds such as the "get source" script.

          Description
          -----------

          To address these issues, a prototype of a consolidated forest has been developed. The prototype is available at:

              http://hg.openjdk.java.net/jdk10/consol-proto/

          Some of the supporting conversion scripts used to create the prototype are attached as `unify.zip`.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per-file level, with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved.

          The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the JDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          As a consequence, from the root of the repository the relative path of a source file in a module preserved after the consolidation and src directory combination.

          An analogous but less aggressive reorganization is done for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          While the regression tests will run with the current state of the prototype, further consolidations of the jtreg configuration files are possible and may be done in the future.

          Alternatives
          ------------

          One alternative is to simply stay with the current set of repositories. The history of some or all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------

          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be finished before the consolidations is put into production. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward- and back- ports as opposed to exporting and importing changesets.
          darcy Joe Darcy made changes -
          Status Candidate [ 10003 ] Proposed to Target [ 10004 ]
          darcy Joe Darcy made changes -
          Duration S [ 19115 ] M [ 19116 ]
          mr Mark Reinhold made changes -
          Labels jdk10-ptt-2017-04-18
          darcy Joe Darcy made changes -
          Description Summary
          -------

          Combine the numerous repositories of the JDK forest into a single repository in order to simplify and streamline development.

          Non-Goals
          ---------

          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9 there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable source-code management operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the JDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs, since some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Mercurial repos and unity of the engineering dilutes one of the main benefits of modern source-code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools such as Mercurial bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds such as the "get source" script.

          Description
          -----------

          To address these issues, a prototype of a consolidated forest has been developed. The prototype is available at:

              http://hg.openjdk.java.net/jdk10/consol-proto/

          Some of the supporting conversion scripts used to create the prototype are attached as `unify.zip`.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per-file level, with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved.

          The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the JDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          As a consequence, from the root of the repository the relative path of a source file in a module preserved after the consolidation and src directory combination.

          An analogous but less aggressive reorganization is done for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/tests/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          While the regression tests will run with the current state of the prototype, further consolidations of the jtreg configuration files are possible and may be done in the future.

          Alternatives
          ------------

          One alternative is to simply stay with the current set of repositories. The history of some or all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------

          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be finished before the consolidations is put into production. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward- and back- ports as opposed to exporting and importing changesets.
          Summary
          -------

          Combine the numerous repositories of the JDK forest into a single repository in order to simplify and streamline development.

          Non-Goals
          ---------

          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9 there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable source-code management operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the JDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs, since some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Mercurial repos and unity of the engineering dilutes one of the main benefits of modern source-code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools such as Mercurial bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds such as the "get source" script.

          Description
          -----------

          To address these issues, a prototype of a consolidated forest has been developed. The prototype is available at:

              http://hg.openjdk.java.net/jdk10/consol-proto/

          Some of the supporting conversion scripts used to create the prototype are attached as `unify.zip`.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per-file level, with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved.

          The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the JDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          As a consequence, from the root of the repository the relative path of a source file in a module preserved after the consolidation and src directory combination.

          An analogous but less aggressive reorganization is done for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/test/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          While the regression tests will run with the current state of the prototype, further consolidations of the jtreg configuration files are possible and may be done in the future.

          Alternatives
          ------------

          One alternative is to simply stay with the current set of repositories. The history of some or all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------

          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be finished before the consolidations is put into production. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward- and back- ports as opposed to exporting and importing changesets.
          darcy Joe Darcy made changes -
          Description Summary
          -------

          Combine the numerous repositories of the JDK forest into a single repository in order to simplify and streamline development.

          Non-Goals
          ---------

          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9 there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable source-code management operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the JDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs, since some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Mercurial repos and unity of the engineering dilutes one of the main benefits of modern source-code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools such as Mercurial bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds such as the "get source" script.

          Description
          -----------

          To address these issues, a prototype of a consolidated forest has been developed. The prototype is available at:

              http://hg.openjdk.java.net/jdk10/consol-proto/

          Some of the supporting conversion scripts used to create the prototype are attached as `unify.zip`.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per-file level, with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved.

          The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the JDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          As a consequence, from the root of the repository the relative path of a source file in a module preserved after the consolidation and src directory combination.

          An analogous but less aggressive reorganization is done for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/test/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          While the regression tests will run with the current state of the prototype, further consolidations of the jtreg configuration files are possible and may be done in the future.

          Alternatives
          ------------

          One alternative is to simply stay with the current set of repositories. The history of some or all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------

          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be finished before the consolidations is put into production. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward- and back- ports as opposed to exporting and importing changesets.
          Summary
          -------

          Combine the numerous repositories of the JDK forest into a single repository in order to simplify and streamline development.

          Non-Goals
          ---------

          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9 there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable source-code management operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the JDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs, since some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Mercurial repos and unity of the engineering dilutes one of the main benefits of modern source-code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools such as Mercurial bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds such as the "get source" script.

          Description
          -----------

          To address these issues, a prototype of a consolidated forest has been developed. The prototype is available at:

              http://hg.openjdk.java.net/jdk10/consol-proto/

          Some of the supporting conversion scripts used to create the prototype are attached as `unify.zip`.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per-file level, with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved.

          The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the JDK forest there are module-based directories like

              $ROOT/jdk/**src**/java.base
              ...
              $ROOT/langtools/**src**/java.compiler
              ...

          In the consolidated forest, this code is instead organized as

              $ROOT/**src**/java.base
              $ROOT/**src**/java.compiler
              ...

          As a consequence, from the root of the repository the relative path of a source file in a module preserved after the consolidation and src directory combination.

          An analogous but less aggressive reorganization is done for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/test/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          While the regression tests will run with the current state of the prototype, further consolidations of the jtreg configuration files are possible and may be done in the future.

          Alternatives
          ------------

          One alternative is to simply stay with the current set of repositories. The history of some or all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------

          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be finished before the consolidations is put into production. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward- and back- ports as opposed to exporting and importing changesets.
          darcy Joe Darcy made changes -
          Description Summary
          -------

          Combine the numerous repositories of the JDK forest into a single repository in order to simplify and streamline development.

          Non-Goals
          ---------

          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9 there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable source-code management operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the JDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs, since some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Mercurial repos and unity of the engineering dilutes one of the main benefits of modern source-code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools such as Mercurial bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds such as the "get source" script.

          Description
          -----------

          To address these issues, a prototype of a consolidated forest has been developed. The prototype is available at:

              http://hg.openjdk.java.net/jdk10/consol-proto/

          Some of the supporting conversion scripts used to create the prototype are attached as `unify.zip`.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per-file level, with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved.

          The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the JDK forest there are module-based directories like

              $ROOT/jdk/**src**/java.base
              ...
              $ROOT/langtools/**src**/java.compiler
              ...

          In the consolidated forest, this code is instead organized as

              $ROOT/**src**/java.base
              $ROOT/**src**/java.compiler
              ...

          As a consequence, from the root of the repository the relative path of a source file in a module preserved after the consolidation and src directory combination.

          An analogous but less aggressive reorganization is done for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/test/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          While the regression tests will run with the current state of the prototype, further consolidations of the jtreg configuration files are possible and may be done in the future.

          Alternatives
          ------------

          One alternative is to simply stay with the current set of repositories. The history of some or all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------

          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be finished before the consolidations is put into production. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward- and back- ports as opposed to exporting and importing changesets.
          Summary
          -------

          Combine the numerous repositories of the JDK forest into a single repository in order to simplify and streamline development.

          Non-Goals
          ---------

          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9 there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable source-code management operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the JDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs, since some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Mercurial repos and unity of the engineering dilutes one of the main benefits of modern source-code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools such as Mercurial bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds such as the "get source" script.

          Description
          -----------

          To address these issues, a prototype of a consolidated forest has been developed. The prototype is available at:

              http://hg.openjdk.java.net/jdk10/consol-proto/

          Some of the supporting conversion scripts used to create the prototype are attached as `unify.zip`.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per-file level, with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved.

          The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the JDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          As a consequence, from the root of the repository the relative path of a source file in a module preserved after the consolidation and src directory combination.

          An analogous but less aggressive reorganization is done for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/test/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          While the regression tests will run with the current state of the prototype, further consolidations of the jtreg configuration files are possible and may be done in the future.

          Alternatives
          ------------

          One alternative is to simply stay with the current set of repositories. The history of some or all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------

          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be finished before the consolidations is put into production. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward- and back- ports as opposed to exporting and importing changesets.
          darcy Joe Darcy made changes -
          Attachment unify-OpenJDK-second-prototype.zip [ 67736 ]
          mr Mark Reinhold made changes -
          Endorsed By Mark Reinhold [ mr ]
          mr Mark Reinhold made changes -
          Status Proposed to Target [ 10004 ] Targeted [ 10005 ]
          darcy Joe Darcy made changes -
          Description Summary
          -------

          Combine the numerous repositories of the JDK forest into a single repository in order to simplify and streamline development.

          Non-Goals
          ---------

          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9 there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable source-code management operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the JDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs, since some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Mercurial repos and unity of the engineering dilutes one of the main benefits of modern source-code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools such as Mercurial bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds such as the "get source" script.

          Description
          -----------

          To address these issues, a prototype of a consolidated forest has been developed. The prototype is available at:

              http://hg.openjdk.java.net/jdk10/consol-proto/

          Some of the supporting conversion scripts used to create the prototype are attached as `unify.zip`.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per-file level, with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved.

          The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the JDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          As a consequence, from the root of the repository the relative path of a source file in a module preserved after the consolidation and src directory combination.

          An analogous but less aggressive reorganization is done for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/test/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          While the regression tests will run with the current state of the prototype, further consolidations of the jtreg configuration files are possible and may be done in the future.

          Alternatives
          ------------

          One alternative is to simply stay with the current set of repositories. The history of some or all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------

          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be finished before the consolidations is put into production. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward- and back- ports as opposed to exporting and importing changesets.
          Summary
          -------

          Combine the numerous repositories of the JDK forest into a single repository in order to simplify and streamline development.

          Non-Goals
          ---------

          Adding the FX sources to the JDK forest is not part of the proposal.

          Motivation
          ----------

          For many years, the full code base of the JDK has been broken into numerous Mercurial repositories. In JDK 9 there are eight repos: root, corba, hotspot, jaxp, jaxws, jdk, langtools, and nashorn.

          While this model of multiple repos offers some advantages, it also has many downsides and does a poor job of supporting various desirable source-code management operations. In particular, it is not possible to perform an atomic commit across repositories of inter-dependent changesets. For example, if the code for a single bug fix or RFE spans both the jdk and hotspot repos today, the change to both repositories cannot be done atomically in the forest hosting those two distinct repos. Changes spanning multiple repos are a common occurrence; over 1,100 bug ids have been reused across repositories in the JDK forest. The 1,100+ repo-crossing bugs is only a lower bound on the number of logically repo-crossing bugs, since some engineers use separate bug ids to push to different repos.

          This mismatch between the divisions of the Mercurial repos and unity of the engineering dilutes one of the main benefits of modern source-code management: tracking changes to sets of files rather than just individual files. As a corollary, this mismatch between SCM transactions and logical transactions complicates use of tools such as Mercurial bisect.

          The individual repos don't have a development cycle separate from the JDK as a whole; all the repos advance in lockstep with the JDK promotion cycle. The multiplicity of repos presents a larger than necessary barrier to entry to new developers and has lead to workarounds such as the "get source" script.

          Description
          -----------

          To address these issues, a prototype of a consolidated forest has been developed. The prototype is available at:

              http://hg.openjdk.java.net/jdk10/consol-proto/

          Some of the supporting conversion scripts used to create the prototype are attached as `unify.zip`.

          In the prototype. the eight repositories have been combined into a single repository using an automated conversion script that preserves history on a per-file level, with the consolidated forest being synchronized at the tags used to mark JDK promotions. The changeset comments and creation date are also preserved.

          The prototype has another level of code reorganization. In the consolidated forests, code for Java modules is generally combined under a single top-level src directory. For example, today in the JDK forest there are module-based directories like

              $ROOT/jdk/src/java.base
              ...
              $ROOT/langtools/src/java.compiler
              ...

          In the consolidated forest, this code is instead organized as

              $ROOT/src/java.base
              $ROOT/src/java.compiler
              ...

          As a consequence, from the root of the repository the relative path of a source file in a module is preserved after the consolidation and src directory combination.

          An analogous but less aggressive reorganization is done for the test directories to go from

              $ROOT/jdk/test/Foo.java
              $ROOT/langtools/test/Bar.java

          to

              $ROOT/test/jdk/Foo.java
              $ROOT/test/langtools/Bar.java

          Since the effort is currently a prototype, not all portions of it are entirely complete and the fit and finish can be improved in some areas. The HotSpot C/C++ sources are moved to the shared src directory alongside the modularized Java code.

          While the regression tests will run with the current state of the prototype, further consolidations of the jtreg configuration files are possible and may be done in the future.

          Alternatives
          ------------

          One alternative is to simply stay with the current set of repositories. The history of some or all of the repositories could have been dropped when moving to a single repository, but that was rejected. Consolidating a core subset of the repositories was considered, but rejected in favor of the simplicity of a single repository.

          Testing
          -------

          To validate the file contents, for each promotion tag a script was used to verify the contents of the split forest at that tag matched the contents of the consolidated repository at that tag. For a recent JDK 9 tag, builds of the split forest and consolidated forest at the same tag were compared; there were only minor and explainable differences.

          Risks and Assumptions
          ---------------------

          The testing described above should mitigate the most serious risks of file corruptions and faulty builds. While the major portions of the needed work for the consolidation are complete in the prototype, various smaller supporting features may not be finished before the consolidations is put into production. The pre and post consolidation code bases are not related in a Mercurial sense. Diffs (with suitably massaged paths) will have to be used for forward- and back- ports as opposed to exporting and importing changesets.
          mtrudeau Michel Trudeau made changes -
          Link This issue relates to BKLG-1250 [ BKLG-1250 ]
          Hide
          darcy Joe Darcy added a comment -
          The three main lines of development for JDK 10, master, client, and hotspot, were switched to consolidated repos in September 2017:

              http://mail.openjdk.java.net/pipermail/jdk10-dev/2017-September/000499.html

          A few small issues associated with the consolidation continue to be filed and fixed. Since development has been switched to the consolidated repo, marking this JEP as integrated.
          Show
          darcy Joe Darcy added a comment - The three main lines of development for JDK 10, master, client, and hotspot, were switched to consolidated repos in September 2017:      http://mail.openjdk.java.net/pipermail/jdk10-dev/2017-September/000499.html A few small issues associated with the consolidation continue to be filed and fixed. Since development has been switched to the consolidated repo, marking this JEP as integrated.
          darcy Joe Darcy made changes -
          Status Targeted [ 10005 ] Integrated [ 10007 ]
          darcy Joe Darcy made changes -
          Link This issue relates to JDK-8187444 [ JDK-8187444 ]
          darcy Joe Darcy made changes -
          Link This issue relates to JDK-8187443 [ JDK-8187443 ]

            People

            • Assignee:
              darcy Joe Darcy
              Reporter:
              darcy Joe Darcy
              Owner:
              Joe Darcy
              Reviewed By:
              Brian Goetz, Mikael Vidstedt
              Endorsed By:
              Mark Reinhold
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated: