Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8211383

Mercurial clonebundles for hg.openjdk.java.net

    XMLWordPrintable

    Details

    • Type: Enhancement
    • Status: New
    • Priority: P3
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: infrastructure
    • Labels:
      None

      Description

      (this was originally posted at http://mail.openjdk.java.net/pipermail/skara-dev/2018-September/000020.html)

      This might not be under the umbrella of Skara project, and it may be irrelevant if/when we migrate to Git, but can we repeat what Mozilla does for their monorepo and enable clonebundles [1] on hg.openjdk.java.net, at least on busy repositories?

      It seems to be easy to configure: enable extension on server side, "hg bundle" the current state of repository, put the bundle somewhere, list the bundles in .hg/clonebundles.manifest on server side, and everything works magically from there. Clients would choose the bundle they can understand, or fallback to regular clone. Clients would also pull all the new changesets after the bundle was created, which means bundles can be refreshed only once in a while.

      My dirty experiments with clonebundles are very promising. Even within-Europe clones have improved quite a bit [2][3], almost down to the clone performance of local seeding [4]. I think that means: a) clonebundles test bottlenecks on hg client unbundling performance; b) non-clonebundles test bottlenecks on hgserver CPU. With clonebundles, hgserver CPU utilization drops to almost zero.

      The bundle is streamed by the usual HTTP(S) servers, which can be the part of geo-local CDN. Of course, doing similar thing with Git and Skara's GitHub mirror is much faster [5], quite probably because Git server+client is faster at dealing with the repo.

      Bottom-line: Enabling clonebundles would simplify our development lives before Skara arrives, and it would certainly be usable on repositories which Skara migration would not touch (e.g. all jdk-updates/*?). It should get CPU costs for hg.openjdk.java.net down too.

      [1] https://www.mercurial-scm.org/wiki/ClonebundlesExtension

      [2] ------- remote hgserver, clonebundles disabled

      $ time hg clone --config ui.clonebundles=false http://shipilev.net:8080/ jdk-jdk
      requesting all changes
      adding changesets
      adding manifests
      adding file changes
      added 51849 changesets with 448820 changes to 180327 files
      new changesets fd16c54261b3:54afe70c50b6
      updating to branch default
      66512 files updated, 0 files merged, 0 files removed, 0 files unresolved

      real 7m18.963s
      user 2m27.424s
      sys 0m16.004s

      [3] ------- remote hgserver, clonebundles enabled, remote seed

      $ time hg clone http://shipilev.net:8080/ jdk-jdk
      applying clone bundle from https://builds.shipilev.net/scratch/jdk-jdk.zstd-max.hg
      adding changesets
      adding manifests
      adding file changes
      added 51849 changesets with 448820 changes to 180327 files
      finished applying clone bundle
      searching for changes
      no changes found
      updating to branch default
      66512 files updated, 0 files merged, 0 files removed, 0 files unresolved

      real 1m59.017s
      user 2m10.728s
      sys 0m14.323s

      [4] ------- remote hgserver, clonebundles enabled, local seed

      $ time hg clone http://shipilev.net:8080/ jdk-jdk
      applying clone bundle from http://localhost:8000/jdk-jdk.zstd-max.hg
      adding changesets
      adding manifests
      adding file changes
      added 51849 changesets with 448820 changes to 180327 files
      finished applying clone bundle
      searching for changes
      no changes found
      updating to branch default
      66512 files updated, 0 files merged, 0 files removed, 0 files unresolved

      real 1m55.545s
      user 2m9.893s
      sys 0m14.186s


      [5] ------- remote git, GitHub mirrors

      $ time git clone https://github.com/Project-Skara/jdk.git
      Cloning into 'jdk'...
      remote: Enumerating objects: 881253, done.
      remote: Counting objects: 100% (881253/881253), done.
      remote: Compressing objects: 100% (160363/160363), done.
      remote: Total 881253 (delta 665729), reused 881253 (delta 665729), pack-reused 0
      Receiving objects: 100% (881253/881253), 258.61 MiB | 7.16 MiB/s, done.
      Resolving deltas: 100% (665729/665729), done.
      Checking out files: 100% (66258/66258), done.

      real 1m1.359s
      user 0m47.248s
      sys 0m6.129s

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              shade Aleksey Shipilev
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: