Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8189376

Unsorted $(wildcard) causes unstable module-deps.gmk

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: P3
    • Resolution: Fixed
    • Affects Version/s: 10
    • Fix Version/s: 10
    • Component/s: infrastructure
    • Labels:
      None
    • Subcomponent:
    • Resolved In Build:
      b29

      Description

      Here at Google we've been seeing build failures like this:

      /<cloudy-filesystem>/build/linux-x86_64-normal-server-release/make-support/module-deps.gmk:6: *** missing separator. Stop.
      make/Main.gmk:143: recipe for target 'java.base-gensrc-moduleinfo' failed

      After investigation, they appear only if:
      - we're using at least GNU make 3.82
      - we're using a fancy cloud filesystem that keeps reporting values from directory traversals in different orders (as filesystems are generally permitted to do)
      - multiple build jobs are running (JOBS > 1)

      I discovered this change in the make 3.82 NEWS:

      * WARNING: Backward-incompatibility!
        Wildcards were not documented as returning sorted values, but the results
        have been sorted up until this release.. If your makefiles require sorted
        results from wildcard expansions, use the $(sort ...) function to request
        it explicitly.

      The problem is that module-deps.gmk keeps getting regenerated, but only when it is already in use by multiple jobs, causing other jobs to read a half-constructed or corrupted makefile!

      The "obvious" workaround is to re-add calls to $(sort ...)

      +++ /...make/common/Modules.gmk
      @@ -239,20 +239,20 @@
       # configuration.
       # Param 1 - Module to find for, set to * for finding all
       FindAllModuleInfos = \
      - $(wildcard \
      + $(sort $(wildcard \
               $(foreach sub, $(SRC_SUBDIRS), \
                 $(patsubst %,%/$(strip $1)/$(sub)/module-info.java, $(TOP_SRC_DIRS))) \
      - $(patsubst %,%/$(strip $1)/module-info.java, $(IMPORT_MODULES_SRC)))
      + $(patsubst %,%/$(strip $1)/module-info.java, $(IMPORT_MODULES_SRC))))
       
       # Find module-info.java files in the specific source dir
       # Param 1 - Src dir to find module-info.java files in
       
      BUT the deeper problem is that the makefile system pretends that it can handle updates to the set of modules discovered during the build, when it cannot, due to build concurrency. Perhaps this code should be rewritten to compute module-deps.gmk exactly once? If you really want to support the set of modules changing dynamically during the build, you will probably need to write a separate module-info.gmk file for each JOB, which sounds difficult (not worth doing?).

        Attachments

          Activity

            People

            • Assignee:
              ihse Magnus Ihse Bursie
              Reporter:
              martin Martin Buchholz
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: