Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8165852

(fs) Mount point not found for a file which is present in overlayfs

    Details

    • Subcomponent:
    • Resolved In Build:
      b01
    • CPU:
      x86_64
    • OS:
      linux

      Description

      FULL PRODUCT VERSION :
      java version "1.8.0_77"
      Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
      Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)

      ADDITIONAL OS VERSION INFORMATION :
      [root@6b8ab63d2271 /]# uname -a
      Linux 6b8ab63d2271 4.6.0-0.bpo.1-amd64 #1 SMP Debian 4.6.3-1~bpo8+1 (2016-07-13) x86_64 x86_64 x86_64 GNU/Linux

      [root@6b8ab63d2271 /]# cat /etc/redhat-release
      CentOS Linux release 7.2.1511 (Core)

      EXTRA RELEVANT SYSTEM CONFIGURATION :
      Host System: Debian Jessie
      Runs Docker version 1.12.1, build 23cf638 started with --storage-driver=overlay
      Error does occur inside of the docker container which is centos 7

      A DESCRIPTION OF THE PROBLEM :
      When running java inside of a docker container which root fs is an overlayfs

      e.g. from mount: overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay/7f380aa866f1411985b0e3d9533e89f0907e11adb81c2c1e20a5733fd3112f33/root,upperdir=/var/lib/docker/overlay/58f650cc4005e9becf01c20b2d8144a372115d11add9e70774322232e3fef590/upper,workdir=/var/lib/docker/overlay/58f650cc4005e9becf01c20b2d8144a372115d11add9e70774322232e3fef590/work)

      And accessing a file like /home/someuser/somefile.txt

      An Exception will be thrown.


      A very similar Case seems to exist with btrfs and subvolumes:
      https://github.com/elastic/elasticsearch/issues/14337 and
      https://github.com/elastic/elasticsearch/issues/12018


      STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
      Start a docker container where dockerd was started with --overlfay and try to access files via Files.getFileStore(path)

      EXPECTED VERSUS ACTUAL BEHAVIOR :
      EXPECTED -
      The Result should be / (if there is no other mount)
      ACTUAL -
      java.io.IOException: Mount point not found

      ERROR MESSAGES/STACK TRACES THAT OCCUR :
      Exception in thread "main" java.io.IOException: Mount point not found
              at sun.nio.fs.LinuxFileStore.findMountEntry(LinuxFileStore.java:91)
              at sun.nio.fs.UnixFileStore.<init>(UnixFileStore.java:65)
              at sun.nio.fs.LinuxFileStore.<init>(LinuxFileStore.java:44)
              at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:51)
              at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:39)
              at sun.nio.fs.UnixFileSystemProvider.getFileStore(UnixFileSystemProvider.java:368)
              at java.nio.file.Files.getFileStore(Files.java:1461)
              at MyMain.main(MyMain.java:8)

      REPRODUCIBILITY :
      This bug can be reproduced always.

      ---------- BEGIN SOURCE ----------
      import java.io.IOException;
      import java.nio.file.Files;
      import java.nio.file.Paths;

      public class MyMain {

      public static void main(String[] args) throws IOException {
      System.out.println(Files.getFileStore(Paths.get(args[0])));
      }
      }

      ---------- END SOURCE ----------

        Issue Links

          Activity

          webbuggrp Webbug Group created issue -
          psonal Pallavi Sonal made changes -
          Field Original Value New Value
          Assignee Pallavi Sonal [ psonal ]
          psonal Pallavi Sonal made changes -
          Status New [ 10000 ] Open [ 1 ]
          psonal Pallavi Sonal made changes -
          Affects Version/s 8 [ 15409 ]
          Affects Version/s 8u102 [ 18510 ]
          psonal Pallavi Sonal made changes -
          Affects Version/s 9 [ 16400 ]
          psonal Pallavi Sonal made changes -
          Labels webbug dcs-pso reproducer-no webbug
          psonal Pallavi Sonal made changes -
          Labels dcs-pso reproducer-no webbug dcs-pso reproducer-other webbug
          psonal Pallavi Sonal made changes -
          Project Java Incidents [ 10301 ] JDK [ 10100 ]
          Key JI-9043551 JDK-8165852
          Workflow JBS Incident Workflow [ 4911439 ] JBS Workflow [ 4911439 ]
          Component/s core-libs [ 10300 ]
          Component/s core-libs [ 10701 ]
          Affects Version/s 8 [ 11815 ]
          Affects Version/s 9 [ 14949 ]
          Affects Version/s 8 [ 15409 ]
          Affects Version/s 9 [ 16400 ]
          psonal Pallavi Sonal made changes -
          Subcomponent java.nio [ 499 ] java.nio [ 228 ]
          psonal Pallavi Sonal made changes -
          Assignee Pallavi Sonal [ psonal ]
          psonal Pallavi Sonal made changes -
          Status Open [ 1 ] New [ 10000 ]
          alanb Alan Bateman made changes -
          Status New [ 10000 ] Open [ 1 ]
          jeff Jeff Dinkins made changes -
          Assignee Brian Burkhalter [ bpb ]
          bpb Brian Burkhalter made changes -
          Link This issue relates to JDK-8165323 [ JDK-8165323 ]
          bpb Brian Burkhalter made changes -
          Link This issue relates to JDK-8166162 [ JDK-8166162 ]
          Hide
          bpb Brian Burkhalter added a comment -
          This problem is caused by breaking out of the loop in step 2 in LinuxFileStore.findMountEntry() when the parent's device (stat st_dev) is not equal to the device of the current path but there is no entry for the parent path in the set of mounted file systems listed in /proc/mounts. This was observed to occur in Docker containers running in an Ubuntu 16.04 amd64 VM for the overlay and overlay2 storage drivers but not for the aufs storage driver for the same test conditions. The btrfs and devicemapper storage drivers have not yet been tested.

          It could be argued that this is a bug in the overlayfs storage drivers by virtue of the inconsistency between what stat(2) and perhaps other system calls return versus what is present in /proc/mounts. Nonetheless it might be worth investigating a benign workaround which would enable this case without breaking others.
          Show
          bpb Brian Burkhalter added a comment - This problem is caused by breaking out of the loop in step 2 in LinuxFileStore.findMountEntry() when the parent's device (stat st_dev) is not equal to the device of the current path but there is no entry for the parent path in the set of mounted file systems listed in /proc/mounts. This was observed to occur in Docker containers running in an Ubuntu 16.04 amd64 VM for the overlay and overlay2 storage drivers but not for the aufs storage driver for the same test conditions. The btrfs and devicemapper storage drivers have not yet been tested. It could be argued that this is a bug in the overlayfs storage drivers by virtue of the inconsistency between what stat(2) and perhaps other system calls return versus what is present in /proc/mounts. Nonetheless it might be worth investigating a benign workaround which would enable this case without breaking others.
          bpb Brian Burkhalter made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Understanding Cause Known [ 10000 ]
          Hide
          bpb Brian Burkhalter added a comment - - edited
          The problem as it manifests in btrfs is essentially the same situation as observed for overlayfs in docker: the parent of a path component has a different device ID than that of the current path component and so the loop of the upwards traversal towards the root is broken out of but there is no entry for the current path component in /proc/mounts. This was observed to occur specifically for the case of a file contained in a btrfs sub-volume which is not itself explicitly mounted. If the sub-volume is mounted via `mount` it shows up in /proc/mounts and the error does not occur. This failure might be due to the nature of btrfs sub-volumes and unrelated to Docker per se.
          Show
          bpb Brian Burkhalter added a comment - - edited The problem as it manifests in btrfs is essentially the same situation as observed for overlayfs in docker: the parent of a path component has a different device ID than that of the current path component and so the loop of the upwards traversal towards the root is broken out of but there is no entry for the current path component in /proc/mounts. This was observed to occur specifically for the case of a file contained in a btrfs sub-volume which is not itself explicitly mounted. If the sub-volume is mounted via `mount` it shows up in /proc/mounts and the error does not occur. This failure might be due to the nature of btrfs sub-volumes and unrelated to Docker per se.
          bpb Brian Burkhalter made changes -
          Summary Mount point not found for a file which is present in overlayfs (fs) Mount point not found for a file which is present in overlayfs
          bpb Brian Burkhalter made changes -
          Fix Version/s tbd_major [ 11972 ]
          bpb Brian Burkhalter made changes -
          Status In Progress [ 3 ] Open [ 1 ]
          mseledtsov Mikhailo Seledtsov made changes -
          Labels dcs-pso reproducer-other webbug dcs-pso docker reproducer-other webbug
          alanb Alan Bateman made changes -
          Priority P4 [ 4 ] P3 [ 3 ]
          alanb Alan Bateman made changes -
          Fix Version/s 10 [ 16302 ]
          Fix Version/s tbd_major [ 11972 ]
          alanb Alan Bateman made changes -
          Fix Version/s 11 [ 18723 ]
          Fix Version/s 10 [ 16302 ]
          bpb Brian Burkhalter made changes -
          Link This issue duplicates JDK-8194283 [ JDK-8194283 ]
          bpb Brian Burkhalter made changes -
          Remote Link This issue links to "nio-dev post: a possible fix (Web Link)" [ 15082 ]
          bpb Brian Burkhalter made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          bpb Brian Burkhalter made changes -
          Understanding Cause Known [ 10000 ] Fix Understood [ 10001 ]
          bpb Brian Burkhalter made changes -
          Labels dcs-pso docker reproducer-other webbug dcs-pso docker noreg-hard reproducer-other webbug
          bpb Brian Burkhalter made changes -
          Remote Link This issue links to "nio-dev review, Jan. 2018 (Web Link)" [ 15083 ]
          Hide
          hgupdate HG Updates added a comment -
          URL: http://hg.openjdk.java.net/jdk/jdk/rev/bdbbf56c302e
          User: bpb
          Date: 2018-01-12 19:21:15 +0000
          Show
          hgupdate HG Updates added a comment - URL: http://hg.openjdk.java.net/jdk/jdk/rev/bdbbf56c302e User: bpb Date: 2018-01-12 19:21:15 +0000
          hgupdate HG Updates made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          Resolved In Build master [ 18256 ]
          Understanding Fix Understood [ 10001 ]
          Resolution Fixed [ 1 ]
          hgupdate HG Updates made changes -
          Resolved In Build master [ 18256 ] b01 [ 17304 ]

            People

            • Assignee:
              bpb Brian Burkhalter
              Reporter:
              webbuggrp Webbug Group
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: