Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8165852

(fs) Mount point not found for a file which is present in overlayfs

    Details

    • Subcomponent:
    • Understanding:
      Cause Known
    • CPU:
      x86_64
    • OS:
      linux

      Description

      FULL PRODUCT VERSION :
      java version "1.8.0_77"
      Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
      Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)

      ADDITIONAL OS VERSION INFORMATION :
      [root@6b8ab63d2271 /]# uname -a
      Linux 6b8ab63d2271 4.6.0-0.bpo.1-amd64 #1 SMP Debian 4.6.3-1~bpo8+1 (2016-07-13) x86_64 x86_64 x86_64 GNU/Linux

      [root@6b8ab63d2271 /]# cat /etc/redhat-release
      CentOS Linux release 7.2.1511 (Core)

      EXTRA RELEVANT SYSTEM CONFIGURATION :
      Host System: Debian Jessie
      Runs Docker version 1.12.1, build 23cf638 started with --storage-driver=overlay
      Error does occur inside of the docker container which is centos 7

      A DESCRIPTION OF THE PROBLEM :
      When running java inside of a docker container which root fs is an overlayfs

      e.g. from mount: overlay on / type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay/7f380aa866f1411985b0e3d9533e89f0907e11adb81c2c1e20a5733fd3112f33/root,upperdir=/var/lib/docker/overlay/58f650cc4005e9becf01c20b2d8144a372115d11add9e70774322232e3fef590/upper,workdir=/var/lib/docker/overlay/58f650cc4005e9becf01c20b2d8144a372115d11add9e70774322232e3fef590/work)

      And accessing a file like /home/someuser/somefile.txt

      An Exception will be thrown.


      A very similar Case seems to exist with btrfs and subvolumes:
      https://github.com/elastic/elasticsearch/issues/14337 and
      https://github.com/elastic/elasticsearch/issues/12018


      STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
      Start a docker container where dockerd was started with --overlfay and try to access files via Files.getFileStore(path)

      EXPECTED VERSUS ACTUAL BEHAVIOR :
      EXPECTED -
      The Result should be / (if there is no other mount)
      ACTUAL -
      java.io.IOException: Mount point not found

      ERROR MESSAGES/STACK TRACES THAT OCCUR :
      Exception in thread "main" java.io.IOException: Mount point not found
              at sun.nio.fs.LinuxFileStore.findMountEntry(LinuxFileStore.java:91)
              at sun.nio.fs.UnixFileStore.<init>(UnixFileStore.java:65)
              at sun.nio.fs.LinuxFileStore.<init>(LinuxFileStore.java:44)
              at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:51)
              at sun.nio.fs.LinuxFileSystemProvider.getFileStore(LinuxFileSystemProvider.java:39)
              at sun.nio.fs.UnixFileSystemProvider.getFileStore(UnixFileSystemProvider.java:368)
              at java.nio.file.Files.getFileStore(Files.java:1461)
              at MyMain.main(MyMain.java:8)

      REPRODUCIBILITY :
      This bug can be reproduced always.

      ---------- BEGIN SOURCE ----------
      import java.io.IOException;
      import java.nio.file.Files;
      import java.nio.file.Paths;

      public class MyMain {

      public static void main(String[] args) throws IOException {
      System.out.println(Files.getFileStore(Paths.get(args[0])));
      }
      }

      ---------- END SOURCE ----------

        Issue Links

          Activity

          Hide
          bpb Brian Burkhalter added a comment -
          This problem is caused by breaking out of the loop in step 2 in LinuxFileStore.findMountEntry() when the parent's device (stat st_dev) is not equal to the device of the current path but there is no entry for the parent path in the set of mounted file systems listed in /proc/mounts. This was observed to occur in Docker containers running in an Ubuntu 16.04 amd64 VM for the overlay and overlay2 storage drivers but not for the aufs storage driver for the same test conditions. The btrfs and devicemapper storage drivers have not yet been tested.

          It could be argued that this is a bug in the overlayfs storage drivers by virtue of the inconsistency between what stat(2) and perhaps other system calls return versus what is present in /proc/mounts. Nonetheless it might be worth investigating a benign workaround which would enable this case without breaking others.
          Show
          bpb Brian Burkhalter added a comment - This problem is caused by breaking out of the loop in step 2 in LinuxFileStore.findMountEntry() when the parent's device (stat st_dev) is not equal to the device of the current path but there is no entry for the parent path in the set of mounted file systems listed in /proc/mounts. This was observed to occur in Docker containers running in an Ubuntu 16.04 amd64 VM for the overlay and overlay2 storage drivers but not for the aufs storage driver for the same test conditions. The btrfs and devicemapper storage drivers have not yet been tested. It could be argued that this is a bug in the overlayfs storage drivers by virtue of the inconsistency between what stat(2) and perhaps other system calls return versus what is present in /proc/mounts. Nonetheless it might be worth investigating a benign workaround which would enable this case without breaking others.
          Hide
          bpb Brian Burkhalter added a comment - - edited
          The problem as it manifests in btrfs is essentially the same situation as observed for overlayfs in docker: the parent of a path component has a different device ID than that of the current path component and so the loop of the upwards traversal towards the root is broken out of but there is no entry for the current path component in /proc/mounts. This was observed to occur specifically for the case of a file contained in a btrfs sub-volume which is not itself explicitly mounted. If the sub-volume is mounted via `mount` it shows up in /proc/mounts and the error does not occur. This failure might be due to the nature of btrfs sub-volumes and unrelated to Docker per se.
          Show
          bpb Brian Burkhalter added a comment - - edited The problem as it manifests in btrfs is essentially the same situation as observed for overlayfs in docker: the parent of a path component has a different device ID than that of the current path component and so the loop of the upwards traversal towards the root is broken out of but there is no entry for the current path component in /proc/mounts. This was observed to occur specifically for the case of a file contained in a btrfs sub-volume which is not itself explicitly mounted. If the sub-volume is mounted via `mount` it shows up in /proc/mounts and the error does not occur. This failure might be due to the nature of btrfs sub-volumes and unrelated to Docker per se.

            People

            • Assignee:
              bpb Brian Burkhalter
              Reporter:
              webbuggrp Webbug Group
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated: