Uploaded image for project: 'JDK'
  1. JDK
  2. JDK-8146115

Improve docker container detection and resource configuration usage

    Details

    • Type: Enhancement
    • Status: Resolved
    • Priority: P3
    • Resolution: Fixed
    • Affects Version/s: 9
    • Fix Version/s: 10
    • Component/s: hotspot
    • Environment:

      Docker container

    • Subcomponent:
    • Resolved In Build:
      team
    • CPU:
      x86
    • OS:
      linux

      Description

      Java startup normally queries the operating system in order to setup runtime defaults for things such as the number of GC threads and default memory limits. When running in a container, the operating system functions used provide information about the host and do not include the container configuration and limits. The VM and core libraries will be modified as part of this RFE to first determine if the current running process is running in a container. It will then cause the runtime to use the container values rather than the general operating system functions for configuring and managing the Java process. There have been a few attempts to correct some of these issue in the VM but they are not complete. The CPU detection in the VM currently only handles a container that limits cpu usage via CPU sets. If the Docker --cpu or --cpu-period along with --cpu-quota options are specified, it currently has no effect on the VMs configuration.
      The experimental memory detection that has been implemented only impacts the Heap selection and does not apply to the os::physical_memory or os::available_memory low level functions. This leaves other parts of the VM and core libraries to believe there is more memory available than there actually is.

      To correct these shortcomings and make this support more robust, here's a list of the current cgroup subsystems that we be examined in order to update the internal VM and core library configuration.

      Number of CPUs
      -----------------------
      Use a combination of number_of_cpus() and cpu_sets() in order to determine how many processors are available to the process and adjust the JVMs os::active_processor_count appropriately. The number_of_cpus() will be calculated based on the cpu_quota() and cpu_period() using this formula: number_of_cpus() = cpu_quota() / cpu_period(). If cpu_shares has been setup for the container, the number_of_cpus() will be calculated based on cpu_shares()/1024. 1024 is the default and standard unit for calculating relative cpu usage in cloud based container management software.

      Also add a new VM flag (-XX:ActiveProcessorCount=xx) that allows the number of CPUs to be overridden. This flag will be honored even if UseContainerSupport is not enabled.

      Total available memory
      -------------------------------
      Use the memory_limit() value from the cgroup file system to initialize the os::physical_memory() value in the VM. This value will propagate to all other parts of the Java runtime.

      Memory usage
      --------------------
      Use memory_usage_in_bytes() for providing os::available_memory() by subtracting the usage from the total available memory allocated to the container.

      As as troubleshooting aid, we will dump any available container statistics to the hotspot error log and add container specific information to the JVM logging system. Unified Logging will be added to help to diagnose issue related to this support. Use -Xlog:os+container=trace for maximum logging of container information.

      A new option -XX:-UseContainerSupport will be added to allow the container support to be disabled. The default for this flag will be true. Container support will be enabled by default.


        Issue Links

          Activity

          Hide
          bobv Bob Vandette added a comment - - edited
          Attached is a program that demonstrates a more robust way of extracting cgroup configuration data for memory and cpuset subsystems. It also shows how to determine if cgroup v1 versus v2 is operating.
          Show
          bobv Bob Vandette added a comment - - edited Attached is a program that demonstrates a more robust way of extracting cgroup configuration data for memory and cpuset subsystems. It also shows how to determine if cgroup v1 versus v2 is operating.
          Hide
          dholmes David Holmes added a comment -
          From a slack chat:

          1:30 PM]
          Fairoz Matte How to identify the JVM crashes occured on Cloud environment from hs_error file. Ex System configuration "running on docker 17.09 in Ubuntu 16.04 server" I don't find this sort of information in hs_error logs..

          [1:50 PM]
          David Holmes The vm doesn't know anything about docker versions, or even if it is running in docker. Even the upcoming container support is based on cgroups, not docker.

          [4:06 PM]
          Ioi Lam Google tells me this: https://forums.docker.com/t/get-a-containers-full-id-from-inside-of-itself/37237/2
          Docker Forums
          Get a container's full id from inside of itself
          Well, it seems that I already found the solution ! I simply have to run the following command from inside a container cat /proc/self/cgroup | head -1 | tr --delete ‘10:memory:/docker/’ Best

          [4:20 PM]
          Fairoz Matte Thanks Ioi, How about adding this information under System Section of hs_error file? Something like "OS:CentOS Linux release 7.4.1708 (Core)
          //if docker get docker info
          docker 17.09 in Ubuntu 16.04 server
          uname:Linux 4.4.0-96-generic #119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017 x86_64
          Show
          dholmes David Holmes added a comment - From a slack chat: 1:30 PM] Fairoz Matte How to identify the JVM crashes occured on Cloud environment from hs_error file. Ex System configuration "running on docker 17.09 in Ubuntu 16.04 server" I don't find this sort of information in hs_error logs.. [1:50 PM] David Holmes The vm doesn't know anything about docker versions, or even if it is running in docker. Even the upcoming container support is based on cgroups, not docker. [4:06 PM] Ioi Lam Google tells me this: https://forums.docker.com/t/get-a-containers-full-id-from-inside-of-itself/37237/2 Docker Forums Get a container's full id from inside of itself Well, it seems that I already found the solution ! I simply have to run the following command from inside a container cat /proc/self/cgroup | head -1 | tr --delete ‘10:memory:/docker/’ Best [4:20 PM] Fairoz Matte Thanks Ioi, How about adding this information under System Section of hs_error file? Something like "OS:CentOS Linux release 7.4.1708 (Core) //if docker get docker info docker 17.09 in Ubuntu 16.04 server uname:Linux 4.4.0-96-generic #119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017 x86_64
          Hide
          bobv Bob Vandette added a comment -
          I do not believe I can determine the docker version from within a container.

          The only sign that a container process is running via docker is the fact that docker currently
          mounts the containers cgroup file system under /docker.

          /proc/self/cgroup contains paths such as these:

          10:freezer:/docker/42cd8b1ebfeb1d6ce991ecb87916292cda8c41d364c6615c497b6fb3b4865977
          9:devices:/docker/42cd8b1ebfeb1d6ce991ecb87916292cda8c41d364c6615c497b6fb3b4865977
          8:memory:/docker/42cd8b1ebfeb1d6ce991ecb87916292cda8c41d364c6615c497b6fb3b4865977
          7:hugetlb:/docker/42cd8b1ebfeb1d6ce991ecb87916292cda8c41d364c6615c497b6fb3b4865977
          6:cpu,cpuacct:/docker/42cd8b1ebfeb1d6ce991ecb87916292cda8c41d364c6615c497b6fb3b4865977
          5:blkio:/docker/42cd8b1ebfeb1d6ce991ecb87916292cda8c41d364c6615c497b6fb3b4865977
          4:net_cls,net_prio:/docker/42cd8b1ebfeb1d6ce991ecb87916292cda8c41d364c6615c497b6fb3b4865977
          3:cpuset:/docker/42cd8b1ebfeb1d6ce991ecb87916292cda8c41d364c6615c497b6fb3b4865977
          2:perf_event:/docker/42cd8b1ebfeb1d6ce991ecb87916292cda8c41d364c6615c497b6fb3b4865977
          1:name=systemd:/docker/42cd8b1ebfeb1d6ce991ecb87916292cda8c41d364c6615c497b6fb3b4865977

          I don't think it's wise to assume that this convention will continue. In addition, there are a few competing
          libcontainer implementations that we will want to support. These will most likely not follow this convention.


          Show
          bobv Bob Vandette added a comment - I do not believe I can determine the docker version from within a container. The only sign that a container process is running via docker is the fact that docker currently mounts the containers cgroup file system under /docker. /proc/self/cgroup contains paths such as these: 10:freezer:/docker/42cd8b1ebfeb1d6ce991ecb87916292cda8c41d364c6615c497b6fb3b4865977 9:devices:/docker/42cd8b1ebfeb1d6ce991ecb87916292cda8c41d364c6615c497b6fb3b4865977 8:memory:/docker/42cd8b1ebfeb1d6ce991ecb87916292cda8c41d364c6615c497b6fb3b4865977 7:hugetlb:/docker/42cd8b1ebfeb1d6ce991ecb87916292cda8c41d364c6615c497b6fb3b4865977 6:cpu,cpuacct:/docker/42cd8b1ebfeb1d6ce991ecb87916292cda8c41d364c6615c497b6fb3b4865977 5:blkio:/docker/42cd8b1ebfeb1d6ce991ecb87916292cda8c41d364c6615c497b6fb3b4865977 4:net_cls,net_prio:/docker/42cd8b1ebfeb1d6ce991ecb87916292cda8c41d364c6615c497b6fb3b4865977 3:cpuset:/docker/42cd8b1ebfeb1d6ce991ecb87916292cda8c41d364c6615c497b6fb3b4865977 2:perf_event:/docker/42cd8b1ebfeb1d6ce991ecb87916292cda8c41d364c6615c497b6fb3b4865977 1:name=systemd:/docker/42cd8b1ebfeb1d6ce991ecb87916292cda8c41d364c6615c497b6fb3b4865977 I don't think it's wise to assume that this convention will continue. In addition, there are a few competing libcontainer implementations that we will want to support. These will most likely not follow this convention.
          Hide
          mseledtsov Mikhailo Seledtsov added a comment -
          From SQE point of view, this change is ready for integration.
          Show
          mseledtsov Mikhailo Seledtsov added a comment - From SQE point of view, this change is ready for integration.
          Hide
          hgupdate HG Updates added a comment -
          URL: http://hg.openjdk.java.net/jdk/hs/rev/7f22774a5f42
          User: bobv
          Date: 2017-11-16 23:09:57 +0000
          Show
          hgupdate HG Updates added a comment - URL: http://hg.openjdk.java.net/jdk/hs/rev/7f22774a5f42 User: bobv Date: 2017-11-16 23:09:57 +0000

            People

            • Assignee:
              bobv Bob Vandette
              Reporter:
              acorn Karen Kinnear
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: