context: kubernetes/kubernetes#68478
The inotify code was removed from golang.org/x/exp several years ago. Therefore
importing it from that path prevents downstream consumers from using any module
that makes use of more recent features of golang.org/x/exp.
Given that this code is by definition frozen and that the long term path should
be to migrate to fsnotify, replacing the current code by an identical standalone
copy doesn't have maintenance cost, and will unlock other activities for
kubernetes for example.
We see a lot of logs in k/k CI as follows:
"Factory "mesos" was unable to handle container "/system.slice/home-kubernetes-containerized_mounter.mount"
It would be better if we do some sanity check for mesos running before
we try to use it.
Change-Id: I5f6ebcd44fdd4f8d724b85857edf1600473ef1ab
GetSpec() can be called concurrently in
manager/container.go.updateSpec()
results into a concurrent map access on the labels map because we're
directly updating the map inside GetSpec(). The labels map from the
container handler is not a copy of the map itself, just a reference,
that's why we're getting the concurrent map access.
Fix this by moving the label update with restartcount to the handler's
initialization method which is not called concurrently.
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
Per-CPU stats are more expensive to transport and store, and that
level of detail is not required in many cases.
We export overall total cpu in the same metric as per-cpu, so that
dashboards which previously summed over cpu will work identically.
As these can otherwise block indefinitely due to docker issues.
This is to fix https://github.com/kubernetes/kubernetes/issues/53207,
where kubelet relies on cadvisor for gathering docker information as
part of its periodic node status update.
This commit includes changes to integrate containerd
runtime to cadvisor to collect container stats
Signed-off-by: abhi <abhi@docker.com>
Test cases and minor changes
This commit include test cases and minor fixes
for the same
Signed-off-by: abhi <abhi@docker.com>
When cAdvisor starts up, it would read the `vendor` files in
`/sys/bus/pci/devices/*` to see if any NVIDIA devices (vendor ID: 0x10de) are
attached to the node. If no NVIDIA devices are found, this code path would
become dormant for the rest of cAdvisor lifetime. If NVIDIA devices are found,
we would start a goroutine that would check for the presence of NVML by trying
to dynamically load it at regular intervals. We need to do this regular
checking instead of doing it just once because it may happen that cAdvisor is
started before the NVIDIA drivers and NVML are installed. Once the NVML
dynamic loading succeeds, we would use NVML’s query methods to find out how
many devices exist on the node and create a map from their minor numbers to
their handles and cache that map. The goroutine would exit at this point.
If we detected the presence of NVML in the previous step, whenever a new
container is detected by cAdvisor, cAdvisor would read the `devices.list` file
from the container's devices cgroup. The `devices.list` file lists the
major:minor number of all the devices that the container is allowed to access.
If we find any device with major number 195 (which is the major number assigned
to NVIDIA devices), we would cache the list of corresponding minor numbers for
that container.
During every housekeeping operation, in addition to collecting all the existing
metrics, we will use the cached NVIDIA device minor numbers and the map from
minor numbers to device handles to get metrics for GPU devices attached to the
container.
As of the 4.7 kernel, the cpustats field returned from libcontainer
contains values for every possible cpu (including nonexistent ones).
The extra values are all 0s.
If we assume that hotplug events won't happen, we can get a more
accurage cpu count by using runtime.NumCPU and then ignoring any values
beyond that.
Cache the Docker thin pool name in the dockerFactory and pass it to each Docker container handler,
instead of calling `docker info` each time a new container handler is created, as the thin pool name
is extremely unlikely to change.
PerDiskStats reported from cgroups were not being surfaced into
prometheus. In order to properly correlate the metrics, we need to
assign a device label to each metric (which is the FS or device path).
Since blkio cgroup tracks devices, we create a synthetic device
`/dev/NAME` for the metric.
Assign a Device label to each PerDiskStat for the handlers up front, and
then surface the PerDiskStat values into the prometheus metrics. Report
two new metrics - total bytes read and total bytes written.
If CPU quota is configured (cpu.cfs_quota != -1) the CFS will provide
stats about elapsed periods and throtting in cpu.stats. This change
makes these information available as container_cpu_cfs_* metrics.
Ensure that kernel >= 4.4.0 or RHEL/Centos 7 kernel >= 3.10.0-366 exists before starting the thin
pool watcher. Prior versions have a bug in which reserving and releasing the metadata snapshot can
cause thin pool corruption.
Change working set calculation to usage - total_inactive_file, rather than
usage - total_inactive_anon - total_inactive_file. Since writes to tmpfs
get tracked as total_inactive_anon when swap is disabled, the old
calculation would under-report memory pressure.
See this Kubernetes issue for context:
https://github.com/kubernetes/kubernetes/issues/28619
