When cAdvisor starts up, it would read the `vendor` files in
`/sys/bus/pci/devices/*` to see if any NVIDIA devices (vendor ID: 0x10de) are
attached to the node. If no NVIDIA devices are found, this code path would
become dormant for the rest of cAdvisor lifetime. If NVIDIA devices are found,
we would start a goroutine that would check for the presence of NVML by trying
to dynamically load it at regular intervals. We need to do this regular
checking instead of doing it just once because it may happen that cAdvisor is
started before the NVIDIA drivers and NVML are installed. Once the NVML
dynamic loading succeeds, we would use NVML’s query methods to find out how
many devices exist on the node and create a map from their minor numbers to
their handles and cache that map. The goroutine would exit at this point.
If we detected the presence of NVML in the previous step, whenever a new
container is detected by cAdvisor, cAdvisor would read the `devices.list` file
from the container's devices cgroup. The `devices.list` file lists the
major:minor number of all the devices that the container is allowed to access.
If we find any device with major number 195 (which is the major number assigned
to NVIDIA devices), we would cache the list of corresponding minor numbers for
that container.
During every housekeeping operation, in addition to collecting all the existing
metrics, we will use the cached NVIDIA device minor numbers and the map from
minor numbers to device handles to get metrics for GPU devices attached to the
container.
This ensures each goroutine is given its own Netlink connection, and
presumably avoids having a message destined for one goroutine read by
another goroutine.
If in getCgroupPath in cgroups we have some other hierarchies after
"devices" using ".*" sometimes will cause matching wrong string as
a container path so we need negated character class here: "[^:]*".
e.g.
If cgroups string is
"153:name=systemd:/system.slice/docker-f55e7cad1fcc02f992e0c33c210ecdc6d641858a665f28370523c27c05bdde0e.scope,4:freezer,devices,name=container:/system.slice/docker-f55e7cad1fcc02f992e0c33c210ecdc6d641858a665f28370523c27c05bdde0e.scope,3:cpuacct,cpu,cpuset,name=fairsched:/system.slice/docker-f55e7cad1fcc02f992e0c33c210ecdc6d641858a665f28370523c27c05bdde0e.scope,2:memory:/system.slice/docker-f55e7cad1fcc02f992e0c33c210ecdc6d641858a665f28370523c27c05bdde0e.scope,1:blkio,name=beancounter:/system.slice/docker-f55e7cad1fcc02f992e0c33c210ecdc6d641858a665f28370523c27c05bdde0e.scope"
match[1] will be "blkio" but not:
/system.slice/docker-f55e7cad1fcc02f992e0c33c210ecdc6d641858a665f28370523c27c05bdde0e.scope
These fixes the commit:
4cbd91c761 Make getCgroupPath work in case of named or multi- hierarchies
v2: use negated character class, correct the example, remove .* on
either end as they don't do anything in FindSubmatch.
In case we have devices hierarchies mounted in named cgroup
or together with other hierarchy regexp parse will fail.
So after "devices" and before ":" can be name of cgroup or
other hierarchies names.
E.g.:
1) remount cgroups:
umount /sys/fs/cgroup/devices
mkdir /sys/fs/cgroup/named_cgroup
mount -n -t cgroup -o devices,name=named_cgroup cgroup
/sys/fs/cgroup/named_cgroup
2) add some task to nested device cgroup and check ps output
mkdir /sys/fs/cgroup/named_cgroup/test.slice
sleep 1000 &
[1] 22734
echo 22734 > /sys/fs/cgroup/named_cgroup/test.slice/tasks
ps -ao pid,cgroup | grep 22734
22734
14:devices,name=named_cgroup:/test.slice,1:name=systemd:/user.slice/user-1000.slice/session-1.scope
Signed-off-by: Pavel Tikhomirov <ptikhomirov@virtuozzo.com>
Remove spec-related fields from stat.
We can simplify the stats a bit further by handling Int and Float better.
But this was big enough change already.
Verified v1 and v2 spec/stats/appmetrics APIs.
Docker does not provide the rootfs path through docker inspect or statefile
and the path is dependent on the storage driver being used.
Instead of enumerating the storage drivers, we pick a pid from the container
and get the config from /proc/pid/root. Although a bit expensive, this method
works for non-docker containers too.
