Add white_listed_container_labels flag to white list container labels for prometheus metrics
Signed-off-by: Mikkel Oscar Lyderik Larsen <mikkel.larsen@zalando.de>
when cadvisor exports metrics for docker containers, there is a root cgroup (/) and cgroup for a docker container (/docker/uuid).
If docker container has a label on it, then this label is applied to all containers including the root container.
Because some containers don't have that label, the label will have an empty value. The reason for this is that Prometheus
does not allow sending a metric with the same name, but different labels, so cadvisor uses empty label values based on
the set of all labels for a given metric. This can result in many docker containers getting a large number of empty labels
because another container has that label.
If large number of docker labels vary a lot across images, then the set of labels will be enormous, where most of the labels
will be empty and have no value as prometheus metrics. To avoid this problem, a flag is provided that allows a user to
disable exporting docker labels as metrics.
Per-CPU stats are more expensive to transport and store, and that
level of detail is not required in many cases.
We export overall total cpu in the same metric as per-cpu, so that
dashboards which previously summed over cpu will work identically.
The structure is generic to support most hardware accelerators like
GPUs, TPUs etc.
Note that the prometheus label for id is called acc_id, so that it
doesn't conflict with some other label that maybe called id.
PerDiskStats reported from cgroups were not being surfaced into
prometheus. In order to properly correlate the metrics, we need to
assign a device label to each metric (which is the FS or device path).
Since blkio cgroup tracks devices, we create a synthetic device
`/dev/NAME` for the metric.
Assign a Device label to each PerDiskStat for the handlers up front, and
then surface the PerDiskStat values into the prometheus metrics. Report
two new metrics - total bytes read and total bytes written.
Only include cpu's in the Prometheus metrics endpoint that were used.
In recent kernels the cpuacct.statcpus behavior has changed to include
all possible cpu's. This can results in a high number of stale metrics
in the Prometheus endpoint.
This change generalizes the existing ContainerNameToLabelsFunc to allow the user to fully control all labels attached to exported Prometheus metrics. The existing behavior is available as DefaultContainerLabelsFunc and is used if no custom function is provided.
This will allow Kubernetes to filter out its internal Docker labels.
If CPU quota is configured (cpu.cfs_quota != -1) the CFS will provide
stats about elapsed periods and throtting in cpu.stats. This change
makes these information available as container_cpu_cfs_* metrics.
collection would stay counted in the gauge "container_scrape_errors",
making that particular metric useless. Instead, it must be reset on
every scrape.
This add Envs to container spec as a metadata source. When using prometheus
exposition format, they will be merged into the list of metrics' labels.
Also changed the cli flag to docker_env_metadata_whitelist, and add refenrences
of whitelist envs to API
Signed-off-by: Daniel Dao <dqminh@cloudflare.com>
With this change, all definitions and functionality for a given metric
are in a single place only instead of being distributed all over the
file. This makes it easier to inspect the code for correctness and
adding/changing metrics.
