With a docker setup running with Prometheus node exporter and a docker container running cAdvisor, I’ve been seeing error messages similar to the following repeatedly appearing in syslog:
Jan 14 15:15:25 dockerserver-live prometheus-node-exporter[603]: time="2021-01-14T15:15:25Z" level=error msg="Error on statfs() system call for \"/var/lib/docker/containers/719fe4c20d2d274bb034e914006ecfe6760d8aec98efdc8010c85a01cf4059aa/mounts/shm\": permission denied" source="filesystem_linux.go:57" Jan 14 16:23:25 dockerserver-live prometheus-node-exporter[28623]: time="2021-01-14T16:23:25Z" level=error msg="Error on statfs() system call for \"/var/lib/docker/overlay2/8c25cb3049b4cfc9bebfd4df0ea6104560155bed2c18a9bd75d21323931570f4/merged\": permission denied" source="filesystem_linux.go:57"
These errors are being generated by a Prometheus node exporter process running with the following args:
--collector.diskstats.ignored-devices=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\\d+n\\d+p)\\d+$ \
--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|run)($|/) \
--collector.netdev.ignored-devices=^lo$ \
--collector.textfile.directory=/var/lib/prometheus/node-exporter
I’m not sure where this block originally came from, but in our case it’s in /etc/defaults/prometheus-node-exporter and easily edited to fix the regular expressions. Specifcally, because there’s problems with permissions on the shm and overlay they can be added to the ignored-mount-point regular expression as below:
--collector.diskstats.ignored-devices=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\\d+n\\d+p)\\d+$ \
--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|run|var\/lib\/docker\/containers\/.*\/mounts\/shm|var\/lib\/docker\/overlay2\/.*\/merged)($|/) \
--collector.netdev.ignored-devices=^lo$ \
--collector.textfile.directory=/var/lib/prometheus/node-exporter
following this change and restarting the prometheus-node-exporter process stopped new entries appearing in our syslog as it should be ignoring both troublesome directories.
Leave a Reply