Monitoring Container and Node Metrics Using Prometheus, cAdvisor and Node-Exporter

Prometheus

Prometheus is one of the best open source tools out there to monitor aggregated metrics related to servers . There’s a misconception that it is a standalone tool which pulls all metrics related to a server but in reality it performs querying and aggregation of metrics while tools like cAdvisor and Node-exporter pull the metrics and push the metrics to prom. Usually this is followed by exporting data to grafana for visualistion and more visibility of the data collected. Let’s deep dive into setting everything up. We shall be using only docker-compose files to setup services because it’s just so much easier to setup stuff using docker.

Let’s assume there’s a server which has a few containers running and we want to monitor that server for container and node metrics. Node exporter is the tool you would setup in the server to pull node metrics . Docker-compose is given below:

version: '3.8'
networks:
  monitoring:
    driver: bridge
services:
  node-exporter:
   image: prom/node-exporter:latest
   container_name: node-exporter
   restart: unless-stopped
   volumes:
   - /proc:/host/proc:ro
   - /sys:/host/sys:ro
   - /:/rootfs:ro
   command:
   -'- path.procfs=/host/proc'
   -'- path.rootfs=/rootfs'
   -'— path.sysfs=/host/sys'
   -'-collector.filesystem.mount-points-    exclude=^/(sys|proc|dev|host|etc)($$|/)'
   expose:
   - 9100
   networks:
   - monitoring

Certain volumes would be mounted in order for node exporter to collect node metrics. For deploying cAdvisor, use the below docker-compose:

version: '3.8'
networks:
  monitoring:
    driver: bridge
services:
    cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    ports:
    - 8080:8080
    volumes:
    - /:/rootfs:ro
    - /var/run:/var/run:rw
    - /sys:/sys:ro
    - /var/lib/docker/:/var/lib/docker:ro

Next let’s setup prometheus . First let’s setup the initial config of prometheus to point it towards cAdvisor and node-exporter:

global:
 scrape_interval: 1m
scrape_configs:
 - job_name: “prometheus”
   scrape_interval: 1m
   static_configs:
   - targets: [“localhost:9090”]
 - job_name: cadvisor
   scrape_interval: 5s
   static_configs:
   - targets:
     - (IP of cAdvisor node):8080
 - job_name: “node”
   static_configs:
   - targets: ["(IP of node exporter node):9100"]

Save the file as prometheus.yml . This will be used to mount into the prometheus container while deploying. Prometheus and grafana docker-compose is as below:

version: '3.8'
networks:
  monitoring:
    driver: bridge
volumes:
  prometheus_data: {}
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: always
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--web.enable-lifecycle'
    expose:
      - 9090
    networks:
      - monitoring
  grafana:
    image: grafana/grafana:latest
    restart: always
    ports:
      - "0.0.0.0:3000:3000"
    networks:
      - monitoring

Once we bring this up, we can see all the metrics and visualisations in the grafana dashboard.

Hope this helps :) I also wrote a blog with how we can use similar strategies to use loki here . While in Kubernetes , all of this can be deployed easily using helm charts, there needs to be done some manual work while doing it for standalone nodes.