Browsed by
Tag: prometheus

Attaching Docker Swarm Services to an Overlay Network

Attaching Docker Swarm Services to an Overlay Network

When I originally configured Prometheus with a variety of exporters I had it scraping ports on a specific docker swarm host. This is dangerous as if that host goes down the underlying service will pop back up on a different host but Prometheus won’t be able to scrape it. I considered using haproxy to round robin onto the docker swarm nodes, but Kubernetes can resolve services by service name – is there no way to do this in Docker Swarm?

There is, but unlike Kubernetes the services can’t resolve each other by default. We must create a specific network and attach the services to it.

Before:

/prometheus $ nslookup unifi_exporter
Server:    127.0.0.11
Address 1: 127.0.0.11

nslookup: can't resolve 'unifi_exporter'

Create overlay network:

sudo docker network create -d overlay monitoring
tb3iw12k7xaw5olz7rasdcnm0

Redeploy Prometheus on network:

docker service create --replicas 1 --name prometheus \
    --mount type=bind,source=/data/docker/prometheus/config/prometheus.yml,destination=/etc/prometheus/prometheus.yml \
    --mount type=bind,src=/data/docker/prometheus/data,dst=/prometheus \
    --publish published=9090,target=9090,protocol=tcp \
    --network monitoring \
    prom/prometheus

Redeploy our exporter, this time attached to the overlay network. Note we no longer need to publish a port.

docker service create --replicas 1 --name unifi_exporter \
    --mount type=bind,src=/data/docker/unifi-exporter/config.yml,dst=/config.yml \
    --mount type=bind,src=/etc/ssl,dst=/etc/ssl,readonly \
    --replicas=1 \
    --network monitoring \
    louisvernon/unifi_exporter:0.4.0-18-g85455df -config.file=/config.yml

Confirm Prometheus can resolve the exporter by service name:

/prometheus $ nslookup unifi_exporter
Server:    127.0.0.11
Address 1: 127.0.0.11

Name:      unifi_exporter
Address 1: 10.0.1.15
Unifi to Grafana (using Prometheus and unifi_exporter)

Unifi to Grafana (using Prometheus and unifi_exporter)

Documenting the process of getting this up and running. We already had Prometheus and Grafana running on our docker swarm cluster (we promise to document this all one day).

There was only one up to date image of unifi_exporter in DockerHub and it had no documentation so we were not comfortable using it.

1) Download, build and push unifi_exporter.

$ git clone [email protected]:mdlayher/unifi_exporter.git
...
$ cd unifi_exporter
$ sudo docker build -t louisvernon/unifi_exporter:$(git describe --tags) . # yields a tag like 0.4.0-18-g85455df
$ sudo docker push louisvernon/unifi_exporter:$(git describe --tags)

2) Create read only admin user for unifi_exporter service:

3) Create config.yml on storage mounted on dockerswarm node. In our case we have a glusterfs volume mounted across all nodes. If you are using the self-signed cert on your unifi controller then you will need to set insecure to true.

$ $ cat /data/docker/unifi-exporter/config.yml 
listen:
  address: :9130
  metricspath: /metrics
unifi:
  address: https://unifi.vern.space
  username: unifiexporter
  password: random_password
  site: Default 
  insecure: false
  timeout: 5s

4) Deploy to docker swarm. The docker image does not contain any trusted certs, so we mounted the host certs as readonly.

$ docker service create --replicas 1 --name unifi_exporter \
    --mount type=bind,src=/data/docker/unifi-exporter/config.yml,dst=/config.yml \
    --mount type=bind,src=/etc/ssl,dst=/etc/ssl,readonly \
    --publish 9130:9130 \
    --replicas=1 \
    louisvernon/unifi_exporter:0.4.0-18-g85455df -config.file=/config.yml

5) You should see something like this from the logs (we use portainer to quickly inspect our services).

2018/06/12 01:10:47 [INFO] successfully authenticated to UniFi controller
2018/06/12 01:10:47 Starting UniFi exporter on ":9130" for site(s): Default

First time around (before we bind mounted /etc/ssl) we had an x509 error due to the missing trusted certs..

6) Add unifi_exporter as a new target for prometheus.

$ cat /data/docker/prometheus/config/prometheus.yml
...
  - job_name: 'unifi_exporter'
    static_configs:
      - targets: ['dockerswarm:9130']
        labels:
          alias: unifi_exporter
...

7) Point your browser at http://dockerswarm:9130/metrics and make sure you see stats. In our case the payload was 267 lines.

8) Restart the prometheus service: `docker service update –force prometheus`

9) Hop on over to prometheus to make sure the new target is listed and UP: http://dockerswarm:9090/targets

10) Finally we import the dashboard into Grafana. Our options are a little sparse right now, but this dashboard gives us somewhere to start. we made some tweaks to this to make it multi-AP friendly with some some extra stats:
Unifi-1516201148080

The result:

Setup node_exporter on Proxmox

Setup node_exporter on Proxmox

node_exporter is one of the most useful exporters for your Prometheus/Grafana installation, providing a wealth of statistics about the state of your servers/nodes.

These are the steps we used to install node_exporter on our Proxmox nodes.

Download and extract binary:

$ wget https://github.com/prometheus/node_exporter/releases/download/v0.16.0/node_exporter-0.16.0.linux-amd64.tar.gz
...
$ tar xvf node_exporter-0.16.0.linux-amd64.tar.gz
$ cd node_exporter-0.16.0.linux-amd64/

Create user to run node_exporter

$ useradd --no-create-home --shell /bin/false node_exporter

Copy binary to /usr/local/bin and modify owner:

$ cp node_exporter /usr/local/bin/.
$ chown node_exporter:node_exporter  /usr/local/bin/node_exporter

Create service entry for node_exporter. Create /etc/systemd/system/node_exporter.service:

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
# ExecStart=/usr/local/bin/node_exporter --collectors.enabled meminfo,loadavg,filesystem

[Install]
WantedBy=multi-user.target

Enable service and check it is running:

$ systemctl daemon-reload
$ systemctl start node_exporter
$ systemctl status node_exporter
systemctl status node_exporter
● node_exporter.service - Node Exporter
   Loaded: loaded (/etc/systemd/system/node_exporter.service; disabled; vendor preset: enabled)
   Active: active (running) since Sun 2018-06-10 14:10:42 MDT; 7s ago
 Main PID: 3456142 (node_exporter)
    Tasks: 5 (limit: 4915)
   Memory: 2.1M
      CPU: 9ms
   CGroup: /system.slice/node_exporter.service
           └─3456142 /usr/local/bin/node_exporter

Jun 10 14:10:42 superdave node_exporter[3456142]: time="2018-06-10T14:10:42-06:00" level=info msg=" - stat" source="node_exporter.go:97"
Jun 10 14:10:42 superdave node_exporter[3456142]: time="2018-06-10T14:10:42-06:00" level=info msg=" - textfile" source="node_exporter.go:97"
Jun 10 14:10:42 superdave node_exporter[3456142]: time="2018-06-10T14:10:42-06:00" level=info msg=" - time" source="node_exporter.go:97"
Jun 10 14:10:42 superdave node_exporter[3456142]: time="2018-06-10T14:10:42-06:00" level=info msg=" - timex" source="node_exporter.go:97"
Jun 10 14:10:42 superdave node_exporter[3456142]: time="2018-06-10T14:10:42-06:00" level=info msg=" - uname" source="node_exporter.go:97"
Jun 10 14:10:42 superdave node_exporter[3456142]: time="2018-06-10T14:10:42-06:00" level=info msg=" - vmstat" source="node_exporter.go:97"
Jun 10 14:10:42 superdave node_exporter[3456142]: time="2018-06-10T14:10:42-06:00" level=info msg=" - wifi" source="node_exporter.go:97"
Jun 10 14:10:42 superdave node_exporter[3456142]: time="2018-06-10T14:10:42-06:00" level=info msg=" - xfs" source="node_exporter.go:97"
Jun 10 14:10:42 superdave node_exporter[3456142]: time="2018-06-10T14:10:42-06:00" level=info msg=" - zfs" source="node_exporter.go:97"
Jun 10 14:10:42 superdave node_exporter[3456142]: time="2018-06-10T14:10:42-06:00" level=info msg="Listening on :9100" source="node_exporter.go:111"

Configure to start at boot:

$ systemctl enable node_exporter
Created symlink /etc/systemd/system/multi-user.target.wants/node_exporter.service → /etc/systemd/system/node_exporter.service.

Then you are done. You just need to setup a target in Prometheus.


Many of these steps were re-purposed from
https://www.digitalocean.com/community/tutorials/how-to-install-prometheus-on-ubuntu-16-04