Monitoring Docker within EC2 with Prometheus

We use plenty of docker containers while we use microservice architecture. As a result, we have to monitor performance metrics for these containers, such as memory, I / O, cpu, etc.

Enter cAdvisor. Out of the box, cAdvisor exposes Prometheus metrics. In this guide we will accomplish the following:

  • Initialize an EC2 instance
  • Build a Docker Compose multi-container installation that includes containers running Prometheus, cAdvisor, and a Redis server.

Create EC2 instance and SSH into it

First let's create a new EC2 instance and SSH into it (see step by step instructions here on how to create and connect to EC2 instance). Then let's perform the following commands. Next, run the following command to ensure we have the latest

# perform updates
sudo yum update -y
# install docker
sudo amazon-linux-extras install docker
sudo yum install docker
# start docker
sudo service docker start
sudo usermod -a -G docker ec2-user

# install docker-compose
sudo curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

Then log out and log back in to enable some of the changes.

Configuring Prometheus

First, to scrape metrics from cAdvisor, you’ll need to configure Prometheus. Create and populate a prometheus.yml file with this configuration:

scrape_configs:
- job_name: cadvisor
scrape_interval: 5s
static_configs:
- targets:
- cadvisor:8080

Upload the file via ssh

scp -i monitoring-challenge.pem prometheus.yml ec2-user@ec1-3-22-223-41.us-east-2.compute.amazonaws.com:/home/ec2-user

Composing Docker

We will now need to build a Docker Compose configuration that defines which containers are part of our installation, which ports each container is exposed to, which volumes are used, and so on.

Create a docker-compose.yml file and populate it with this Docker Compose configuration:

version: '3.2'
services:
prometheus:
image: prom/prometheus:latest
container_name: prometheus
ports:
- 9090:9090
command:
- --config.file=/etc/prometheus/prometheus.yml
depends_on:
- cadvisor
cadvisor:
image: gcr.io/google-containers/cadvisor:latest
container_name: cadvisor
ports:
- 8080:8080
volumes:
- /:/rootfs:ro
- /var/run:/var/run:rw
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
depends_on:
- redis
redis:
image: redis:latest
container_name: redis
ports:
- 6379:6379

Docker Compose is instructed by this configuration to run three services, each corresponding to a Docker container:

  1. The prometheus service uses a local configuration file called prometheus.yml
  2. The cAdvisor service relies on a number of local volumes (/, /var / run, etc.) and exposes port 8080 (the default port for cAdvisor metrics).
  3. The Redis service is the standard server for Redis. Without any further setup, cAdvisor will automatically obtain container metrics from this container.

Upload the file via ssh

scp -i monitoring-challenge.pem docker-compose.yml ec2-user@ec1-3-22-223-41.us-east-2.compute.amazonaws.com:/home/ec2-user

Then run the following command within SSH

docker-compose up

Explore metrics

Now we can explore a suite of metrics at the following URLs:

  • http://localhost:8080 — web UI of cAdvisor is a useful interface to explore the kinds of items that are being tracked by cAdvisor, but it does not have an interface to explore container metrics.
  • http:/localhost:9090/graph — provides an interface for exploring container metrics

Summary

In this summary, we covered how to use Docker Compose that contains separate containers in a single installation, and how to deploy it within an EC2 instance. For further reading, see Prometheus documentation.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store