Using RightLink with Docker

Using the RightLink 10 agent to inventory Docker container info and enable container monitoring.

One of the challenges faced by many users deploying Docker on a larger scale is keeping track of all the containers that are currently running or that were running at some point. In addition, while containers are running they typically need to be monitored and have application-level alerts configured. These two problems are now made simpler by Docker integrations built into new versions of the RightScale RightLink agent.

The RightScale RightLink (version 10) agent is a lightweight agent that connects servers to the RightScale platform and enables remote management, monitoring and other metric collection. Generally, monitoring uses the standard Linux collectd agent, but recent versions of RightLink also incorporate some basic monitoring directly and this now extends to containers.

Container Inventory

The need for keeping track of the containers running on each server comes from the fact that a container is in between a virtual machine and an application. When deployments only use virtual machines, the virtual machine image and the RightScale ServerTemplate used provide relatively detailed information about which software is installed on a VM and also which applications should be expected to be running. That means that questions, such as “which versions of OpenSSL are we using?” or “how many VM instances are running our web front-end?” can be answered with good precision using only that information.

The use of containers, however, adds another level of encapsulation and typically also more dynamism to the problem. Each container starts from a base operating system image, then installs additional software, and finally pre-configures the application(s) to be run in the container. On a server running containerized workloads, the software being used is the sum of every program and shared library in any running containers; although software continues to run directly on the host, it is typically not directly involved in delivering service. Thus the question about the number of web front-ends tends to turn into “how many web front-end containers do we have running across all instances?” and the question about OpenSSL turns into “which versions of OpenSSL do we have installed among all the containers that are running?” On a single host with five containers, we might be running five different versions of OpenSSL!

These questions lead to the desire to inventory all running containers with information about their provenance (i.e. their images) and to keep track of changes over time. This inventory would allow the above questions to be answered by counting the number of web front-end containers currently running or examining all images for the version of openssl installed.

The first step in constructing this centralized inventory is to capture the information about what is running in containers and this is where a new version of the RightScale Rightlink 10 agent comes into play. It connects to the local Docker daemon and reads a feed of container lifecycle events so it can keep track of new container launches and container terminations. Significant events in this feed are then sent to the RightScale platform in the form of tags that are applied to the server.

For example, a server running 3 containers has the following tags:

rs_docker:c-2a0eccfcde2965992bac7e76388f69ad738a6dcca1d433bd57f1244c8d7b778a=
["/collectd","655966ce257d687c4d0498a19f1904beb423ba0c09ebe29f9772f44b46bc008b",0,0]    

rs_docker:c-acb63f2cf6779fede54f25752f2785670105245abc63d3bb60edbaae1ffcb980=
["/mysql","a5ad9eb2ff48c02a17c62c3f28769b1a33fd50abd6752c55b138fa5e8aa76096",0,0]       

rs_docker:c-b5907cbe3c158280dd881bd9b2d57cb44b9118243b30403a1144dd450f7b06b4=
["/rightlink","e2024a55f7b193ac6e86c7b5955e1319963611c1d12fd2716a75710cf844a3cb",0,0]   

rs_docker:i-655966ce257d687c4d0498a19f1904beb423ba0c09ebe29f9772f44b46bc008b=
["rightscale/collectd:latest"]  

rs_docker:i-a5ad9eb2ff48c02a17c62c3f28769b1a33fd50abd6752c55b138fa5e8aa76096=
["mysql:latest"]        

rs_docker:i-e2024a55f7b193ac6e86c7b5955e1319963611c1d12fd2716a75710cf844a3cb=
["rightscale/rightlink:docker"]

rs_docker:host=["F36P:BYUQ:ZSI4:4FF3:FPIU:BAFX:RN3C:4ZAX:WQZ3:4UWJ:CQYN:4WFZ",2003,1]   

The first three tags describe the containers running with their SHA, name, image and cpu/memory quotas. We thus have a collectd, a mysql, and a rightlink container on this host. The next three tags describe the images for these containers and the final tag provides the UUID of the docker host, memory (in MB), and cpu cores.

While the tags are not very human-friendly in this format, they can be consumed by scripts that can provide information across all hosts in an account or perform actions to specific containers. For example, the following snippet lists the names of all images on all servers with their SHA, which allows the number of different versions of an image to be tallied:

$ rsc --xm '.name:contains("rs_docker:i")' -h us-3.rightscale.com -r $RS_KEY \
    cm15 by_tag /tags/by_tag \
    'include_tags_with_prefix=rs_docker:' \
    'tags[]=rs_docker:host=*' \
    'resource_type=instances' | \
  ruby -ne 'puts "#{$_[/=.\\"([^:]+):/,1]}: #{$_[/:i-(\w+)=/,1]}"'

Sample output with two servers running looks as follows:

rightscale/rightlink: 266f9bb521a29298e82efe83a2d5124980b83bd2b249a1c2ead16723bc1e3b5c
rightscale/collectd: 655966ce257d687c4d0498a19f1904beb423ba0c09ebe29f9772f44b46bc008b
rightscale/collectd: 655966ce257d687c4d0498a19f1904beb423ba0c09ebe29f9772f44b46bc008b
mysql: a5ad9eb2ff48c02a17c62c3f28769b1a33fd50abd6752c55b138fa5e8aa76096
rightscale/rightlink: e2024a55f7b193ac6e86c7b5955e1319963611c1d12fd2716a75710cf844a3cb

In this example, the difference in SHAs for the rightlink images (first and last line) indicates that the two servers are running different versions of the RightLink container.

The task of monitoring containers can be subdivided into three levels: monitoring processes running in containers, monitoring containers themselves, and performing application-level monitoring in containers.

The first is very simple and seemingly often overlooked: all processes running in containers show up as processes at the host level. Thus if a server has 10 apache processes running across a number of containers and the monitoring system is configured to capture the aggregated cpu utilization of the “apache” processes (or “httpd”, depending on distro) then it does not matter whether apache is running in containers: the monitoring will work the same way regardless. One limitation of this approach is that if two completely different application containers are deployed on a server and both use apache then the usage of the two cannot be distinguished.

In addition to monitoring the processes running in containers, it is possible to monitor the usage of all processes in a container in aggregate. For example, you can look at the total cpu usage of all processes running in a web front-end container. This level of monitoring is exposed by the linux kernel directly and is also collected by the docker daemon. RightLink can now poll the docker daemon every 20 seconds for this data and send it to the RightScale monitoring system. This monitoring is built into RightLink itself and does not rely on collectd. Pulling directly from the kernel is also possible, but that makes it more difficult to correlate the data with the names of containers that users see.

As an example, on a server with collectd, mysql, and rightlink containers the monitoring shows cpu, disk, and memory usage info for all three containers. The sample screenshot below displays the detail graph for the collectd container’s pagefaults.

Screen shot displaying the detail graph for the collectd container
pagefaults

Application-Level Monitoring

The last level is application-level monitoring of applications running in containers. Typically the approach used for application-level monitoring is to run collectd next to RightLink and send the monitoring data to the RightScale platform. For a few common applications, such as apache, nginx, and mysql, collectd contains built-in plugins. In order to collect data, collectd needs to be able to connect to these applications and send them requests/queries. With containers, the typical approach used is to bind the application’s port to one of the host’s network interfaces, such as localhost. This is common practice and generally already in place so other containers can access the apps in the containers.

For more custom monitoring, collectd supports plugins. These are typically scripts that are run in separate processes by collectd, that collect data from an application and then push the data to collectd from where it gets forwarded to the RightScale platform. There are two approaches possible in using these plugins. One is to run the plugin at the host level and to gather the application specific data by connecting to the application in the same way the built-in plugins do, e.g. through a mapped port. The second option, which may be easier from a software engineering perspective, is to run the plugin in the same container as the application. The benefit is that the plugin code then gets deployed and updated in sync with the application and there is no concern about which version of the plugin is installed on the host vs. which version of the application is running in the container.

The easiest way to run the monitoring plugin in the same container as the application is to use docker exec. In that case, collectd launches a plugin shim which performs a docker exec in order to run the actual plugin in the container itself.

Try it out

The RightScale RightLink agent with docker integration is now available in beta form as version 10.2.docker1. It is based on RightLink 10.2.1 but includes enhancements other than just the docker features. In order to boot a server the key is to add the following tag to the MultiCloud Image (MCI) or the server itself:

rs_agent:mime_include_url=https://rightlink.rightscale.com/rll/10.2.docker1/rightlink.boot.sh

Once the server is up and Rightlink is running, the following snippet will enable the docker monitoring features:

sudo rsc rl10 put_control /rll/tss/control enable_monitoring=all

And to enable the reporting of docker containers in the instance’s tags, use something like:

. <(sudo cat /var/run/rightlink/secret)
curl -sS -H "X-Rll-Secret: $RS_RLL_SECRET" -X POST \
  "http://localhost:${RS_RLL_PORT}/rll/docker/control?enable_docker=all"

(In the future the RightScale Client rsc will support this more elegantly.) You can also boot a demo Ubuntu server with all this baked into the ServerTemplate by importing: http://www.rightscale.com/library/server_templates/RL10-2-docker1-Ubuntu-Demo/lineage/56907. This server boots a standard Ubuntu image, installs and runs RightLink, installs docker, and enables all the docker related features in RightLink 10.

If you want to see everything in action, try

sudo docker run --name database -d -e MYSQL_ROOT_PASSWORD=secret55 mysql

Stay tuned for the next blog post in this series describing how to run RightLink 10 in a container on CoreOS.