Running RightLink on RancherOS

Running the RightLink Agent in a container on RancherOS

RancherOS is a new take on how to optimize Linux for running docker containers. Similar to CoreOS all applications run in containers but unlike CoreOS there is no “Host OS” around of docker, rather every process in the entire system runs in a container: The docker daemon is pid 1, i.e., the first process created by the kernel at boot time.

The first blog post in this series described how RightLink integrates with docker and the second post described how to run RightLink in a container on CoreOS. This post focuses on running RightLink in a container on RancherOS, including how to monitor applications on RancherOS. Many of the considerations are the same as on CoreOS so this blog post will be a bit shorter than the previous ones.

The RightScale RightLink (version 10) agent is a lightweight agent that connects servers to the RightScale platform and enables remote management of the servers as well as monitoring and other metric collection. Generally, monitoring uses the standard Linux collectd agent, but recent versions of RightLink also incorporate some basic monitoring directly and this now extends to containers.

Overview

It turns out that even though everything is a container in RancherOS there is a notion of host vs. applications in that RancherOS runs two docker daemons: a system-docker and a user-docker. System services run in containers managed by system-docker and applications in containers managed by user-docker. One reason for the split is to ensure that system services are not affected by user actions in user-docker, such as terminating all containers. In the case of RightLink and collectd it makes the most sense to run them in a system-docker container.

In addition to running RightLink and collectd, the configuration described in this blog post also includes mysql as a sample application that is monitored by collectd. In the end we will have the following components shown in the diagram below:

Diagram of system and user docker

The first step is to launch RightLink in a container. The container itself is the same as the one used for CoreOS but the launching mechanism is different. In CoreOS the container is managed by systemd whereas in RancherOS it is launched using custom cloud-init configuration lines.

The cloud-init configuration is shown below and primarily consists of a rancher services section to describe the rightlink service. In this section, the restart statement configures the RightLink container to automatically restart should it exit and the labels ensure that RightLink is started in a system container after the user-docker is running and all cloud-init processes have completed. The volume mounts ensure that RightLink can communicate with system-docker and user-docker, and they create and mount a config area for collectd. Finally, the ports section sets up port forwarding on the system-docker bridge for the port used by RightLink: 172.18.0.0/16 is the network used by system docker and 172.17.0.0/16 is the network used by user-docker.

#cloud-config
rancher:
  services:
    rightlink:
      restart: always
      image: rightscale/rightlink:10.2.docker1
      command: "/usr/local/bin/rightlink -listen 0.0.0.0:88 -rootfs /host"
      labels:
        - io.rancher.os.detach=true
        - io.rancher.os.scope=system
        - io.rancher.os.after=cloud-init,docker
      volumes_from:
        - command-volumes
        - system-volumes
      volumes:
        - /usr/bin/docker:/usr/bin/docker:ro
        - /proc:/host/proc:ro
        - /sys/block:/host/sys/block:ro
        - /etc/collectd:/etc/collectd
      ports:
        - "172.18.42.1:88:88"

Launching collectd

Running collectd on RancherOS is the same as running it on CoreOS as described in the previous blog post. The key issues that need to be handled are:

The resulting docker run invocation looks as follows:

sudo system-docker run --name collectd \
  -d --restart=on-failure \
  -v /proc:/host/proc:ro \
  -v /etc/collectd:/etc/collectd \
  -e RS_INSTANCE_UUID=$RS_INSTANCE_UUID \
  -e RLL_IP=$rll_ip \
  rightscale/collectd /root/configure.sh

After launching RightLink and collectd the containers managed by system-docker look as follows:

$ sudo system-docker ps
CONTAINER ID  IMAGE                         STATUS         PORTS                    NAMES
3166ab24770e  rightscale/collectd           Up 22 seconds                           collectd
c32418c77c37  rightscale/rightlink:docker   Up 2 minutes   172.18.42.1:88->88/tcp   rightlink
a0d2e6cdac18  rancher/os-docker:v0.4.1      Up 3 minutes                            docker
bb16b371966c  rancher/os-console:v0.4.1     Up 3 minutes                            os_console_1
6adfb234b7ab  rancher/os-ntp:v0.4.1         Up 4 minutes                            ntp
016b7883e830  rancher/os-network:v0.4.1     Up 4 minutes                            network
9c47b7f00bf6  rancher/os-udev:v0.4.1        Up 4 minutes                            udev
a005d8f9e8c4  rancher/os-acpid:v0.4.1       Up 4 minutes                            acpid
aa6def0c6a30  rancher/os-syslog:v0.4.1      Up 5 minutes                            syslog

At this stage, user-docker is still idle with no containers. That will change as we launch MySQL as a sample application being launched and monitored by the RightScale platform:

Launching and Monitoring MySQL

Launching MySQL is straight-forward and uses the standard mysql container from dockerhub. The only special wrinkle used is to port-forward from port 3306 of the host’s interface on on the user-docker network:

docker run --name mysqld -d --restart=on-failure -e MYSQL_ROOT_PASSWORD=$MYSQL_PASS \
           -p $MYSQL_IP:3306:3306 mysql

The RightScript used to launch MySQL contains the above snippet plus the following one, which adds the mysql plugin to the collectd configuration:

plugins=/etc/collectd/plugins

# add the mysql plug-in
cat <<-EOF >$plugins/mysql.conf
LoadPlugin mysql
<Plugin mysql>
  <Database mysql>
    Alias "$RS_INSTANCE_UUID"
    Host "$MYSQL_IP"
    User "root"
    Password "$MYSQL_PASS"
  </Database>
</Plugin>
EOF

# add monitoring of the mysqld process
sed -i 's/<Plugin processes>/<Plugin processes>\n  process "mysqld"/' $plugins/processes.conf

# restart collectd container to pick up new config
if [[ -e /run/systemd && -x /bin/systemctl ]];
then /bin/systemctl restart collectd        # coreos
else sudo system-docker restart collectd    # rancheros
fi

The end result is this container in the user-docker and some nice MySQL monitoring graphs:

$ docker ps
CONTAINER ID        IMAGE               STATUS              PORTS                        NAMES
913802fbe5e6        mysql               Up 54 seconds       172.18.42.1:3306->3306/tcp   mysqld

Screen shot displaying mysql monitoring graphs

Summary

Running RightLink on RancherOS ends up being relatively straight-forward even though RancherOS operates system daemons very differently from other Linux distributions. Thanks to the fact that RightLink 10 has the hooks necessary to run in a container it is “just” a matter of configuring it appropriately. Most of the configuration is straight-forward: persistent data needs to be placed in a mounted volume, access to the docker socket needs to be provided through bin-mounts, /proc needs to be mapped to /host/proc, etc.

The one configuration that does not have an obvious right answer is how to set-up the network rendez-vous ports: how do other containers, such as collectd, make requests to RightLink and how does collectd query processes in other containers for monitoring data. The solution used in this blog post is to bind ports to the host’s interface on the system docker bridge network, which basically uses this interface as a replacement for the standard localhost loopback interface. While this works well with the standard configuration of RancherOS, other solutions may be more appropriate when using different network configurations.

The configuration of collectd shown here is unique in that it shows how to perform application-level monitoring in a container environment without running a monitoring daemon in each container or running a monitoring side-car container next to each application container. The side-car approach is interesting if large clustered applications are dynamically deployed on many hosts and the monitoring is purely about the application: then it makes sense that the application’s footprint on each host (which may involve multiple containers) includes a monitoring container. However, in simpler set-ups where container placement is more static it is easier to run a single monitoring container per host.

If you want to try RightLink on RancherOS yourself, all the pieces are available in this ServerTemplate: RL10.2.docker1 MySQL RancherOS ServerTemplate. A slightly simpler RL10.2.docker1 RancherOS ServerTemplate without the MySQL pieces is also available.

This post concludes the mini-series on running RightLink10 in a docker environment. If you have suggestions, questions, or concerns about RightLink 10 and docker please don’t hesitate to comment or email us!