RancherOS is a new take on how to optimize Linux for running docker containers. Similar to CoreOS all applications run in containers but unlike CoreOS there is no “Host OS” around of docker, rather every process in the entire system runs in a container: The docker daemon is pid 1, i.e., the first process created by the kernel at boot time.
The first blog post in this series described how RightLink integrates with docker and the second post described how to run RightLink in a container on CoreOS. This post focuses on running RightLink in a container on RancherOS, including how to monitor applications on RancherOS. Many of the considerations are the same as on CoreOS so this blog post will be a bit shorter than the previous ones.
The RightScale RightLink (version 10) agent is a lightweight agent that connects servers to the RightScale platform and enables remote management of the servers as well as monitoring and other metric collection. Generally, monitoring uses the standard Linux collectd agent, but recent versions of RightLink also incorporate some basic monitoring directly and this now extends to containers.
It turns out that even though everything is a container in RancherOS there is a notion of host vs. applications in that RancherOS runs two docker daemons: a system-docker and a user-docker. System services run in containers managed by system-docker and applications in containers managed by user-docker. One reason for the split is to ensure that system services are not affected by user actions in user-docker, such as terminating all containers. In the case of RightLink and collectd it makes the most sense to run them in a system-docker container.
In addition to running RightLink and collectd, the configuration described in this blog post also includes mysql as a sample application that is monitored by collectd. In the end we will have the following components shown in the diagram below:
- RightLink runs in a system docker container and listens for local HTTP requests on the system docker bridge
- Collectd is launched from a RightScript to run in another system container
- Persistent configuration data for RightLink and Collectd is mounted as volumes
- Mysql is launched from a RightScript to run in a user-docker container and listens for connections on the user docker bridge
- Collectd connects to mysql to retrieve monitoring data via the user docker bridge
The first step is to launch RightLink in a container. The container itself is the same as the one used for CoreOS but the launching mechanism is different. In CoreOS the container is managed by systemd whereas in RancherOS it is launched using custom cloud-init configuration lines.
The cloud-init configuration is shown below and primarily consists of a rancher services section to describe the rightlink service. In this section, the restart statement configures the RightLink container to automatically restart should it exit and the labels ensure that RightLink is started in a system container after the user-docker is running and all cloud-init processes have completed. The volume mounts ensure that RightLink can communicate with system-docker and user-docker, and they create and mount a config area for collectd. Finally, the ports section sets up port forwarding on the system-docker bridge for the port used by RightLink: 172.18.0.0/16 is the network used by system docker and 172.17.0.0/16 is the network used by user-docker.
#cloud-config rancher: services: rightlink: restart: always image: rightscale/rightlink:10.2.docker1 command: "/usr/local/bin/rightlink -listen 0.0.0.0:88 -rootfs /host" labels: - io.rancher.os.detach=true - io.rancher.os.scope=system - io.rancher.os.after=cloud-init,docker volumes_from: - command-volumes - system-volumes volumes: - /usr/bin/docker:/usr/bin/docker:ro - /proc:/host/proc:ro - /sys/block:/host/sys/block:ro - /etc/collectd:/etc/collectd ports: - "172.18.42.1:88:88"
Running collectd on RancherOS is the same as running it on CoreOS as described in the previous blog post. The key issues that need to be handled are:
- using a base OS for the collectd container that has all the shared libraries needed for the desired collectd plugins
- mounting the host’s /proc onto /host/proc in the collectd container (while there is no “host OS” in RancherOS there still is a top-level /proc filesystem exported by the Linux kernel)
- modifying collectd’s source code to use /host/proc instead of /proc to collect data
- creating /etc/collectd to persist configuration data and mounting that in the collectd container
The resulting docker run invocation looks as follows:
sudo system-docker run --name collectd \ -d --restart=on-failure \ -v /proc:/host/proc:ro \ -v /etc/collectd:/etc/collectd \ -e RS_INSTANCE_UUID=$RS_INSTANCE_UUID \ -e RLL_IP=$rll_ip \ rightscale/collectd /root/configure.sh
After launching RightLink and collectd the containers managed by system-docker look as follows:
$ sudo system-docker ps CONTAINER ID IMAGE STATUS PORTS NAMES 3166ab24770e rightscale/collectd Up 22 seconds collectd c32418c77c37 rightscale/rightlink:docker Up 2 minutes 172.18.42.1:88->88/tcp rightlink a0d2e6cdac18 rancher/os-docker:v0.4.1 Up 3 minutes docker bb16b371966c rancher/os-console:v0.4.1 Up 3 minutes os_console_1 6adfb234b7ab rancher/os-ntp:v0.4.1 Up 4 minutes ntp 016b7883e830 rancher/os-network:v0.4.1 Up 4 minutes network 9c47b7f00bf6 rancher/os-udev:v0.4.1 Up 4 minutes udev a005d8f9e8c4 rancher/os-acpid:v0.4.1 Up 4 minutes acpid aa6def0c6a30 rancher/os-syslog:v0.4.1 Up 5 minutes syslog
At this stage, user-docker is still idle with no containers. That will change as we launch MySQL as a sample application being launched and monitored by the RightScale platform:
Launching and Monitoring MySQL
Launching MySQL is straight-forward and uses the standard mysql container from dockerhub. The only special wrinkle used is to port-forward from port 3306 of the host’s interface on on the user-docker network:
docker run --name mysqld -d --restart=on-failure -e MYSQL_ROOT_PASSWORD=$MYSQL_PASS \ -p $MYSQL_IP:3306:3306 mysql
The RightScript used to launch MySQL contains the above snippet plus the following one, which adds the mysql plugin to the collectd configuration:
plugins=/etc/collectd/plugins # add the mysql plug-in cat <<-EOF >$plugins/mysql.conf LoadPlugin mysql <Plugin mysql> <Database mysql> Alias "$RS_INSTANCE_UUID" Host "$MYSQL_IP" User "root" Password "$MYSQL_PASS" </Database> </Plugin> EOF # add monitoring of the mysqld process sed -i 's/<Plugin processes>/<Plugin processes>\n process "mysqld"/' $plugins/processes.conf # restart collectd container to pick up new config if [[ -e /run/systemd && -x /bin/systemctl ]]; then /bin/systemctl restart collectd # coreos else sudo system-docker restart collectd # rancheros fi
The end result is this container in the user-docker and some nice MySQL monitoring graphs:
$ docker ps CONTAINER ID IMAGE STATUS PORTS NAMES 913802fbe5e6 mysql Up 54 seconds 172.18.42.1:3306->3306/tcp mysqld
Running RightLink on RancherOS ends up being relatively straight-forward even though RancherOS operates system daemons very differently from other Linux distributions. Thanks to the fact that RightLink 10 has the hooks necessary to run in a container it is “just” a matter of configuring it appropriately. Most of the configuration is straight-forward: persistent data needs to be placed in a mounted volume, access to the docker socket needs to be provided through bin-mounts, /proc needs to be mapped to /host/proc, etc.
The one configuration that does not have an obvious right answer is how to set-up the network rendez-vous ports: how do other containers, such as collectd, make requests to RightLink and how does collectd query processes in other containers for monitoring data. The solution used in this blog post is to bind ports to the host’s interface on the system docker bridge network, which basically uses this interface as a replacement for the standard localhost loopback interface. While this works well with the standard configuration of RancherOS, other solutions may be more appropriate when using different network configurations.
The configuration of collectd shown here is unique in that it shows how to perform application-level monitoring in a container environment without running a monitoring daemon in each container or running a monitoring side-car container next to each application container. The side-car approach is interesting if large clustered applications are dynamically deployed on many hosts and the monitoring is purely about the application: then it makes sense that the application’s footprint on each host (which may involve multiple containers) includes a monitoring container. However, in simpler set-ups where container placement is more static it is easier to run a single monitoring container per host.
If you want to try RightLink on RancherOS yourself, all the pieces are available in this ServerTemplate: RL10.2.docker1 MySQL RancherOS ServerTemplate. A slightly simpler RL10.2.docker1 RancherOS ServerTemplate without the MySQL pieces is also available.
This post concludes the mini-series on running RightLink10 in a docker environment. If you have suggestions, questions, or concerns about RightLink 10 and docker please don’t hesitate to comment or email us!