As a developer, I’ve long struggled with the problem of how to deploy the applications I create. In an ideal world, I could spend all of my energy doing what I do best (building applications), and none of my energy dealing with operations concerns. That sounds like a good reason to have an operations team. But an operations team has the same problem because ideally, the operations team could spend all their time handling operations concerns, and none of their time worrying about how applications were created.
Deploying an application is largely an exercise in defining (or discovering) the relationship between an application and its environment. This can be a tricky and error-prone job because there is so much variety in applications, environments and the people who create them. If everyone involved could agree on an interface contract, we’d all save a lot of time and energy.
This is what PaaS has tried to do. Solutions like EngineYard, Heroku, Google App Engine, and OpenShift have sprung up to varying degrees of success. Of these, Heroku has had the largest impact on the way we think about software service deployment and what PaaS can do. You can find an entire ecosystem of software packages on GitHub designed to make your applications adhere to the tenets of The Twelve-Factor App. And that’s a good thing because we’re starting to see what life could be like in a world where apps fit neatly into PaaS-shaped boxes.
I was planning to write a lot about why I’m such a big fan of PaaS, and why Flynn makes sense, but I can’t do a better job than Jeff Lindsay already did. In case you haven’t heard of it, Flynn is a collection of services that essentially comprise a free and open-source PaaS. The project is crowd-funded and written almost entirely in Go. After playing with Flynn off and on for a few months, I was hooked on the idea, and I had to talk with Jonathan Rudenberg to hear more about what his plans are for the future of this project.
Flynn, when viewed as a whole system, is a Heroku-like PaaS that you run and manage yourself. But it’s more flexible than Heroku because it’s free and open-source. Another nice benefit is you can run services that bind to arbitrary TCP ports like data stores or mail servers. Eventually it will also be available as a paid service. Jonathan didn’t get too much more specific on the paid service idea, but did mention you’d be able to bring your own hardware and let the service manage Flynn for you.
Flynn is not a production-ready system at the moment, but that hasn’t stopped me from playing with it. flynn-demo is a project that uses Vagrant to launch a local demo of Flynn. Following along with the Vagrantfile, I was able to create a RightScale ServerTemplate to deploy flynn nodes. You can create a free RightScale account and import this ServerTemplate to try it out for yourself.
Bootstrapping a Flynn node is a pretty simple process thanks to flynn-bootstrap which does most of the work. flynn-bootstrap uses Docker to download and run images for each component of the system. Although you do have to provide a compatible Docker environment and pass some configuration to end up with a working system, and that’s worth a little explanation.
If you aim to try this out right now, you’ll need to install Docker v0.10.0. Flynn requires Docker to support hairpin NAT configuration, which is broken in more recent Docker releases. So we’re stuck with Docker v0.10.0 for now. Here’s the boot script I used to install Flynn’s dependencies on a plain Ubuntu 12.04 EC2 instance and bootstrap Flynn:
#!/bin/sh apt-get install -y --force-yes apt-transport-https apparmor sudo groupadd docker apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9 echo deb https://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list apt-get update apt-get install -y lxc-docker-0.10.0 # Fix for https://github.com/flynn/flynn/issues/13 echo 3600 > /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_close_wait docker pull flynn/host docker pull flynn/discoverd docker pull flynn/etcd echo "Configuring flynn with host ip: $HOST_IP_ADDRESS" docker run -d -v=/var/run/docker.sock:/var/run/docker.sock -p=1113:1113 flynn/host -external $HOST_IP_ADDRESS -force docker pull flynn/postgres docker pull flynn/controller docker pull flynn/gitreceive docker pull flynn/strowger docker pull flynn/shelf docker pull flynn/slugrunner docker pull flynn/slugbuilder docker run -e CONTROLLER_DOMAIN="$CONTROLLER_DOMAIN" -e=DISCOVERD=$HOST_IP_ADDRESS:1111 flynn/bootstrap
The Flynn stack is divided into two logical layers. The first layer (layer-0 or “the grid”) is designed to link a cluster of nodes to an abstract set of containerized processes. Flynn provides its own service discovery system called discoverd which is currently backed by etcd, and all services managed by Flynn are registered with discoverd. Flynn-host is the outermost grid component and acts as the glue between flynn and all the containerized processes managed by flynn. When flynn-host starts, it registers itself with discoverd and awaits instructions to do things like start and stop docker containers. You just need to provide it with an IP address that can be used to reach it and the IP addresses of other nodes if you’re building a multi-node system.
flynn-host is the Flynn host service. An instance of it runs on every host in the Flynn cluster. It is responsible for running jobs (in Linux containers) and reporting back to schedulers and the leader.
Flynn’s second layer (layer-1) takes care of higher level concerns. The controller provides an API to manage Flynn itself and application deployments, strowger handles TCP routing and gitreceive is an SSH server that receives git pushes and funnels them through the buildpack-based build process and on to flynn-host for deployment.
From the outside, you can manage Flynn using the controller API. You can do it on your own, or you can use flynn-cli which is what I’m using. But in order to actually reach the controller, you must give the controller a hostname. Strowger routes HTTP/HTTPS traffic by hostname so that multiple web services can share the same TCP port and the controller is just one of those services.
Lets see what happens when you actually run these bootstrap steps:
Pulling repository flynn/host Pulling repository flynn/discoverd Pulling repository flynn/etcd Configuring flynn with host ip: 10.225.143.38 26d55886b9713306c6e02baaf3e81812e38987a3940becbc395bf7251a7ba482 Pulling repository flynn/postgres Pulling repository flynn/controller Pulling repository flynn/gitreceive Pulling repository flynn/strowger Pulling repository flynn/shelf Pulling repository flynn/slugrunner Pulling repository flynn/slugbuilder Unable to find image 'flynn/bootstrap' locally Pulling repository flynn/bootstrap 01:38:04.728829 run-app postgres 01:38:06.855957 gen-random controller-key 01:38:06.859304 gen-random controller-key bb1fbd27bdc88fc897f286ed3646d522 01:38:06.859440 wait postgres-wait 01:38:22.556066 run-app controller 01:38:24.084380 wait controller-wait 01:38:25.118462 add-app controller-inception 01:38:25.197556 add-app postgres-app 01:38:25.219009 scale-app scheduler-scale 01:38:25.227535 run-app scheduler 01:38:25.834894 deploy-app shelf 01:38:26.675131 deploy-app strowger 01:38:26.720940 gen-ssh-key gitreceive-key 01:38:31.364098 deploy-app gitreceive 01:38:31.423730 gen-tls-cert controller-cert 01:38:36.533851 gen-tls-cert controller-cert pin: Hey8twCAAZtTJBc8QO7zyOhh+EOMRU6MHdAv5Sb32og= 01:38:36.534140 wait strowger-wait 01:38:36.544377 add-route gitreceive-route 01:38:36.599113 add-route controller-route 01:38:36.609110 log log-complete 01:38:36.611568 log log-complete Flynn bootstrapping complete. Install flynn-cli and paste the line below into a new terminal window: flynn server-add -g localhost:2201 -p Hey8twCAAZtTJBc8QO7zyOhh+EOMRU6MHdAv5Sb32og= default https://flynn.lithostech.com:8081 bb1fbd27bdc88fc897f286ed3646d522
Pay close attention to the very last line because it contains all the connection parameters needed to talk to the flynn controller on this server. Actually not quite because there’s a bug in this output. It should actually read:
flynn server-add -g flynn.lithostech.com:2222 -p Hey8twCAAZtTJBc8QO7zyOhh+EOMRU6MHdAv5Sb32og= default https://flynn.lithostech.com bb1fbd27bdc88fc897f286ed3646d522
But that’s ok because as long as you stay on the rails, it all works great from here. The full Flynn stack is running on a single server and waiting for you to connect and deploy some apps. Here’s a quick walkthrough:
Start by adding your public SSH key so gitreceive can authenticate you:
$ flynn key-add Key 15:2d:8e:5a:32:ba:88:74:f1:ad:9d:19:ea:10:1e:45 added
You’ll need an app to deploy. Flynn has an example app you can use:
$ git clone https://github.com/flynn/nodejs-flynn-example Cloning into 'nodejs-flynn-example'... remote: Reusing existing pack: 18, done. remote: Total 18 (delta 0), reused 0 (delta 0) Unpacking objects: 100% (18/18), done. Checking connectivity... done.
From your app’s git repository, run ‘flynn create example’ which creates an app named ‘example’ on the remote flynn server automatically and adds the remote server as a git remote named ‘flynn.’
$ flynn create example Created example
Now you’re ready to deploy. Just push to the remote named ‘flynn’ and flynn will deploy your app.
$ git push flynn master The authenticity of host '[flynn.lithostech.com]:2222 ([220.127.116.11]:2222)' can't be established. ECDSA key fingerprint is 9c:7a:48:42:cc:1c:d5:e7:ed:d4:47:79:b2:15:df:4d. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '[flynn.lithostech.com]:2222,[18.104.22.168]:2222' (ECDSA) to the list of known hosts. Counting objects: 18, done. Delta compression using up to 4 threads. Compressing objects: 100% (17/17), done. Writing objects: 100% (18/18), 1.84 KiB | 0 bytes/s, done. Total 18 (delta 6), reused 0 (delta 0) -----> Building example... -----> Node.js app detected -----> Requested node range: 0.10.x -----> Resolved node version: 0.10.29 -----> Downloading and installing node -----> Writing a custom .npmrc to circumvent npm bugs -----> Installing dependencies npm WARN package.json firstname.lastname@example.org No description npm WARN package.json email@example.com No repository field. npm WARN package.json firstname.lastname@example.org No README data -----> Cleaning up node-gyp and npm artifacts -----> Building runtime environment -----> Discovering process types Procfile declares types -> web -----> Compiled slug size is 5.2M -----> Creating release... =====> Application deployed To ssh://email@example.com:2222/example.git * [new branch] master -> master
By default, there are no processes actually running your app, so you’ll need to add some web workers to run the web service. Just as in Heroku and Foreman, you define your workers in a Procfile. The one for this example app defines only one worker named ‘web.’ Scale it up to three workers using flynn scale.
$ flynn scale web=3
Now your app is running, but you still can’t reach it until you tell Flynn which hostname should be routed to your application. Add an HTTP route for your new app and you should be able to hit it with curl. Obviously, you’ll need to make a corresponding DNS entry as well:
$ flynn route-add-http example.lithostech.com http/ae2ae90b2946e116bb8c6b4b52933e22 $ curl example.lithostech.com Hello from Flynn on port 55008 $ curl example.lithostech.com Hello from Flynn on port 55009 $ curl example.lithostech.com Hello from Flynn on port 55007
You can see that Flynn has set up three web workers inside three Docker containers and exposed them on sequential TCP ports on the host server. Strowger routes incoming HTTP requests to the workers at random, but having looked at the code, it looks like other routing methods could be added without too much trouble.
Keep in mind, this is a young project with lofty goals, so there are some rough edges. But I’m impressed with what I’ve seen so far. The Flynn team, and especially Jonathan, has been more than helpful in helping me understand this system. I can’t wait to see how this shapes up.