Loading front-end code at will with Scoutfiles

A look at front-end code

It’s hardly news that modern apps are written as single page webapps consuming a remote API. It became good practice to write web apps as static clients consuming the same public API as any other clients, perhaps with the exception of being hosted on the service’s main domain. Because we create highly dynamic webapps that create interdependent experiences, we assume that the user will download the entire web application into their browser. As such we attempt to minimize the size of the client code so that the user has the fastest experience possible.

This leads us into a world where webapps more resemble traditional software: we can assume the client to have the entire source present; we have a build process and build artefacts and we tend to end up with some concept of versions (even if we practice some form of continuous delivery).

We’ve developed an interesting methodology of dealing with this. Here’s a quick video demo of this system, the rest of the article describe the details of the system.

In the video, I make some changes to the styles and demonstrate how I can get a version of the app running on production with a shareable link. If you have a Cloud Analytics account, you can try Pony Analytics right here:

https://analytics.rightscale.com/dashboard?scout=6239b1e2f8a1963eff137c5ca7f7520f5fda8bc0

I also show how we gain separation between front-end and back-end code, because each can be run with the production version of the other.

Overview

Our code is split into a back-end and front-end system. The back-end system is deployed using our Cloud Management product (we eat our own dog food). The front-end system is ‘deployed’ on every git push to GitHub. This is not visible to users, since which of these ‘deployments’ is visible is governed by a database record on the back-end. This field propagates to the entry point of the front-end: the index.html. From here we load what we call a Scoutfile, this gives us considerable flexibility to load any other ‘deployment’ if specified, or default to the ‘production’ deployment.

We wanted the ability for front-end engineers and designers to be able to run the front-end against our staging or production back-ends. This allows them to work on the front-end without having to go trough the trouble of having to setup the back-end (which is admittedly painful) and of acquiring realistic datasets in their development environment.

Once we had that ability, we realised that it would be nice to have the ability to deploy and test every single commit and be able to share these deployments in our team.

Hosting

We push code to GitHub. Travis CI picks up the push and builds, tests, packages and finally uploads our build artefacts to S3. We serve our build artefacts from S3, but for security reasons we have a reverse proxy set up that we load the assets through, so users always see assets loading from our IP addresses.

Scoutfile

The production app then only loads a tiny file that checks a scout url parameter. If the parameter is present, it will load the entire front-end assets from the S3 bucket, from the folder corresponding to the scout parameter. It can also be used to load assets served by a local server.

This enables a much more powerful method of communication in the team, where any change can be instantly and automatically previewed by any other team member.

Chrome Extension

To make this form of communication even easier, I wrote a Chrome extension, that adds preview buttons to GitHub Pull Requests:

UI Previewer Chrome Extension

When you click on the preview button, you are redirected to the production application with the front-end corresponding to the latest commit in the pull request is loaded.

The extension is generic, so if you wish to use a similar approach, you can get it here and change the settings so that it work for your setup.

Technical Overview

Here I would like to show some of the key pieces of this infrastructure. This is a highly technical bit, and essentially this is almost literal programming, so consider yourself warned.

index.html - The Stringent Gatekeeper

We render our single page app through our Rails app. This is not strictly necessary - our original concept simply had a global file in the root of the S3 bucket. But having Rails handle the entry point into the application means that Rails can redirect the user to the login page and can easily bootstrap the app with some initial data. But more importantly, we store the Git SHA of the production version of the app in the database. This means that only people who have write access to the production database can release a new version of the application to the customer (which given my pony example, is probably a good thing), whereas every commit can (and indeed is) written to the production S3 bucket.

To enable this all front-end request go through this controller

 class SinglePageAppsController < ApplicationController

First we check whether a user is logged in.

   before_filter :ensure_login
  helper_method :environment, :analytics_ui_sha

We render the view, passing in some information through helper functions:

   def index
    render :index, :layout => false, :formats => [:html]
  end

private

We pass in the GIT sha of the current release tag to load the correct default version of the UI.

   def analytics_ui_sha
    GlobalState.get_value('analytics_ui_sha')
  end

We also pass in a hash of values that allow us to bootstrap the app and we save one HTTP request.

   def environment
    {
      environment: Rails.env,
      csrf_token: csrf_token,
      CurrentUser: {
        id: @current_user.id,
        first_name: @current_user.first_name,
        last_name: @current_user.last_name,
        company: @current_user.company
      }
    }
  end
end

This renders the following view:

<!DOCTYPE html>
<html ng-app="CloudAnalytics">
  <head>
    <meta charset="utf-8">
    <title>RightScale Cloud Analytics</title>

We use this simple method to inject into our code bootstrapping information.

    <script charset="utf-8">
      window.CA = <%=raw environment.to_json %>;
    </script>
  </head>
  <body>

Then we setup a place for our application to render in.
```
   <div ui-view></div>
      
```

Finally we inject the scoutfile. We have different endpoints for local environments, testing environments and production/staging environments.

   <script
     src="<%= Rails.application.config.assets_endpoint -%>
        <%= analytics_ui_sha %>/scripts/scoutfile.js">
   </script>
  </body>
</html>

Rails.application.config.assets_endpoint is a configuration option that in production points to a proxy server that ensures that all requests are through RightScale IP addresses. In development it points to a different server on localhost.

Travis - The Ingenious Builder

Travis is a system that requires surprisingly little code to get very powerful things done:

Travis will run npm test by default, which we have configured to a grunt ci task. I won’t post the code to of our Gruntfile, as there is no way to simplify that file. I’ll take this opportunity to say that Grunt is terrible for building complex applications, so if you try this at home, use something else.
```
language: node_js
node_js:
  - "0.10"
deploy:
  - provider: s3
```
Use an AWS IAM that has only write access to your production S3 bucket. Travis will encrypt it for you, so you can then have it checked into your repo.
```
    access_key_id: MY_AWS_ACCESS_KEY_ID
    secret_access_key:
      secure: MY_ENCRYPTED_AWS_SECRET_ACCESS_KEY
    bucket: my-production-bucket
```
The last task in our grunt pipeline find the SHA of the current commit, and copies the build artefacts to a directory called by that name. So for commit 6239b1e2f8a1963eff137c5ca7f7520f5fda8bc0 grunt will copy all artefacts into ./dist/relase/6239b1e2f8a1963eff137c5ca7f7520f5fda8bc0, where Travis will find them and copy them to S3.
```
    local-dir: dist/release
    acl: public_read
    skip_cleanup: true
    region: us-west-2
    on:
      repo: rightscale/analytics_ui
      all_branches: true
```

scoutfile.js - The Courages Source Finder

Inspired by the influential post by Alex Sexton, The scoutfile is there to figure out what version of the app to load and then to do it.

First, we have to figure out where to load our assets from (we want to persist the setting to localStorage):
```
 function findBase() {

      
```

If there is a scout parameter in the url, we will try to do something with it:

   var match = window.location.search.match(/scout=(.+?)(&|$)/),
      scout = match && decodeURIComponent(match[1]);
  if (scout) {

If the scout param is dev, then we persist the setting and return localhost.

     if (scout === 'dev') {
      localStorage['scout-dev'] = 'yes';
      return localhost();
    } else {

Otherwise we assume it is either a URL or a Git SHA. If it is a domain, we need to validate it against a whitelist for security reasons.

       if (scout.match(/^https:\/\/analytics-assets\.rightscale\.com\//i)) {
        persist(scout);
        return scout;

If it is a Git SHA, we complete it into our default assets domain (which is in fact a proxy to our production S3 bucket).

       } else if(scout.match(/^[a-f\d]{40}$/)) {
        persist('https://analytics-assets.rightscale.com/' + scout);
        return 'https://analytics-assets.rightscale.com/' + scout;
      }
    }
  }

There is no scout param, we will repeat the process with whatever is stored in local storage.

   if (localStorage['scout-dev'] === 'yes') {
    return localhost();
  }
  if (localStorage['scout-base']) {
    return localStorage['scout-base'];
  }

If neither are set, we return the default.
```
   return defaultBase();

      
```
We want to save whatever the user passed through the URL, so that refreshing the app doesn’t change the UI version, so we persist it into localStorage.
```
   function persist(base) {
    localStorage['scout-dev'] = 'no';
    localStorage['scout-base'] = base;
  }

      
```

We need to support HTTPS so that a production back-end (always running with SLL) can load local assets. However, for that we need to run our dev server on two different ports and we need to load the assets from the appropriate one.

   function localhost() {
    var protocol = document.location.protocol,
        port     = (protocol === 'https:' ? '9001' : '9000');
    return protocol + '//localhost:' + port + '/' + gitSha + '/';
  }
}

The default base includes the Git SHA from which the scoutfile was loaded. To find that we need to get the <script> element from which the scoutfile was loaded and parse the src attribute.

 function defaultBase() {
  var scoutUri = document.querySelector('script[src$="scoutfile.js"]')
                    .getAttribute('src'),
      gitSha   = scoutUri.match(/[a-f\d]{40}/i)[0];
  return scoutUri.replace(/scripts\/scoutfile.js$/, '');
}

Next we need to setup our loading infrastructure. We will be inserting our scripts after the first script in the document.

 var firstTag = document.getElementsByTagName('script')[0],
    callbacks = [],
    scriptsLoaded = false;
function onScriptsLoaded() {
  for (var i = 0, l = callbacks.length; i < l; i++) {
    callbacks[i]();
  }
  scriptsLoaded = true;
};

Next we want a helper function to add a tag to the HEAD element.

 function putTag(tag) {
  firstTag.parentNode.insertBefore(tag, firstTag);
}

We then make a specialised maker function for each type of asset being loaded.

Scripts on load trigger the callback sent to them.

 function addScriptTag(url, cb) {
  var tag = document.createElement('script');
  tag.src = url;
  if (cb) {
    tag.addEventListener('load', cb, false);
    tag.addEventListener('error', cb, false);
  }
  putTag(tag);
}

CSS also has a media attribute, which it is better to set.

 function addCssTag(url, mediaType) {
  var tag = document.createElement('link');
  tag.rel = 'stylesheet';
  tag.href = url;
  tag.media = mediaType;
  putTag(tag);
}

Once we have this setup, we can proceed to load our files.

 window.loadScripts = function(scripts) {
  addScriptTag(base, scripts[0], function() {

Finally we also don’t want to keep every single build artefact forever (as they can be quite large), and if we send versions of the app to customers we don’t want the app to silently fail. Therefore we have implemented a canary system, that will cause the default ui to be loaded in case the requested UI cannot be reached.

The scoutfile-canary.js looks like this: window.scoutfileCanary = true. This allows us to detect that the load was succesful, since if there was a silent failure, this variable would be false.
```
     if (!window.scoutfileCanary) {
      base = defaultBase();
      ls.removeItem('scout-base');
      ls['scout-dev'] = 'no';
    }

      
```
Now we know we are ready to load everything, we go through our list of scripts to load and recursively we build up the callback. This guarantees scripts will be loaded in the correct order (if order of scripts doesn’t matter, some performance can be gained by modyfing this bit).
```
     scripts.slice(1).reverse().reduce(function(callbackChain, script) {
      return function() {
        addScriptTag(base, script, callbackChain);
      };
    }, onScriptsLoaded)();

      
```

CSS can be loaded asynchronously, as there order matters less.

     addCssTag(base, 'styles/main.css', 'all');
    addCssTag(base, 'styles/print.css', 'print');
  });
};

Finally we trigger the load: if we are in local mode, then we need to ask the server for the list of dependencies. This is a JSONP request where the response is an array of files to load.
```
 if (localStore['scout-dev'] === 'yes') {
  addScriptTag(base, 'scripts/dependencies.json?callback=loadScripts');
} else {

      
```

In production our dependencies are known, as they are built into a single file.

   window.loadScripts(['scripts/scoutfile-canary.js',
    'scripts/build.min.js']);
}

ui_version_indicator_directive.js - The Handy Informant

Finally we have a tiny Angular directive that shows what version of the assets we’re running:

 angular.module('Shared').directive('uiVersionIndicator', function() {
  return {
    restrict: 'E',

I’ve included the template inline, but better practice would be having it in a separate file. This also assumes some other common directives, such as a tooltip implementation.

     template: '<div class="user_item ui_indicator" ng-if="should_show">' +
              '  <a ng-click="goToDefault()"' +
              '    tooltip="Click to go back to the default UI version"' +
              '    data-placement="bottom">UI version: ' +
              '  </a>' +
              '</li>',
    link: function(scope) {

Git SHA’s are commonly abreviated to 8 charachters, which makes it a bit easier on the eyes.

       function uiVersion(base) {
        var sha = base.match(/\w{40}/);
        return sha ? sha[0].substring(0, 8) : base;
      }

Next we check localStorage to find what version of the UI the user is on. This is basically a reversal of the process in the scoutfile above.

       if (localStorage) {
        if (localStorage['scout-dev'] == 'yes') {
          scope.ui_version = 'development';
          scope.should_show = true;
        } else if (localStorage['scout-base']) {
          scope.ui_version = uiVersion(localStorage['scout-base']);
          scope.should_show = true;
        } else {
          scope.should_show = false;
        }
      } else {
        scope.should_show = false;
      }

Finally we create a function to get out of a customised UI, which prompts the user and then removes the relevant items from localStorage and then redirects them to a non-scoutified URL.

       scope.goToDefault = function() {
        if (confirm('Would you like to revert to the default UI?')) {
          localStorage.removeItem('scout-dev');
          localStorage.removeItem('scout-base');
          location.href = location.href.replace(/scout=.+?(&|$)/, '');
        }
      };
    }
  };
});

Conclusion

This system allows for a big degree of flexibility, communication and easy customer prototypes. BTW, we’re looking for a front-end engineer to join our team if this is something that interests you.