Announcing Cloud Analytics APIs

Things we learnt building our APIs.

Being a back-end engineer is hard. Not because the work is harder, or people expect more of you. It’s just harder to show what you have been working on; but not today! Today I’m proud to announce the availability of the Cloud Analytics public APIs. A part of me wants to give you a link to the API Overview and the API reference docs so you can just dive in and provide feedback.

Background

When we started building Cloud Analytics, our aim was to rapidly ship something so we could answer the following question: Should RightScale invest in Cloud Analytics as a new product? We were able to validate that question quickly by keeping the architecture very basic, mostly centred around an existing reporting app that was in production (that we hacked new functionality into), and starting a new cloud_analytics Rails app where everything else lived.

Since launching Cloud Analytics, we’ve learnt a lot about what our users want, what has and hasn’t worked well in terms of implementation and performance of the product, and what integration points our Cloud Management and Self-Service products needs with Cloud Analytics. Using this knowledge, we’ve been evolving our architecture, where our aim has been to:

  1. Expose publicly documented APIs that can be used by everyone, including the Cloud Analytics UI, customers and other RightScale product development teams.

  2. Break up Cloud Analytics into loosely coupled but highly aligned key components so they are easy to develop, test, deploy, debug and scale as independently as possible.

Things we learnt

We first split out the Cloud Analytics UI into its own system that could be loaded at will with a Scoutfile. We then set about to develop the new systems…

Running two systems in parallel for a while is challenging but sometimes necessary

We couldn’t stop all development on the old system while we developed and migrated data to the new system as we needed to also keep shipping new features that customers wanted, and fix bugs in the old system. We also didn’t want to do a big-bang release where we killed the old system and flipped all traffic to the new system as that would have caused the initial release to take a lot longer (since everything had to be ready). So instead, we decided to deploy the new system in parallel where two tricks helped us:

  1. The load balancers directed traffic to the new system when an X-API-VERSION HTTP header was present. This meant that we were able to design/implement/release chunks of the API, and update our UI to use the API, during our weekly releases over several months. By speeding up the move and reducing development work on the previous system, we tried to reduce the amount time we spent playing catch-up.

  2. We built a system so we could re-process any account’s data using the new system but store the data against the negative account ID in our datastore, so if your account ID was 1234, account -1234 could have a duplicated version of your data that was being processed using our new system. This enabled us to compare the old vs. new data live in production, and flip your account to use the new data-processing system once we confirmed that the data between account 1234 and -1234 matched.

Start consuming your own APIs as soon as possible

By getting people to use your APIs, it’s easier to work out the kinks. When we started building our first few resources we made various assumptions about how they would be used, but when we started updating our front-end, it became apparent that we needed to add various other options.

Having a good API Design-Review-Implement flow is crucial

Over the years, we’ve transitioned from “let’s just make a Rails controller that returns JSON” to “let’s design the API in GitHub Wikis” to “let’s design the API in a Confluence Wiki doc so people can discuss/comment on it” to “let’s design the API in YARD” to using Grape and finally settling on Praxis. The Design-Review-Implement flow that Praxis enables lets us design an API in detail, gather feedback, and implement the API at the same time that the UI is being implemented (using the API docs and mocked-data). Praxis’ auto-generated API docs also make it simpler to see where common patterns are applied differently so APIs can be kept consistent.

The Praxis framework is in active development; and the during the development of the Cloud Analytics API, we created around 20 GitHub issues, and there are some cool Praxis features that we’re thinking about.

What you can do with our APIs

The Cloud Analytics APIs are based around monitoring and forecasting of costs across all of the public and private clouds that RightScale supports. For example, users can easily get the total_cost or lowest_instance_count of deployments tagged with env=production across their AWS and Microsoft Azure accounts. Users can also do various things on top of this data such as setting up Scheduled Reports and Budget Alerts, or creating unique links to share the data with others. The API Overview and the API reference docs provide more details.

We’d love to hear your feedback on our APIs and learn more about your use-cases.

© 2019 RightScale, Inc.