Cloud Analytics API secrets - build your own Dashboard

Undoubtedly, the RightScale platform provides powerful tools for managing cloud costs and analytics. We expose those insights to users through the Cloud Analytics app, while using those tools to power other features throughout the platform. Users can programmatically access the insights that the RightScale platform provides through our APIs. Cloud Analytics uses extremely few private APIs, so the web application becomes an excellent driver of our API design.

In this series, I will explain how the Cloud Analytics app leverages these APIs and how you can recreate those results yourself. To start, we will take a deep-dive into the Cloud Analytics dashboard.

Analytics API Overview

At its core, Cloud Analytics provides two main APIs:

Cloud Analytics API

This API focuses on providing insights into the user’s cloud portfolios, as well as exposing analytics–related RightScale resources such as scheduled reports and billing alerts.

Pricing API

The Pricing API gives the user the ability to search for instance prices across any of our supported public or private clouds that we support as well as provide markups for our customer’s internal billing purposes.

The Cloud Analytics application leverages and combines both of these APIs in order to give the user the information they need. The API references the both of these can be found in our documentation.

The CombinedCosts API

At the heart of the dashboard is the CombinedCosts API. This API combines data from cloud bills, RightScale instance usage data, and custom costs the user has specified. This data can be grouped in a very flexible way, including hierarchically, depending on the user’s preference.

Specifically, the API combines costs from both cloud bills and custom costs, utilising RightScale instance usage data where cloud bills are not available.

This data can be exported as:

Grouped time-series data
CSV data

The dashboard primarily makes use of the former, with different grouping options.

CombinedCosts grouping

What makes the CombinedCosts API powerful (apart from its wide data set), is how costs can be grouped depending on the user’s wishes. This grouping can create unique values based on combining attributes, as well as providing this functionality in a hierarchical manner.

Let’s take the following payload:

  {
    "start_time":"2015-12-01T00:00:00Z",
    "end_time":"2016-12-01T00:00:00Z",
    "group":[
      ["product_category"],
      ["cloud_vendor_name"]
    ]
  }

Naturally, we’re setting the start and end dates we want data for (the API returns data at a monthly granularity), then we are defining our grouping. In this case, the grouping is hierarchical, grouping data by product category (Compute, Network, Database, etc.), then having a subgrouping by cloud vendor (Amazon Web Services, Google, Microsoft Azure, etc.).

[
  {
    "kind": "ca#time_series_metrics_result",
    "results": [
      {
        "kind": "ca#metrics_result",
        "group": {
          "product_category": "Administration & Security"
        },
        "breakdown_metrics_results": [
          {
            "kind": "ca#metrics_result",
            "group": {
              "cloud_vendor_name": "Amazon Web Services"
            },
            "breakdown_metrics_results": [

            ],
            "metrics": {
              "kind": "ca#metrics",
              "total_cost": 9.03
            }
          }
        ],
        "metrics": {
          "kind": "ca#metrics",
          "total_cost": 9.03
        }
      }
    ]
  }
]

Fig 1. Example CombinedCosts response

This will return a list of ca#time_series_metrics_results objects (Fig. 1) with one for each month. Within each of these, a list of ca#metrics_result objects are returned. Each of these ca#metrics_results objects contains 3 important attributes:

group - an object representing the values for each of the attributes listed for grouping at this level. In this case we set product category as the top-level grouping, and we have one result, for Administration & Security. Should we have listed multiple attributes to group at this level their values would also be listed here.
breakdown_metrics_results - a list of ca#metrics_result objects providing the results of the grouping at the next level down. Since we requested that the subgrouping be cloud_vendor_name, then this will be the list of results that match the ‘Adminstration & Security’ product category, broken down by cloud vendor (in this case, just Amazon Web Services). When the result is at the lowest level of grouping, this will be an empty array.
metrics - an object containing the cost of the current result. When this ca_metrics_result object contains groupings below it, this will be the sum of the groupings below it, otherwise it will just be the cost of that line item.

In this case, we grouped just one attribute at each level, but if, for instance, at the second level, I were to combine cloud_vendor_name and account_id, it would provide a unique result at that level for each combination of cloud vendor and account (i.e. The results for AWS/Account A would be separate from AWS/Account B)

The Dashboard

The Cloud Analytics Dashboard can be broken down into 4 main parts:

Monthly breakdown chart
Product category breakdown (for selected month)
Detailed cloud vendor breakdown (for selected month)
RightScale account breakdown (for selected month)

All 4 of these parts utilise the CombinedCosts API, and due to its flexibility, just 2 API calls provide all the data this page requires, reducing load times, bandwidth and load on our servers.

Monthly breakdown chart

This chart gives a relative representation of the spend of each month, broken down by product category. If the user hovers over a month, it displays the exact spend for each category for the month, as well as the total.

If that sounds like we’ve just asked for monthly data grouped by product category… well that’s because it is! To achieve this result, a POST request is sent to to

https://analytics.rightscale.com/api/combined_costs/actions/grouped_time_series

with payload:

{
  "start_time":"2015-12-01T00:00:00Z",
  "end_time":"2016-12-01T00:00:00Z",
  "group":[
    ["product_category"]
  ]
}

And that’s it!

(Well, that’s how you could do it, we actually make a slightly different call - read on for more details)

Product category breakdown

This is a pretty simple one. This is just displaying the same data that we have already loaded in the previous API call, just for the selected month. The percentages are simply computed client-side, so we don’t need to make another API call.

Detailed cloud vendor breakdown (for selected month)

Ok, this is where it gets a little more complicated. There’s a lot of data here, showing each line item from all cloud vendors for that particular month. This data is filtered by cloud vendor, followed by cloud vendor account (Note: not RightScale account since we may have data for accounts that aren’t linked to RightScale).

In this case, the API call seems simple - filter by cloud vendor, followed by cloud vendor account to get the product and product category (remember, we colour-code the line items by category) like this:

POST https://analytics.rightscale.com/api/combined_costs/actions/grouped_time_series

{
  "start_time":"2016-11-01T00:00:00Z",
  "end_time":"2016-12-01T00:00:00Z",
  "group":[
    ["cloud_vendor_name"],
    ["cloud_vendor_account"],
    ["product", "product_category"]
  ]
}

Simple! Well…. not quite.

Our sleight-of-hand trick

Again, you absolutely could do it this way, but we do it slightly differently. Remember back in the monthly breakdown chart, I mentioned there was more to the API call made than just filtering by product category? Well here’s where this comes back into play.

UIs are made to be used by humans, and humans are impatient. So we employ a little smoke-and-mirrors to give the user at least some useful information before we have the full breakdown. In the monthly breakdown chart we actually filter by product category, then by cloud vendor name.

POST https://analytics.rightscale.com/api/combined_costs/actions/grouped_time_series

{
  "start_time":"2015-12-01T00:00:00Z",
  "end_time":"2016-12-01T00:00:00Z",
  "group":[
    ["product_category"],
    ["cloud_vendor_name"]
  ]
}

(It’s the same payload as I used in the original CombinedCosts API example)

Why would we do that? Well this call is always made, regardless of which month you want to look at (since the graph will always be there), so we can make a fair assumption that this API call will be made. So once we have that data, we reprocess it client-side to give us just the total cost for each cloud vendor, for each month. Therefore when a user clicks on a particular month (or the latest is auto-selected on page load), we can populate the cloud vendor total cost for that month immediately, even while we’re waiting for the in-flight complex breakdown.

In our experience, the user typically doesn’t have more than 1 or 2 expanded cloud vendors at any one time, so by the time a user tries to expand a cloud vendor for detailed costs, the longer-running detailed breakdown request has been fulfilled and we’ve already silently populated the tables. It’s just a sneaky UX trick to give the illusion that we had the data all along.

If you want to see this in action, expand a number of the cloud vendors and reload the page. You’ll see that the graph and the cloud vendor totals are populated in advance of the detailed breakdowns.

The actual detailed breakdown

So how do we do the detailed breakdown? Here’s our request payload:

POST https://analytics.rightscale.com/api/combined_costs/actions/grouped_time_series

{
  "start_time":"2016-11-01T00:00:00Z",
  "end_time":"2016-12-01T00:00:00Z",
  "group":[
    [
      "cloud_vendor_account_id",
      "cloud_vendor_account_name",
      "account_id",
      "account_name",
      "cloud_vendor_name"
    ],
    [
      "product",
      "product_category"
    ]
  ]
}

Because we already have the cloud vendor totals, we can ask for fairly fine-grained data to display. Where we need to, we can easily collect and regroup this data client-side for whatever purposes we require, so we don’t have to make lots of API calls.

For the detailed breakdowns, we group client-side by cloud vendor, then by cloud vendor account ID. Once we have these, we’ve already broken this down by product and product category, so we don’t need to do it again.

(Why didn’t we just ask the API to group this way? Keep reading!)

This same call will also give us various other information that our UI uitilises, specifically the cloud vendor account name, RightScale account ID and RightScale account name, all of which provides extra visibility to the user.

RightScale account breakdown

This is the other application of our sleight-of-hand trick. There’s nothing here that our detailed breakdown API call didn’t already give us, so once again, we reuse our API calls to save on loading time. In this case, we take the data from the detailed breakdown and regroup client-side by RightScale account (specifically by RightScale account ID & RightScale account name).

If we have data for cloud accounts that aren’t linked to RightScale accounts, we show the total cost on those accounts, along with a small reminder that we can give more data if these accounts were linked to a RightScale account.

Since this requires our detailed breakdown API call, we employ the same UX trick as before, making the product category tab the default (since that is populated from the faster, initial API call), and require the user to click on the tab in order to see this data. By the time the user clicks on the tab, the longer-running detailed breakdown call should have been fulfilled and we’ll have already silently updated the table.

What’s next?

Whilst the Dashboard UI looks complicated, it allows RightScale’s powerful API platform to do most of the heavy lifting, utilising its wide data set, and provides an uncomplicated view to the user. In this article, I hope I’ve managed to demonstrate that the public Cloud Analytics API provides all the necessary functionaility to obtain all the same data we use on the dashboard programatically, for our customers’ own internal tooling as well as for whatever purposes they desire!

If you wish to find out more about out our APIs, head over to our documentation and have a look!