One of the many improvements in our recently re-deployed ELK stack is the use of CloudWatch to store performance metrics of our Logstash pipeline. Event rates for our shippers and indexers, buffer lengths, and host information are now all in a common platform without having to maintain a separate time series system. To track how the system is performing over time, we have a CloudWatch Dashboard that displays many of these key metrics. This is a terrific single pane of glass, but it does have some limitations. Graph types in CloudWatch are limited to line charts with a minimal amount of static text, and all graphs displayed on a common dashboard must use a single time window. This last one is particularly limiting as metrics that are measured on a daily basis (such as the number of objects in a S3 bucket) come across as either a single dot or, if midnight UTC is not in the displayed time window, fail to display at all. Getting this data out of CloudWatch and into something more versatile has long been on my wish list.

We have a major investment in Tableau as our internal corporate standard for visualizations. Being a not-small enterprise, software upgrades are complex efforts and we’ve only recently gotten onto Tableau 9.1 after a long run on the 8.x train. Tableau 9.1 brings a number of useful features, one of which is the Tableau Web Data Connector (WDC). WDCs are bits of client-side JavaScript wrapped up with Tableau SDK magic, enabling Tableau to talk to any arbitrary data source which you can access with a browser. As Amazon has a browser-side SDK for AWS, I took an afternoon during a long weekend to start working on connecting Tableau to CloudWatch.

After a bit of clean up, I’m happy to publicly release our Tableau WDC CloudWatch connector. Building this was a straightforward process even with my cursory level of familiarity with JavaScript (while I can code up some basic D3 plots, it’s not a language I work in regularly). I was able to take advantage of the AWS SDK Builder to get a minified version of the AWS SDK stripped down to the core CloudWatch functionality I needed. After a brief dalliance with jQuery, a dependency Bob Rudis wisely encouraged me to avoid, and a very minimal amount of CSS formatting, the final code is a reliable method for making CloudWatch data available to Tableau. The following very crude mock up is a collection of four separate CloudWatch data sources of five different statistics across different stages of our pipeline.

Sample CloudWatch Graphs

The connector works well, though there are a number limitations in the WDC, mostly surrounding the inability to edit parameters given to the AWS SDK apart from the initial connection. At the moment, once a time frame is passed to AWS for query data, that period is locked in and cannot be changed without establishing a whole new data connection.

So, give this new WDC a try and see if you don’t find your BI analysts and AWS admins having conversations along the lines of whether they have “Your Tableau in my CloudWatch!” or “Your CloudWatch in my Tableau!”