The Elasticsearch ecosystem moves very fast, with a large number of releases coming out in the closing months of 2015. While we have been keeping our install as current as possible, we couldn’t easily move to the newer 2.x branches of both Elasticsearch and Logstash due to AWS OpsWorks being stuck on Chef 11. This was finally resolved at the start of December with the release of Chef 12 for Linux. With a bit of time over the holiday break, it was time to take this card off the backlog and bring us current with this large set of changes.
AWS OpsWorks and Chef 12
With the long awaited release of Chef 12 on OpsWorks, I was excited to start converting things over. With VulnPryer safely running on Chef 12, I was ready to tackle the more complex ELK environment. As a multi-node setup (our ELK deployment scales up to 10 nodes at maximum load), service discovery was a key requirement. Under the Chef 11 versions of OpsWorks, AWS provided an opsworks-branded branch of node attributes that were easy to leverage to allow nodes to find other systems running in the stack. Under OpsWorks implementation of Chef 12, the standard Chef Search mechanism is enabled. Unfortunately, the data bags provided by AWS are not the same as regular node searches under white-box Chef and I was forced to make my recipes a bit more AWS specific than I wished.
ELK Ecosystem Updates
We’ve been running Kibana 4 on a limited basis for some time now, but Logstash and Elasticsearch were still on the 1.x trains. The latest versions of Kibana require the 2.x Elasticsearch train so this turned into a big bang upgrade with major updates to Elasticsearch, Logstash, and even our Kibana deployment process.
While I have no love for creating CloudFormation templates from scratch, I’m not quite ready to go all in on a tool such as Terraform. With a bit of reluctance, I created a new CloudFormation template to stand up this ELK stack. In the process, I found another annoying limitation in CloudFormation’s support for OpsWorks that I had to work around. CloudFormation turns out not to support custom JSON at the layer level, which forced me to write a bit of custom code specifically into my cookbook rather than the clean overrides I wanted to perform. Despite some dark mutterings (my new mantra when looking at new AWS services - if it doesn’t support VPC, tagging, and CloudFormation out of the gate then it’s not ready for use), I managed to get this up and running. Having a template for this stack allowing me to repeatedly build a replica of production for testing. Now tested, I can use this exact template to deploy the new stack in production.
With the configuration management and discovery pieces well, the next step will be to move the orchestration and scheduling pieces over to CloudFormation. This will hopefully be the last bit of bare CloudFormation stacks I have to write by hand.