AWS’s annual customer conference took place last week in Vegas. As the big shock and awe event for new feature releases and bringing the user community together, re:Invent has grown to be a huge affair with attendance over 35,000 this year. That’s over three times the attendance at BlackHat and closing in on 40,000 numbers of RSA. My own tendencies run towards the smaller venues, where I can spend more time interacting with both speakers and attendees while keeping my FOMO at manageable levels. Rather than venture to the desert, I followed re:Invent this year via Twitter, the AWS Blog posts, health doses of Slack channels, and follow up with the re:Invent YouTube playlist. In conversation with attendees, I feel I’ve gotten just as good, if not actually better, overview of the major new changes.
While a full recap of all the service changes would be very long, there are announcements around OpsWorks, Lightstream, Docker, Lambda, Step Functions, Athena, RDS, and Organizations that particularly caught my eye.
AWS OpsWorks for Chef Automate
I credit Chef for really getting me hooked on the power of infrastructure automation. Chef Co. has been on a tear with new products lately, including Habitat, the super interesting Inspec, and the Chef Automate commercial offering rolling together a full CI/CD pipeline with build, compliance, and automation. All of my work has been with the open source products (Chef Zero, FTW!) and I haven’t been able to try out the new Chef Automate goodness. AWS OpsWorks for Chef Automate is Amazon’s Chef-server-as-a-service, providing access to all the goodness of the Chef Automate product without having to set up the server itself or to make a big commitment to a long term license. I’m looking forward to getting the chance to play out with some of these features first hand.
The AWS Reddit forum regularly gets questions from newcomers to AWS that are trying to get basic web hosting or other similar functionality up and running, but who are overwhelmed with the details of setting up VPCs, security groups, Route53, and all the other plumbing required for a basic service. Enter Lightsream, which targets the various VPS providers by providing a (mostly) fixed price for a given instance size over a monthly term, with all the VPC and other details auto configured by AWS. I say mostly fixed price as there are still per-usage charges for services that go over certain quotas, such as Route53 services. This may be a threat to some of the less compelling VPS providers. For folks already on AWS, I don’t see a lot of benefit apart from a potential use for quickly providing trivially simple developer/demo environments.
Docker, docker, docker, docker…. Oy. There were a number of open source and scheduling items announced around the AWS ECS service. My use patterns are largely jumping from EC2 instances straight into Lambda-style environments, so Docker isn’t all that exciting to me. Only so many hours in the day…
The biggest announcement here came out pre-re:Invent, with the long awaited introduction of managed environment variables (with KMS integration to boot). C# support was also introduced, which should make some shops very happy and was probably partly necessary to compete against Azure. I am truly disappointed not to get Python3 or Go support in Lambda and the continued silence on getting tags in Lambda is appalling. Couple with some of the other announcements, the lack of tagging support for access control, billing, and operational management is a major weakness of the product.
Speaking of Lambda, Step Functions allow the user to create collections of functions (including EC2, on-premises, and yes, Lambda functions) with business logic to have auto-scaled and stateful managed executions. I’ve penciled out a potential security use case I want to explore. Initial explorations are a bit limited. The Step Function team seems to have ignored the positive changes made in CloudFormation and has chosen JSON rather than the slightly more sane YAML format for describing workflows. There are other rough edges that immediately made themselves apparent in a lack of access to the generated graph via the API (not even an option to download a simple dot file), no way to see the code of a state machine once it’s been activated, etc. It looks like a promising replacement for SWF (which is strongly implied in the Step Functions FAQ), though I see a number of places for potential frustration in this initial release.
Essentially managed Apache Hive as a service, Athena allows users to query S3 data on demand via a SQL like interface, with billing on an as demand basis and no infrastructure to manage. People seem more excited about this than I was initially, but running EMR jobs of one sort or another against S3 is such a common pattern for me that I didn’t see how this was making things easier for people. In going over some of the documentation, while Athena can operate on basic raw file types easily enough, converting files to something like Parquet files looks to be strongly encouraged for both performance and costs savings. It will be interesting to see how much that recommendation becomes a have to have for users and how they will address that process.
The Postgres engine comes to Aurora and all Postgres on RDS versions get eligibility as a HIPAA eligible service under the AWS business associate agreement. Super awesome. I have much love for Postgres. Between these new options and the ability to require SSL connections released earlier this year and that’s a whole passel of love for Postgres lately.
This was the big one. Many people (myself included) have long wanted the ability to create AWS accounts programmatically. Using separate AWS accounts for different projects/teams/functions is a recommended practice for limiting the blast radius for security or operational impacts. This has traditionally required many manual steps and bootstrapping, making a key security and management control very hard to implement. With this new release, the challenges around the spotty tagging and IAM support across the AWS feature space become much more manageable. Organizations is only in preview right now, but as soon as that hits GA out here in us-west-2, I’ll be all over that.
While I wouldn’t turn down the opportunity to attend re:invent, I don’t feel it’s essential for me to attend. There are plenty of resources to track. Indeed, one of the biggest challenges both AWS users and AWS themselves have these days is managing the pace of change. As AWS moves up the stack (e.g. the self-titled AI offerings, AppStream 2.0, etc.), staying deep on all areas is not viable. For example, I’m very ignorant of the Work* suite and the gaming functions apart from a general awareness of their existence and what roles they play. On the other end of the technical stack, the new sets of instance types, Snowmobile for exabyte data moves, IPv6 support for EC2, and various Shield-related products will be sure to excite many. I’m happy to dive deeply where I need to in order to help my organizations move forward with agile environments that truly revolutionize how we do business.