Catching up on the unexpected flurry of new Liquidmatrix Security Digest episodes, I was excited to hear a mention of tying in SSH login alerts to Slack. Having tweeted about a similar setup a couple weeks ago, I hadn’t gotten around to writing our solution up. Seeing another approach was a great motivation to burnish up my code a bit for general release. Hopefully seeing different patterns for solving what seems to be a common pattern will be helpful for others.
Problem Being Solved
Direct SSH logins to our instances is strongly discouraged, with a login a symptom that our config management processes have a problem. To keep visibility on this rare event, we want information on any successful interactive logins to notify my team via our private Slack group.
Log Alert Flow
- The majority of our AWS instances are Chef configured with a base cookbook that forwards key system and security works to CloudWatch Logs.
- From CloudWatch, a subscription filter on the security and auth logs (for the RedHat and Ubuntu platform families we use) sends events matching the filter “[month, day, time, host, daemon=sshd*, accepted=Accepted, …]” over to AWS Lambda. This filter looks for sshd accepted connections and provides a minimally parsed message structure.
- A small python-based Lambda function receives the events of interest, inspects them, and pulls some additional details on the EC2 instance involved such as the service line this instance supports. It then composes the relevant information and sends the alert message to a SNS topic.
- A HTTPS subscription on the SNS topic sends the message to a Slack webhook.
With our system logs already going into CloudWatch, having subscriptions to take actions on events of interest was a natural extension. The subscription filter language in CloudWatch is simple, but robust enough for our needs. Being able to send events of interest in a pre-parsed to our custom, serverless logic implemented in Lambda is a very powerful combination. Forwarding on structure events to SNS allows us to do additional message handling, including tie ins to services such as incident management via PagerDuty.
The current Lambda function’s alert format is a plain text message due to the SNS-Slack webhook not being able to handle rich JSON structures. I’ve got a half-finished effort to port the Scopely SNS->Slack bridge over to Lambda, after which I intend to re-write the login-alerter function to output more structured alert messages.
Lambda is fun service and Amazon has been effective in integrating it with more of their service offerings. With CloudWatch Logs, Config Rules, and hosted Elasticsearch, Amazon has a great story for folks creating all sorts of cloud infrastructure monitoring services. It’s almost enough to make me overlook the lack of VPC and tagging support on Lambda…almost!