CloudWatch Events

Cloudwatch events were announced in January 2016. Within the console the service appears limited, however, under the hood its has some features not immediately obvious.

In this post we explore multiple account support and custom events, together these can be used to create neat solutions.

Recap

An EC2 event, for example an instance failure or impaired disk event uses Cloudwatch events to notify a lambda. The lambda could notify a paging system or attempt to rectify the issue.

Terminology

• AWS Services – generate Events
• Events – JSON format messages generally indicating changes in AWS environments.
• Event Buses – accept events from AWS services, accounts and API calls then distribute events. Currently, each account only permits a single event bus called default.
• Rules – Rules match events and route to targets for processing
• Targets – Process events received in JSON format

Many AWS services [create events] (https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/EventTypes.html), and many services can be used [as targets] (https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/WhatIsCloudWatchEvents.html). Recent updates added ECS tasks, AWS inspector templates, batch and step as targets, enabling many useful combinations.

Multiple Accounts

Support was added in June 2017, it enables event buses to be shared with other accounts, and event rules to forward to buses in different accounts. This is generally used to ‘fan in’ and ‘fan out’ events.

A ‘fan in’ design could be used to consolidate events to a central account for security, monitoring or auditing purposes.

To forward internet connectivity related health events the rule in the source accounts would have a rule such as:

$ aws events describe-rule --name internet-health
{
    "Name": "internet-health",
    "EventPattern": "{\"source\":[\"aws.health\"],\"detail-type\":[\"AWS Health Event\"],\"detail\":{\"service\":[\"INTERNETCONNECTIVITY\"]}}",
    "State": "ENABLED",
    "Arn": "arn:aws:events:ap-southeast-2:111111111111:rule/internet-health",
    "Description": "An internet connectivity health issue"
}

With a target of the destination accounts event bus:

$ aws events list-targets-by-rule --rule internet-health
{
    "Targets": [
        {
            "RoleArn": "arn:aws:iam::111111111111:role/service-role/AWS_Events_Invoke_Event_Bus_123321123321",
            "Id": "Id123321123321",
            "Arn": "arn:aws:events:ap-southeast-2:222222222222:event-bus/default"
        }
    ]
}

Also note Cloudwatch events has a built-in loop detection, in that events forwarded between accounts cannot be forwarded on again.

A ‘fan out’ design could be used from a central management or security account, for example to invoke a lambda against multiple target accounts.

This uses a single event forwarding to multiple targets:

$ aws events list-targets-by-rule --rule multitarget-demo
{
  "Targets": [
    {
      "RoleArn": "arn:aws:iam::111111111111:role/service-role/AWS_Events_Invoke_Event_Bus_123",
      "Id": "Id123321123321",
      "Arn": "arn:aws:events:ap-southeast-2:222222222222:event-bus/default"
    },
    {
      "RoleArn": "arn:aws:iam::111111111111:role/service-role/AWS_Events_Invoke_Event_Bus_123",
      "Id": "Id123321123321",
      "Arn": "arn:aws:events:ap-southeast-2:333333333333:event-bus/default"
    },
    {
      "RoleArn": "arn:aws:iam::111111111111:role/service-role/AWS_Events_Invoke_Event_Bus_123",
      "Id": "Id123321123321",
      "Arn": "arn:aws:events:ap-southeast-2:444444444444:event-bus/default"
    }
  ]
}

Custom Events

Custom events allow your own services to create events in fixed formats, use event filters and send to all the supported targets.

Example 1: A custom application health status

Custom Event Message

{
  "source": "com.mantalus.demoapp",
  "detail-type": "healthcheck status",
  "detail": { 
    "description": "external endpoint failure",
    "environment": "production",
    "application": "website",
    "url": "http://api.demo.com",
    "status": "503"
  }
}

Event Rule for SNS – Only send production to the on-call team

{
  "source": [
    "com.mantalus.demoapp"
  ],
  "detail-type": [
    "healthcheck status"
  ],
  "detail": {
    "environment": [
      "production"
    ]
  }
} 

Event Rule for Lambda – Send failures from all environments to slack (via lambda)

{
  "source": [
    "com.mantalus.demoapp"
  ],
  "detail-type": [
    "healthcheck status"
  ]
}

Example 2: Deployment status event

Different product teams sharing common security tracking and change management systems.

Custom Event Message

{
  "source": "com.mantalus.deploydemo",
  "detail-type": "deployment status",
  "detail": {
    "environment": "production",
    "application": "website",
    "version": "1.21",
    "status": "success"
  }
}

Security Rule: Send all deployment events to a security account where a lambda could save the event for audit logging.

{
  "source": [
    "com.mantalus.deploydemo"
  ],
  "detail-type": [
    "deployment status"
  ]
}

Change Status Rule: Send all successful changes to a shared services account hosting a change management tool to trigger a lambda to auto close a successful change.

{
  "source": [
    "com.mantalus.deploydemo"
  ],
  "detail-type": [
    "deployment status"
  ],
  "detail": {
    "status": [
      "success"
    ]
  }
}

To notify users on a failed deployment

{
  "source": [
    "com.mantalus.deploydemo"
  ],
  "detail-type": [
    "deployment status"
  ],
  "detail": {
    "status": [
      "failure"
    ]
  }
}

Custom Events

As with most services, there are a few things to be aware of when designing:

• Currently a limit of 5 targets per rule, larger designs will need to consider this limitation.
• SSM run command does not support multiple targets.
• PutEvents supports up to 10 entries per request up to 256k, this is a soft limit.
• If a target is unavailable, for example a rate limit or issue with a lambda, cloud watch events will retry for to 24 hours.
• Region based service, however lambdas may be used to pass custom events to multiple regions.

Costs

In addition to the cost of the resources generating and processing the events, there are some minor costs to be aware of for Cloudwatch events. In Sydney these are currently $1 per million custom events and $1 per million cross-account events.

Summary

Cloudwatch events are growing into a highly effective service, capable of assisting with many operational, security and monitoring related tasks.