Automated Notification of Amazon API Gateway Errors

Amazon API Gateway is an AWS service for creating, publishing, maintaining, monitoring, and securing REST, HTTP, and WebSocket APIs at any scale. API Gateway handles authentication, access control, monitoring, throttling, request and response validation and tracing of API requests. This enables you to expose securely internal AWS or non-AWS back end services to external users or applications.

API Gateway uses HTTP status codes to convey its errors to you. 4XX errors are for client side errors which occurs when the client send an invalid request or has restricted access. 5XX errors are caused by the server being unable to fulfil a valid request from a client and something is wrong in server side.

It is necessary to monitor client side and server side error at API Gateway level to take prompt action. The purpose of this post is to help you create access logging with 4XX and 5XX error details and send error notification automatically to SNS topic using a lambda function if 4XX or 5XX error occurred as depicted in the architecture diagram below.

You can extend the architecture to send customised notifications to the preferred channel using a lambda function.

How to enable access logging in API Gateway?
First, you need to create IAM role to allow API Gateway to write access logs in CloudWatch. You can create role via AWS console or using the following CloudFormation.

Make note of the Role ARN. Next, using AWS console, select API gateway and then click on settings. You will get the following form to enter the role ARN.

Next, select your API project, click on stages, select the stage for which you want to enable access logging and click on Logs/Tracing tab, you will be provided with the following screen.

Then, check “Enable Access Logging” as above and enter a CloudWatch group ARN. You can create new CloudWatch group or use existing one. You also need to provide Log Format as below. It should be in CLF / JSON / XML / CSV format.

{
  "requestTime": "$context.requestTime",
  "requestId": "$context.requestId",
  "httpMethod": "$context.httpMethod",
  "path": "$context.path",
  "resourcePath": "$context.resourcePath",
  "status": "$context.status",
  "responseLatency": "$context.responseLatency",
  "xrayTraceId": "$context.xrayTraceId",
  "wafResponseCode": "$context.wafResponseCode",
  "integrationLatency": "$context.integration.latency",
  "integrationServiceStatus":                         "$context.integration.integrationStatus",
  "authorizerServiceStatus": "$context.authorizer.status",
  "authorizerLatency": "$context.authorizer.latency",
  "ip": "$context.identity.sourceIp",
  "userAgent": "$context.identity.userAgent",
  "caller":"$context.identity.caller",
  "user":"$context.identity.user",
  "responseLength":"$context.responseLength",
  "claims": "$context.authorizer.claims.sub",
  "accountId": "$context.accountId",
  "apiId": "$context.apiId",
  "stage": "$context.stage",
  "errorMessage": "$context.error.message",
  "intigrationError": "$context.integrationErrorMessage"
}

This allows to log who has accessed the API and how the caller accessed the API. Access details are populated from $context variables.

For the complete list of variable, please click here; https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html#context-variable-reference?cmpid=docs_apigateway_console

Now, we have enabled access logging and you can view the logs in CloudWatch logs by selecting the corresponding log group. Status variable in the access details will contain https status code.

To create automatic notification for 4XX and 5XX errors, you have to create subscription filters on the CloudWatch group with the following filter patterns and provide a lambda function ARN as Destination ARN.

Filter Patterns
{ $.status = 5* }
{ $.status = 4* }

You need a lambda function to read and process access logs and send notification. You can use following CloudFormation to create subscription filters and lambda function which can be configured to process your access log and send notification to a SNS topic.

You need to provide the SNS topic ARN and the log group name as parameters to the CloudFormation.

From SNS topic, you can send the notification to your preferred channel.

Please check my previous posts to send notification to the different channels from SNS.
Microsoft Team — https://rnanthan.medium.com/publishing-amazon-sns-message-to-microsoft-team-21fd0b48e60e
Email — https://rnanthan.medium.com/sending-email-using-amazon-ses-api-ece28062be52
Azure DevOps Board — https://rnanthan.medium.com/automating-the-creation-of-work-items-in-azure-devops-for-applications-errors-using-aws-lambda-and-a75b8e59420a

This post explains how to enable access logging in API gateway, populate 4XX and 5XX error in the logs and send automatic error notification to a SNS topic.