Avoid Rate Limiting in AWS Cloudformation

Avoid Rate Limiting in AWS Cloudformation

Sage Huynh

With experiences with developing cloud application, I sometimes get "Rate limiting", "Rate exceeded" errors when I use AWS CloudFormation.

So what actually is "Rate limiting" or "Rate exceeded"?

The Rate exceeded/Rate Limiting error occurs when API calls to an AWS service exceed the maximum allowed API requests, and the API calls are throttled. Generally, these errors are temporary and resolve themselves as the API calls lessen.

How to prevent Rate Limitng/Rate Exceeded?

There are some ways to walk arround with this issue, some of them are not out of the box support from aws-cdk.

In brief to prevent Rate exceeded errors, you can:

  • Implement exponential backoff
  • Create or update stacks one at a time
  • Use the DependsOn attribute
  • Request a quota increase

Implement exponential backoff

When using AWS API endpoints, implement exponential backoff to decrease the number of API calls being made.

The following example pseudo code for a DescribeStacks API call is configured to retry the API call after specific amounts of time:

{
Make ‘DescribeStacks’ API call
if throttled: wait 2 sec; Make ‘DescribeStacks’ API call
if throttled: wait 4 sec; Make ‘DescribeStacks’ API call
if throttled: wait 8 sec; Make ‘DescribeStacks’ API call
if throttled: wait 16 sec; Make ‘DescribeStacks’ API call
if throttled: wait 32 sec; Make ‘DescribeStacks’ API call
}

Create or update stacks one at a time

Creating or updating multiple CloudFormation stacks concurrently can result in many API calls being made at the same time. To prevent the API calls from exceeding the maximum allowed API requests, create or update one stack at a time.

Use the DependsOn attribute

Unless a dependency is defined between resources, CloudFormation creates and updates resources at the same time. The DependsOn attribute defines dependencies between resources to control concurrent updates.

The DependsOn attribute lets you specify when each dependent resource is created or updated. For example, if resource B is dependent on resource A, then you can specify that resource A be created or updated before resource B. This limits the number of API calls being made at at the same time and reduces the occurrence of throttling. You can also use the DependsOn attribute with nested stacks.

Request a quota increase

If the preceding resolutions don't work for your situation, then you can request a quota increase. Before you request a quota increase, identify the API call to determine the one that is exceeding the call rate.

In your request for a quota increase, include your AWS Region, the time frame of the API throttling, and the reason for the increase.

That it's. Thanks for Reading!