Learn more about AWS throttling
If the number of API requests per second exceeds set AWS thresholds, AWS applies throttling to those instances and volumes. When throttled, the instances and volumes have all polling suspended and enter an Unknown state - AWS Throttling Applied until the workload lowers. Polling continues, updating the node state and clearing the alert when AWS removes throttling.
The AWS API Gateway constantly checks the incoming requests against your AWS account limit. If your AWS polling requests exceed default or custom set thresholds, AWS throttling applies to your monitoring account, instances, and volumes blocking any requests beyond the limit. The API accepts requests up to the acceptable requests per second: limits the steady-state request rates to 1000 requests per second (rps) and allows bursts of up to 2000 rps across all APIs, stages, and methods within an AWS account. AWS API Gateway responds to any requests beyond the limit with a 429 HTTP response for too many requests.
If necessary, you can request an increase to your AWS account-level limits for polling requests/second and usage plans. For details, review and contact your AWS account service.
Orion cloud infrastructure monitoring includes out-of-the-box alerts to respond to throttling issues.
To help reduce multiple alerts overwhelming the alert list, the alert Alert me when AWS throttling is applied for cloud account aggregates throttling applied issues for any instances or volumes into a single alert. The alert checks every minute for any throttled instances or volumes through CloudWatch data. A single event fires indicating the cloud account has throttling applied. Access the alert to verify the affected instances and volumes. An email notification indicates the number of affected instances and volumes.
For detailed information, you can review those specific instances and volumes in an Unknown - AWS Throttling Applied state.
To always receive alerts per throttled instance and volume, disable the account level alert and enable the following alerts. If multiple instances and volumes exceed thresholds and incur throttling, you could have an extensive list of alerts trigger for each monitored system.
- Alert me when AWS throttling is applied for cloud instance: Disabled by default, this alert checks every minute if throttling is applied to instances. Conditions check for the instance status of Unknown and AWS Throttling applied through EC2 API calls.
- Alert me when AWS throttling is applied for cloud volume: Disabled by default, this alert checks every minute if throttling is applied to instances. Conditions check for the attached volume status of Unknown and AWS Throttling applied through EC2 API calls.
If AWS throttling is applied, a warning displays in the Cloud Summary and Cloud Instance Details pages as appropriate and the Alert Manager. To resolve, work with AWS to increase your usage plan and consider alternatives.
To create additional alerts for throttling, create and edit a duplicate of one of these alerts. For details, see Alerts and events for cloud monitoring.