When deploying an Amazon Web Services (AWS) environment to production, a monitoring system that can detect failures, performance degradation, and resource shortages early is essential. While Amazon CloudWatch can collect metrics and logs, simply configuring the tool is not enough to actually address problems.
In monitoring design, it's essential to pre-determine what to monitor, under what conditions alerts will be issued, who will perform the initial check, and to what extent the response will be conducted. If this is unclear, there may be too many notifications, leading to important anomalies being overlooked, or even when an alert is issued, the responder may be unsure how to make a decision.
This article explains the fundamental items to decide when designing AWS monitoring, major monitoring services including CloudWatch, monitoring items for major AWS services, alert design, and operational structure considerations.
AWS monitoring design is the process of continuously monitoring the status of systems running on AWS and designing a system to detect anomalies and changes. This involves not only setting alarms in Amazon CloudWatch, but also designing the monitoring targets, notification conditions, initial response procedures, and escalation processes.
In a production environment, monitoring includes not only resource downtime, but also performance degradation, increased errors, configuration changes, and security anomalies. If the monitoring scope and notification conditions remain unclear, signs of failure may be missed, and initial response after an alert is triggered will be delayed.
What is necessary for stable operation is a mechanism to detect anomalies and a system that can take action after detection.
In monitoring design, the following are the main things to decide:
Increasing the number of monitoring items doesn't necessarily improve security. An increase in low-priority notifications can bury truly critical anomalies. Organize each item, defining what constitutes an anomaly, who to notify, and what priority to prioritize.
The actions taken after an alert is triggered are also part of the design. By defining the primary verifier, the information to be verified, and the conditions for handing over information to the relevant departments and personnel, the initial response to the incident can be expedited.
Monitoring design should be considered from the build phase, not added after the system goes live. The items to monitor and notification conditions will vary depending on the AWS services used, availability requirements, and the scope of impact in the event of a failure.
Adding monitoring after the system has been built could lead to situations where necessary logs are not collected, notification destinations are not defined, and responsibilities are unclear before operation begins. You must also check which metrics are available by default and which require additional configuration. For example, Amazon EC2 memory usage and OS internal disk usage may require the installation of the CloudWatch agent.
Integrating monitoring from the construction phase minimizes rework after going live and allows for the inclusion of a failure response system in the design. System configuration and monitoring design should not be considered separately, but together.
In monitoring design, the process is organized into detection, notification, decision-making, and response stages. Here, we will explain the basic items that should be checked before going into production.
First, we identify the items to monitor. These include Amazon EC2, Amazon RDS, AWS Lambda, Elastic Load Balancing, network, application logs, security events, and costs.
Next, we'll organize the items to check for each target. For EC2, potential items include CPU usage, memory usage, disk usage, and status checks. For RDS, we'll check CPU usage, number of connections, available storage space, IOPS, and replication latency.
You don't need to review every single item. Prioritize items that have a significant impact on business operations, items that should be identified as signs of failure, and items that will be used to make recovery decisions.
Once you've decided on the monitoring items, define what conditions will be considered abnormal. Specify the conditions for triggering an alert, such as high CPU usage for a certain period, low available RDS storage space, or an increase in Lambda error counts.
Lowering the threshold doesn't necessarily make things safer. If the conditions are too strict, notifications will increase, and those on the ground will start treating the alerts as noise in their normal operations. On the other hand, if the conditions are too lenient, signs of a problem won't be detected.
We handle instantaneous spikes and persistent anomalies separately. By deciding whether to notify us for a temporary increase in load or if it persists for a certain period of time, we can reduce unnecessary alerts.
Alerts are integrated into operations by defining the recipients and notification methods. For minor alerts, email or chat notifications suffice, but for anomalies that could lead to service disruptions or customer impact, direct contact with the relevant personnel or escalation to the operations team will be necessary.
If notifications are left to individuals, responses will halt when the person in charge is absent. Establishing team-based notification systems, duty rosters, and rules for substitute responses will prevent missed responses after an alert occurs.
This clearly defines what the person receiving the notification should check, under what conditions they should determine if it's a problem, and to whom they should hand it over. If the response flow remains unclear, even if an alert is noticed, initial verification and troubleshooting will take time.
In configurations involving multiple AWS services, a procedure is needed to sequentially check whether the cause of an issue lies with Amazon EC2, Amazon RDS, the network, or the application. Establishing procedures for reporting after recovery and preventing recurrence will allow you to use incident response to improve operations.
Monitoring targets are categorized into failures, operation history, configuration changes, security, and cost. Amazon CloudWatch is the central tool, and multiple AWS services are combined as needed.
Services | Main role | Easy to monitor and verify |
Amazon Cloud Watch | Monitoring metrics, logs, and alarms | CPU usage, error count, logs, alarm notifications, dashboard |
AWS Cloud Trail | Recording of operation history | Who performed which API operation, and when? |
AWSConfig | Recording and evaluating resource configuration | Configuration changes, configuration history, rule violations |
Amazon Guard Duty | Threat Detection | Suspicious access, misuse of authentication information, unusual communication |
AWS Health Dashboard | Impact assessment on AWS services | AWS outages, maintenance, and impact on resources in use |
AWS Budgets / AWS Cost Explorer | Cost monitoring and usage status confirmation | Budget overruns, increased usage fees, service-specific costs |
Amazon CloudWatch is a core service for handling metrics, logs, and alarms. However, the information you check is divided according to your purpose: AWS CloudTrail for operation history, AWS Config for configuration changes, Amazon GuardDuty for threat detection, and AWS Budgets or AWS Cost Explorer for cost monitoring.
Monitoring items are determined based on the characteristics of the service and the scope of impact in the event of a failure. This includes not only resource metrics such as CPU usage, but also error count, connection count, latency, and throttling.
AWS Services | Main monitoring items | Points to check |
Amazon EC2 | CPU usage, memory usage, disk usage, status check, network traffic. | OS and application operational status, resource shortages, and signs of instance failures. |
Amazon RDS | CPU usage, memory usage, available storage space, number of connections, IOPS, replication latency | Performance degradation, insufficient capacity, excessive connections, backup and redundancy status |
AWS Lambda | Number of executions, number of errors, execution time, timeout, throttling | Processing failure, increased execution time, limitations on the number of concurrent executions, impact on the caller. |
Elastic Load Balancing | Request count, HTTP 4xx/5xx, target response time, number of healthy/abnormal hosts. | Application-side errors, response delays, backend anomalies |
Amazon S3 | Request count, error count, latency, storage usage, access logs | Abnormal object access, increased errors, increased capacity, unexpected access |
Amazon VPC / Network | VPC Flow Logs, NAT Gateway traffic, VPN connection status, Direct Connect connection status | Abnormalities in communication paths, sudden increases in communication volume, disconnections of external connections, unexpected communications |
Some metrics cannot be obtained using standard metrics alone. For example, Amazon EC2 memory usage and OS internal disk usage may require the installation of the CloudWatch agent.
Monitoring items are not determined mechanically on a service-by-service basis, but rather calculated backward from the scope of impact in the event of a failure. Processes with a significant impact on customers, databases whose downtime would halt business operations, and networks involved in external integrations are given priority in the monitoring process.
In alert design, you define everything from the urgency level to the response procedures. If the notification conditions are ambiguous, anomalies requiring immediate attention and notifications for confirmation purposes will be mixed together, making it impossible to determine priority.
Sending all alerts to the same recipient and using the same method fails to convey the differences in urgency. Notifications should be differentiated according to importance: immediate action is required for anomalies that could lead to service disruptions or customer impact, while resource usage spikes or minor errors should be checked during business hours.
For high-priority alerts, use channels that allow staff to be immediately notified, such as chat, phone, or incident management tools. Items that can be addressed through trend checks or periodic reviews are handled via email or dashboards.
When there are too many alerts, field staff begin to treat the notifications as noise in their normal operations. In this state, anomalies that should be checked immediately get buried.
To prevent excessive notifications, adjust the threshold, duration, and notification frequency. A momentary surge in load and a sustained high load lasting several minutes or more require different responses. If multiple alerts are triggered by the same cause, consolidating notifications can be an effective design strategy.
We will define the information to check after an alert occurs, the conditions for determining a problem, and the criteria for handing over the information to the relevant department. If it's unclear which dashboards to look at or which logs to check, the initial response will become dependent on individual expertise.
Establishing response procedures helps reduce oversights and inconsistencies in judgment, ensuring a stable initial response when a problem occurs.
Failures in monitoring design often stem not from a lack of tools, but from going into operation with unclear criteria for post-detection decisions and scope of response.
Common mistakes | Problems that occur | (I.e. |
It all comes down to just configuring CloudView. | An alert is issued, but there are no established procedures or designated personnel for verification. | Design the monitoring items, notification destinations, initial verification procedures, and escalation channels. |
Adding too many monitoring items | The number of low-priority notifications increases, burying high-priority anomalies. | Prioritize items related to business impact, customer impact, and recovery decisions. |
The targets of surveillance are biased. | By only looking at resource status, logs, configuration changes, and security events are overlooked. | Organize resource monitoring, log monitoring, operation history, configuration changes, and threat detection separately. |
Keep the threshold fixed. | The inability to keep up with changes in usage leads to an increase in false positives and undetected results. | Adjust based on the number of alerts, response history, and resource trends. |
The recipient of the notification depends on the individual. | The verification process stops when the person in charge is absent. | Establish rules for team-based notification recipients, duty responsibilities, and proxy response procedures. |
The scope of responsibility for troubleshooting is unclear. | Decision-making is stalled between our company, the development company, and the outsourced operations provider. | Divide the responsibilities into initial response, cause investigation, recovery work, and reporting. |
Cost monitoring is not implemented. | Unexpected charges may arise due to an unawareness of increased log storage volume or data usage. | Use AWS Budgets or AWS Cost Explorer to check your budget, usage, and growth trends. |
By designing a process that encompasses detection, notification, confirmation, response, and review, alerts can be used to improve operations.
Monitoring design should be based on the importance of the system and the scope of impact in the event of a failure, rather than on tool configuration.
1. Prioritize the importance of each system.
The required level of monitoring differs for customer-facing services, internal business systems, and testing environments. We will confirm the scope of impact during downtime, acceptable downtime, and whether nighttime and weekend support is necessary.
2. Identify the targets to be monitored.
In addition to resources such as Amazon EC2, Amazon RDS, AWS Lambda, and Elastic Load Balancing, this also includes application logs, network events, security events, and costs.
3. Determine monitoring items and thresholds.
We prioritize items that indicate potential problems or could impact business operations, based on CPU usage, available storage space, error count, number of connections, latency, and other metrics. Thresholds are set based on normal usage patterns and peak hours.
4. Determine the notification and response flow.
We'll categorize issues into those requiring immediate attention, those that can be checked during business hours, and those that will be handled through regular trend checks. We'll also decide on the primary checker, the logs and dashboards to be reviewed, and the escalation destination.
5. Review after implementation
We will review the number of alerts, false positives, non-detections, and response times. We will also review unused items and items for which there was insufficient information to make decisions during failures. If the system configuration or usage changes, we will update the monitoring design.
Whether you can handle it in-house isn't determined solely by AWS knowledge. We also need to check the number of personnel available, response time, decision-making authority during outages, and the system in place to handle recovery.
Judgment items | Cases where in-house management is suitable | Cases where outsourcing should be considered |
A person familiar with AWS | We have someone in-house with AWS operation experience. | There are few people in charge, or the responsibility is dependent on specific individuals. |
Response time | Weekday daytime support is sufficient. | Monitoring and initial response are required, including during nighttime and holidays. |
System importance | The impact of the shutdown is limited. | Customer-facing services and core systems are among the systems that would be most significantly impacted if they were to be shut down. |
Scope of incident response | Alerts can be checked and simple troubleshooting can be performed internally. | Support is needed for everything from investigating the cause and assisting with recovery to reporting to those involved. |
Scale of operation | Can be managed with a single account and a small number of resources. | Spanning multiple accounts, multiple systems, and multiple locations |
Security monitoring | Basic log checking and permission management are sufficient. | I want to continuously monitor AWS CloudTrail, AWS Config, Amazon GuardDuty, etc. |
Operational improvement | The number of alerts and response history can be reviewed internally. | I want to regularly improve the monitoring items, thresholds, and notification conditions. |
If you require 24/365 monitoring, operation of multiple accounts, security monitoring, and first-line response to failures, maintaining such a system in-house becomes a significant burden. By clearly defining the scope of responsibility to be handled internally and the scope to be outsourced during the monitoring design phase, the scope of responsibility after implementation will also become clearer.
AWS monitoring design involves not only configuring Amazon CloudWatch, but also defining monitoring targets, alert conditions, notification destinations, and response flows. Ambiguous monitoring scopes and thresholds can lead to missing signs of failures and delayed initial response after an alert is triggered.
In a production environment, we monitor resources, logs, operation history, configuration changes, security events, and cost status separately. We design the monitoring scope according to our objectives, using CloudWatch as the central tool, combined with AWS CloudTrail, AWS Config, Amazon GuardDuty, AWS Budgets, and other tools.
For stable operation, clear criteria for decision-making after detection and response procedures are essential. If it is difficult to handle this in-house, you should consider the necessary operational structure, including outsourcing.
