How to design AWS monitoring in Thailand? Required monitoring items and incident response system for local subsidiaries.

The launch of the AWS Asia Pacific (Thailand) region expands the options for using Amazon Web Services (AWS) within Thailand. However, simply choosing a region does not guarantee stable operation. To continuously operate an AWS environment, it is necessary to understand system status, logs, configuration changes, security, and costs, and to design a response system for when anomalies occur.

The Thai subsidiary faces challenges such as not being able to assign a dedicated AWS specialist, insufficient support for incident response during nights and holidays, and difficulty in monitoring operational status from the Japanese headquarters. Even with monitoring tools in place, incident response depends on individual personnel if the verifiers, decision-makers, and scope of recovery actions after an alert are not clearly defined.

This article explains the monitoring items that companies using AWS in Thailand should check, how to utilize CloudWatch, the response flow in the event of a failure, and the division of roles between the Japanese headquarters and the local subsidiary. It also introduces key points to consider when outsourcing AWS monitoring and operations to an external partner.

Why AWS monitoring is often a challenge for Thai subsidiaries

When using AWS at a local subsidiary, it is essential to design operations that anticipate monitoring and troubleshooting after environment setup. If AWS environment management is heavily reliant on local personnel, it becomes difficult for the Japanese headquarters to grasp the operational status, troubleshooting, costs, and configuration changes.

Even if monitoring alerts are received, if the verifier, the scope of initial response, and the conditions for escalating to headquarters remain unclear, incident response will depend on the individual in charge. If the notification recipients, response times, and reporting methods are not defined in advance, the initial response to an incident will be delayed.

With the launch of the AWS Thailand region, more companies are reconsidering their use of the service in the region.

The launch of the Thailand region expands the options for using AWS within Thailand. Even companies that previously used regions such as Singapore now have room to reconsider their chosen region based on factors such as latency, data storage location, and network configuration.

However, simply changing regions does not automatically resolve operational challenges. As the number of AWS environments managed locally increases, it becomes necessary to design systems for fault detection, resource monitoring, log review, security monitoring, and cost management. Along with region selection, monitoring and operational systems should also be reviewed.

Monitoring and troubleshooting tend to become dependent on individual IT personnel.

In some cases, local subsidiaries may not have a dedicated person for AWS. When IT staff are also responsible for internal network management, PC administration, business system support, and vendor coordination, they may not have the time to handle AWS monitoring and incident response.

Determining the meaning of an alert, isolating the scope of impact, and proceeding with recovery requires knowledge of AWS operations and the authority to take action. If only notifications are received and decision-making is left to individual personnel, responses will be delayed during nights, holidays, or when personnel are unavailable.

For AWS monitoring, don't leave it solely to local personnel; establish a monitoring, reporting, and escalation process in coordination with headquarters. Clearly defining who receives alerts, who performs initial troubleshooting, and at what stage information is shared with headquarters or external partners will reduce reliance on individual expertise for incident response.

The time difference and holiday schedules between our Japanese headquarters and other locations can easily lead to ambiguity in our response system.

There is a two-hour time difference between Thailand and Japan. Although it may seem like a small difference, it has an impact on monitoring and incident response. There are cases where the business hours in Thailand are still open even after the Japanese side has finished work. Conversely, there are also situations where the local staff member has not yet started work when the Japanese side begins their morning checks.

Differences in holiday calendars cannot be overlooked. What is a normal business day in Japan may be a public holiday in Thailand. If an alert occurs on a local holiday, the response will stop unless a designated person is in place to check it.

Taking into account time zones and holiday differences, we will organize notification recipients, response times, primary verifiers, and escalation destinations. Especially in a production environment, it is necessary to decide whether to define response times based on Japan Standard Time or Thai Time, and who will receive alerts outside of business hours.

Monitoring items to check when using AWS in Thailand

AWS monitoring isn't just about looking at the operational status of servers and networks. You need to separately check for signs of failure, logs, configuration changes, security risks, and cost fluctuations.

Simply increasing the number of alerts without organizing what to monitor will result in too many notifications, causing you to miss truly important anomalies. First, prioritize items with the greatest impact on business operations and clearly define who will be responsible for checking them—the local team or headquarters.

System Monitoring

The system monitoring targets key resources such as EC2 instances, load balancers, and RDS. It monitors CPU usage, network traffic, disk I/O, and status checks to detect signs of performance degradation or failures.

EC2 CPU usage and network traffic can be obtained using standard Amazon CloudWatch metrics. However, memory usage and OS disk usage may not be obtainable using standard metrics alone. If necessary, the CloudWatch Agent should be implemented to capture data on the internal state of the OS.

Threshold values ​​vary from system to system. For example, even if you set a threshold to issue a warning when CPU usage exceeds 80% for more than 5 minutes, whether that value is appropriate depends on the load characteristics and business impact. The system is designed with the understanding that thresholds should not be fixed in the initial settings but reviewed after operation begins.

Log monitoring

Log monitoring collects application logs, OS logs, error logs, etc., and uses them to investigate the cause of failures and detect anomalies. CloudWatch Logs can be used to aggregate logs from multiple AWS resources and servers.

It's not enough to just look at logs after a failure occurs. Continuously monitoring increases in errors, failures of specific processes, authentication errors, response delays, etc., can help pick up warning signs before a failure develops.

However, logs may contain personal or business information. When handling logs containing personal information, we will take into account Thailand's PDPA (Personal Data Protection Act) and confirm what data is collected, how long it will be kept, where it will be stored, and how it can be accessed. Increasing the amount of logs will increase the amount of information available for investigation, but storing unnecessary logs for extended periods will increase costs and management burden.

Operation audit and monitoring of configuration changes

Operational audits allow you to track "who performed which operation and when." AWS CloudTrail is a service that records API operations and the history of operations on the management console.

In AWS environments at overseas locations, multiple stakeholders, including local staff, headquarters staff, and external partners, may be involved in operations. Without keeping a record of operations, troubleshooting can be time-consuming when failures or configuration changes occur.

AWS Config and similar tools are used to monitor configuration changes. By tracking changes to security groups, IAM, storage settings, etc., it is possible to detect unintended configuration changes or configurations that deviate from headquarters standards.

If you separate AWS accounts between headquarters and local branches, you should also check the management units using AWS Organizations and AWS Control Tower. If you don't decide whether to distribute CloudTrail and AWS Config logs to each account or collect them in a centralized log account, it will be difficult for headquarters to audit and verify them.

Security monitoring

Security monitoring targets include suspicious access, misuse of credentials, excessive permission granting, and public access settings. Using services like Amazon GuardDuty can help detect early signs of threats.

The monitoring targets are not limited to server operational status. It also includes checking the usage of IAM users and roles, security group changes, S3 bucket visibility settings, and the granting of administrator privileges.

When there is a discrepancy between headquarters' security standards and local operations, risks can accumulate unnoticed. Without defining what to detect, who to verify, and what the corrective actions should be, responses will be delayed even after problems are discovered.

Checking for outage information on AWS services

Even if there are no issues with your own resources, outages or maintenance on the AWS service side may have an impact. The AWS Health Dashboard allows you to check events that affect AWS services and your account.

CloudWatch is a tool for checking metrics and logs within your own environment. On the other hand, you'll also use AWS Health Dashboard to monitor events on the AWS infrastructure side. When an alert occurs, you should have procedures in place to determine whether the issue stems from your own system or from an AWS service.

Especially when using a new region, be sure to check the availability and limitations of the services you plan to use. Configuring your system under the same assumptions as an existing region may result in the services and features you want to use not working as expected.

Cost monitoring

Cost monitoring helps detect increases in AWS usage, unnecessary resources, and unexpected usage at an early stage. AWS Budgets and Cost Explorer help you understand budget overruns and usage trends.

AWS usage at overseas branches can be difficult for headquarters to monitor in detail. Leaving unused EC2 instances, excessive storage, failure to shut down testing environments, and increased data transfer volumes unattended can impact monthly costs.

Monitoring itself incurs costs. As your usage of CloudWatch custom metrics, log storage, alarm count, and log analysis increases, so do your monitoring costs. Before adding more monitoring items, clarify the metrics you want to collect, the logs you want to store, the retention period, and the notification conditions.

Checking costs only after seeing the bill at the end of the month is too late. We will design a system that allows for early detection of signs of increases by defining budget thresholds, notification recipients, and check frequency.

How to use CloudWatch for AWS monitoring

CloudWatch is the central service for AWS monitoring. It provides visibility into the state of your AWS environment using metrics, logs, alarms, and dashboards.

On the other hand, CloudWatch is not the operational system itself. The process for decision-making after notification, initial troubleshooting, recovery work, and reporting will be designed separately.

Basic monitoring capabilities with CloudWatch

CloudWatch Metrics is used to collect metrics for AWS resources. You can check things like EC2 CPU usage, network traffic, EBS I/O, RDS connection count and CPU usage, load balancer request count and error count, and more.

CloudWatch Logs is used to aggregate application logs and OS logs. Tracking increases in errors, processing failures, authentication errors, etc., can help not only investigate the cause of failures but also detect anomalies early.

CloudWatch Alarms sends notifications when metrics exceed a certain threshold. For example, it can detect persistently high CPU usage, low available RDS capacity, and an increase in ALB 5xx errors, and can be designed to send notifications in combination with Amazon SNS, chat integration, and on-call management tools.

CloudWatch Dashboards are used to view the status of multiple resources on the same screen. If both the local team and headquarters can access the same metrics, it speeds up the sharing of information during outages.

CloudWatch log storage and custom metrics also impact costs.

CloudWatch is a basic AWS monitoring service, but costs increase with configuration. Factors such as log ingestion volume, retention period, number of custom metrics, number of alarms, and Logs Insights usage all affect billing.

Logs, in particular, can become a cost-increasing factor if the scope of data to be stored is too broad. While storing application logs, OS logs, access logs, and audit logs for extended periods can be useful for investigations, it also increases costs and management burden.

In monitoring design, you need to decide not only what data to collect, but also the granularity of the data, how many days to store it, and under what conditions to send notifications. If you store and send notifications for even low-priority logs under the same conditions, it will increase not only costs but also operational burden.

Considerations when using it in conjunction with third-party monitoring tools.

AWS-native monitoring can be designed around CloudWatch. On the other hand, for companies using existing monitoring infrastructure, or those that want to monitor on-premises environments, other clouds, and SaaS applications all together, using third-party tools in conjunction with AWS is also an option.

Using tools like Datadog, Zabbix, and Mackerel, it may be possible to perform integrated monitoring, including environments outside of AWS. If your headquarters already has a monitoring infrastructure, you can also consider designing a system to integrate CloudWatch data with your existing tools.

However, the more tools you add, the more complex operations become. Duplicate alerts, scattered notification destinations, and an increase in confirmation screens can delay decision-making during failures. It's important to define the boundaries between what can be handled by AWS native tools and what needs to be integrated into the existing monitoring infrastructure.

Using CloudWatch alone does not complete the operational system.

Even if CloudWatch detects an anomaly, operations will stop without response rules. You need to pre-define who receives alerts, which notifications to prioritize, and what to check during the initial troubleshooting phase.

For example, even if an increase in CPU usage is detected, it's impossible to know whether it's a temporary load or a sign of an application failure without further investigation. Similarly, a decrease in RDS capacity may require immediate action in some cases, while in others it can be handled with planned expansion.

Furthermore, even if you have configured CloudWatch notifications, troubleshooting will be delayed if there is no one available to check them at night or on weekends. It's important to define the scope of on-site checks, the conditions for escalating to headquarters, and the scope of work to outsource to external partners.

CloudWatch is a mechanism for detecting anomalies. To ensure stable operation, it is necessary to incorporate the decision-making, response, and reporting processes after notification into the operational design.

Design a response flow after a monitoring alert is triggered.

Monitoring alerts are merely the starting point for incident response. Simply receiving notifications does not allow for assessment of the scope of impact or progress in recovery efforts.

After an alert is triggered, the process is designed in stages: detection, notification, initial troubleshooting, recovery, and reporting. If the response flow remains unclear, there is a risk of overlapping confirmations between the local team and headquarters, or conversely, no one taking action at all.

Consider detection, notification, initial troubleshooting, and recovery response separately.

First, clearly define what needs to be detected. Priorities will vary depending on the type of alert, such as CPU usage, memory usage, disk capacity, error rate, log anomalies, security events, and cost increases.

Next, define the notification recipients. Sending all alerts to the same person will cause them to get buried among less important notifications. Separate alerts into those with a high business impact, those that only require confirmation, and those that can be handled later, and design the notification recipients and response times accordingly.

The initial troubleshooting involves identifying the affected system, the time of the incident, its duration, recent configuration changes, the impact on users, and related logs. Based on this, a decision is made on whether to proceed with recovery work, monitor the situation, or escalate the decision to headquarters.

In recovery operations, it's crucial to define who can perform which tasks. Tasks such as server restarts, scaling, capacity expansion, configuration changes, and restoring from backups cannot be performed without the necessary permissions and criteria.

Routine responses can sometimes be automated using tools such as EventBridge, Lambda, and AWS Systems Manager Automation. However, since automated recovery is highly susceptible to the impact of malfunctions, it should be designed with a limited scope, specifying the target tasks, execution conditions, and approval requirements.

In addition to monitoring, we will also check the redundant configuration and backups.

Monitoring is a mechanism for detecting anomalies. If the system is not configured to allow for recovery after detection, the options for responding will be limited even if an alert is received.

In a production environment, we also verify the Multi-AZ configuration, Auto Scaling, backups, snapshots, and restore procedures. If RDS remains in a single-AZ configuration, the impact of a failure will be significant. Even with backups, they are useless in the event of an actual failure unless the restore procedures and recovery time have been verified.

We will also organize RTO and RPO along with the monitoring design. If it is not decided which systems to restore and within what timeframe, and to what point in time the data should be restored, decisions after an alert is triggered will be inconsistent.

Instead of simply increasing the number of monitoring items, we also need to confirm that the system is configured to recover after detecting a failure.

Determine the division of roles between the Thai subsidiary and the Japanese head office.

If the roles of the local team and headquarters remain unclear, decision-making during a system failure will be delayed. It's important to define the roles beforehand: the local team will assess the impact on operations, headquarters will handle overall policies and key decisions, and external partners will be responsible for initial technical troubleshooting and routine responses.

For example, the local team will check whether the store or business systems are being affected. The head office will determine the acceptable downtime, recovery priority, customer impact, and security considerations. If an external partner is involved, they can be asked to perform tasks such as log review, metrics review, routine restarts, and root cause investigation.

Taking into account the time difference and holiday differences mentioned above, we will also decide whether to use Japan Standard Time or Thai Standard Time as the basis for notifications and response times.

We will also define escalation conditions. Criteria will be set for sharing information with headquarters if the issue is not resolved within a certain time, if there is a customer impact, if there is a security risk, or if cost increases continue.

If roles and responsibilities are determined solely by the name of the person in charge, they become dependent on that individual. Defining the department, response time, contact method, decision-making authority, and work authority will ensure that responses continue even when the assigned person is absent.

Key points to check when outsourcing AWS monitoring and operations to an external partner in Thailand

When outsourcing monitoring and operation to an external partner, it's crucial to differentiate between tasks to be outsourced and those to be kept in-house. While monitoring setup and initial troubleshooting can be outsourced, this doesn't mean you can completely delegate tasks like assessing business impact and determining recovery strategies.

If the scope of the request remains unclear, confusion will arise after an alert occurs regarding "who will see it," "how far the response will go," and "when to share it with headquarters." Before signing a contract, confirm the scope of response, response time, contact method, and reporting format.

Tasks that are easy to outsource to external partners

Tasks that are easy to outsource to external partners include monitoring design, CloudWatch configuration, alert design, log review, initial troubleshooting, routine recovery, and monthly reporting. Security monitoring and cost review may also be included in the scope of services.

For example, tasks such as initial checks when abnormal CPU usage or error rates are detected, log checks, investigation of the scope of impact, routine restarts, and identification of unnecessary resources are easily outsourced. If the local staff member is not dedicated to AWS, outsourcing these tasks can reduce the burden on daily operations.

However, not all responses can be outsourced. Recovery decisions that affect customers, operations that require business interruption, and changes related to security policies must be reviewed and approved by our company.

Things that should be decided within your company

Before engaging an external partner, define your own criteria. If the acceptable downtime, recovery priorities, assessment of business impact, security policies, and work approval processes are unclear, the external partner will be unable to proceed.

For example, you should decide whether external partners are allowed to restart servers, whether approval is required to expand database capacity, and who is authorized to change security groups. If the authority to make these decisions remains unclear, delays in confirmation will occur during failures.

We will also define the conditions for sharing information with headquarters. Without escalation criteria, responses will be inconsistent, such as when there is a customer impact, when recovery is not possible within a certain time, when there are security risks, or when unexpected cost increases occur.

Let's consider the roles of AWS support plans and external partners separately.

When designing a system for handling incidents, you should also check your company's AWS support plan. The scope of technical support, how to contact support, response times, and available operational support vary depending on the support plan.

AWS support can be used for technical inquiries about AWS services and to confirm issues on AWS's end. However, it does not cover troubleshooting, monitoring design, recovery work, or reporting to local or headquarters for your own applications.

When using an external partner, clearly define the scope of inquiries to AWS Support and the scope of initial troubleshooting and operational support provided by the partner. If an AWS infrastructure issue is suspected, defining who will handle inquiries, the process for checking the situation, and how to report to headquarters will clarify the actions to take during an outage.

Scope of actions to be checked

When choosing an external partner, check not only what they monitor, but also the scope of their response after an alert. The quality of operation will vary depending on whether they only provide notifications, perform initial troubleshooting, or handle recovery work.

The items to check include response time, supported languages, contact methods, whether reporting to the Japanese headquarters is possible, escalation procedures in case of incidents, scope of AWS technical support, security monitoring, cost management, and whether monthly reports are provided. We will also look at the reporting system that allows not only the local team but also the headquarters to understand the situation.

We will also check if monitoring items can be reviewed after deployment. In an AWS environment, the monitoring targets change due to system additions, increased access, and configuration changes. Leaving the initial settings as they are may result in unnecessary alerts remaining or new risks not being detected.

Summary: AWS monitoring in Thailand should be designed not only with monitoring settings in mind, but also with an operational structure in mind.

AWS monitoring checks not only the operational status of resources, but also logs, configuration changes, security risks, AWS service outage information, and cost fluctuations. CloudWatch is effective for metric collection, log management, alarm notifications, and visualization through dashboards. On the other hand, increasing log storage and custom metrics will also increase monitoring costs.

Even with monitoring tools in place, operations will not be stable unless the response after an alert is determined. If notification recipients, initial troubleshooting responsibilities, and sharing conditions are not designed in advance, response after a problem occurs will grind to a halt. The time difference between Thailand and Japan, as well as differences in holiday calendars, must also be reflected in the design of notification recipients and response times.

To ensure stable AWS operation in Thailand, it's necessary to clearly define the impact on local operations, headquarters' management policies, the scope of external partners' responsibilities, and the scope of AWS support plans. This involves organizing monitoring items, notification recipients, response times, decision-making authority, reporting methods, and recovery procedures to create a system where incident response doesn't depend on individual personnel.