How to speed up recovery from AWS outages? Design, backup, and operational measures.

Even with redundant systems on Amazon Web Services (AWS), it's not always possible to completely avoid the effects of failures. When preparing for a failure, you need to decide in advance how long it will take to restore the service and to what point in time the data should be restored.

Depending on the objectives, backup, multi-AZ/multi-region, and DR (disaster recovery) strategies will be selected. Equally necessary are recovery procedures and operational structures for when a failure occurs.

This article explains the design and countermeasures for accelerating recovery from AWS outages, recovery procedures, and normal operation. 

When recovering from an AWS outage, first determine the RTO and RPO.

AWS disaster recovery measures begin by assessing the business impact of a system outage and defining the required recovery level as RTO (Recovery Time Objective) and RPO (Recovery Point Objective). Based on these criteria, the backup frequency, redundant configuration, and standby environment are selected.

Organize the impact on operations and prioritize recovery efforts.

Systems handling order processing and payments can impact sales and customer service even with a short outage. On the other hand, systems that can continue operations for a certain period using alternative means require a different level of recovery.

AWS uses Business Impact Analysis (BIA) to understand the impact of system outages on businesses and customers and to define recovery requirements. Classifying business processes and systems by importance and prioritizing recovery allows for faster decision-making during failures.

Set RTO (Recovery Time Objective) and RPO (Recovery Point Objective).

Once recovery priorities are determined, RTO and RPO are set for each system.

• RTO (Recovery Time Objective): The acceptable time from service outage to recovery.

• RPO (Recovery Point Objective): The acceptable time range for data loss in the event of a failure.

For example, if the RTO is 1 hour, you would design backups and data replication to recover service within 1 hour; if the RPO is 15 minutes, you would design backups and data replication to account for a maximum data loss of about 15 minutes.

The higher the level of recovery required, the more resources are needed for redundant configurations, standby environments, and data replication. Realistic targets should be set based on a balance between business impact and cost.

>> Related articles:What can you learn from an AWS environment audit? Checklist and how to proceed with improvements.

Design and countermeasures to support AWS disaster recovery

Depending on your recovery goals, you can choose a configuration such as backup, multi-AZ, or multi-region, and even design the failover method in case of a failure.

Create a state where you can restore from a backup.

To protect against data corruption or loss, regularly back up essential data. Set the backup frequency to match the acceptable level of data loss.

AWS Backup allows you to centrally manage backups across multiple AWS services and automate backup scheduling and retention periods.

>> Related articles:AWS Backup Implementation Guide | Practical knowledge for setting up automatic backups for EC2 and RDS and preventing operational issues

We will also perform a restore to verify whether the data can be used normally and whether it can be recovered within the expected time.

Enhance availability with multi-AZ.

AWS regions consist of multiple Availability Zones (AZs). By distributing workloads across multiple AZs, if one AZ fails, you can continue and recover services using another AZ.

The failover method will vary depending on the AWS services used and the system configuration. Data protection methods and switching procedures should also be designed in advance.

Multi-region preparation for widespread outages

If you anticipate a region-wide failure, you will need to prepare a recovery environment in a separate region.

In terms of configuration, there are two methods: replicating data to a separate region and starting up the environment in the event of a failure, or operating in multiple regions from the start and switching traffic between them.

In a multi-region environment, the scope of design and operation expands to include data synchronization, traffic switching, and rollback after recovery. Therefore, the decision to adopt this approach should be based on the required recovery level and associated costs. 

Choose a disaster recovery (DR) strategy based on RTO/RPO and cost.

When designing disaster recovery across multiple regions, the AWS Well-Architected Framework outlines the following four strategies: 

The shorter the recovery time, the greater the resources and operational burden required during normal operation tend to be. Consider the AWS services used, application configuration, and data synchronization method, and choose a strategy that meets the required recovery level.

AWS recovery procedures when an outage occurs

When a failure occurs, we assess the scope of the impact and select a recovery method according to pre-determined criteria. If the impact on services is significant, we proceed with failover or restore without waiting for a complete identification of the cause, and conduct a detailed root cause analysis after recovery.

1. Check the information about the outage and the scope of its impact.

AWS outages can be checked on the AWS Health Dashboard. Signing in will also allow you to view events related to services and resources within your own account.

For our own systems, we use monitoring information and logs to identify affected resources, error details, and occurrence times, and use this information to determine recovery strategies.

2. Determine the recovery strategy and perform failover restore.

We select the appropriate response based on the type of failure and the recovery objectives. If processing and traffic can be switched to a normal environment, we perform a failover. If data corruption or loss occurs, we determine a recovery point and restore from a backup.

By defining recovery methods and decision-making criteria in a runbook or similar document, you can shorten the time it takes to make decisions and start work after a failure occurs.

3. Verify that the system is functioning correctly after recovery.

This checks application operation, data loss or inconsistency, integration with external systems, and monitoring and alert status.

Recovery is considered complete only after confirming that the infrastructure is operational and that users can access the service normally.

4. Analyze the cause and use it to prevent recurrence.

After service is restored, we will verify the cause of the failure, the factors that caused delays in detection and recovery decisions, whether the recovery procedures functioned as expected, and whether the set recovery objectives were achieved.

We will reflect the identified issues in our architecture, monitoring settings, runbooks, and response systems to shorten recovery time in the event of similar failures.

Operational measures to expedite recovery from failures

To shorten recovery time, in addition to system configuration, monitoring, recovery procedures, response systems, testing, and automation should be established from the outset.

Early detection of failures through monitoring and alerts.

Amazon CloudWatch allows you to monitor metrics for AWS resources and applications, trigger alarms based on certain conditions, and initiate notifications and automated actions.

In addition to defining the targets and thresholds for monitoring, you should also decide what the person receiving the alert should check and under what conditions they should initiate recovery efforts.

>> Related articles:AWS Monitoring Design Basics | Key Points for CloudWatch Utilization and Operation Design

Prepare recovery procedures and response systems in advance.

For each anticipated problem, we will create a runbook or similar document outlining the items to check, the criteria for making a decision, the recovery method, the person in charge, contact information, and the escalation destination.

Systems that operate at night and on holidays require a 24/365 support system. If it is difficult to maintain this in-house, utilizing external support is an option.

Serverworks provides 24/365 monitoring of AWS environments, and depending on your plan, you can also outsource standard incident response and recovery services. 

DR tests will verify whether recovery is actually possible.

DR testing verifies processes such as restoring from backups, failover, and runbook execution.

The verification process is not limited to the success or failure of the recovery. We evaluate the degree to which the objectives were achieved based on the recovery time and data status, and reflect the identified issues in the system configuration, monitoring settings, and recovery procedures. 

Automate and standardize recovery operations.

Automating routine recovery tasks can lead to reduced working time and a decrease in operational errors.

For example, AWS Systems Manager Automation allows you to define a series of actions on AWS resources as a runbook.

Automating tasks is best suited to those with clearly defined processing steps and decision-making criteria that can be tested beforehand. By separating these processes from those requiring human judgment, inconsistencies in responses among different personnel can be minimized. 

Summary

To prepare for AWS outages, we define recovery targets based on the impact on business operations and then select backup, multi-AZ, multi-region, and DR strategies.

In the event of a failure, the recovery method is determined based on the scope of the impact, and failover or restore is performed. Following this, the process involves root cause analysis and measures to prevent recurrence.

By establishing monitoring, runbooks, DR testing, and automation in advance, you can reduce recovery time. If 24/7/365 support is required, utilize AWS operational support to complement your company's monitoring and incident response system.