AWS Configuration Management Approach: How to Track Configuration Changes and Prevent Reliance on Individuals

In Amazon Web Services (AWS), it can become difficult to keep track of the current configuration and the history of changes as resources are added and configurations are modified repeatedly. In environments where multiple people are making changes manually, inconsistencies in configurations and reliance on individual expertise are more likely to occur.

To address these challenges, it's necessary to not only record configurations and change history with AWS Config, but also to utilize Infrastructure as Code (IaC) such as AWS CloudFormation to manage the changes, including how they were made.

This article outlines the types of services and representative services that require configuration management on AWS, and explains how to continuously operate them while understanding and standardizing the configuration.

What is configuration management in AWS?

AWS configuration management records the settings and change history of EC2, VPC, IAM, and other services, allowing you to identify any deviations from your intended configuration.

As resource changes accumulate, the design specifications and the actual environment may no longer match. Therefore, we use tools like AWS Config to continuously monitor the actual configuration.

Why configuration management is necessary on AWS

When multiple people individually change settings, discrepancies can arise between the design configuration and the actual environment, and it becomes difficult to track the reasons for changes. Investigating the cause of failures and reviewing settings also takes more time.

By recording configurations and change histories, and adopting IaC (Infrastructure as Code) where possible to standardize change procedures, you can reduce configuration inconsistencies. The AWS Well-Architected Framework also recommends managing infrastructure as code and using a system to track changes.

Differences from on-premises configuration management

The purpose of configuration management is largely the same whether you're using AWS or on-premises. The difference is,The ability to add and modify resources quickly..

On-premises environments often involve changes that require the procurement and installation of physical equipment. In contrast, AWS allows for immediate resource changes via APIs and the AWS Management Console, making it easier for the actual environment to deviate from its design state.

Therefore, it is necessary to not rely solely on updating the design document, but to continuously monitor the actual configuration and change history.

Main targets for configuration management on AWS

Configuration management encompasses three main areas: AWS resource settings, configuration change history, and the operating system and software within EC2 instances. Since each area requires different information and management methods, we'll organize them separately.

AWS resource configuration information

It manages configuration information for AWS resources such as EC2, VPC, security groups, IAM, and S3.

For example, this includes security group communication rules, the VPC and subnets where EC2 instances are deployed, and the S3 bucket settings. It's important to understand not only the configuration described in the design document, but also the actual state of the running environment.

For information on how to check for configuration errors in your AWS environment, please refer to the following article.

>> How to find AWS configuration vulnerabilities? Types of security assessments and how to proceed.

Configuration change history

Configuration management records not only the current settings, but also "when and what changed."

If a change history is maintained, you can check which settings were changed before and after a failure or unexpected behavior occurs. By comparing it with the operation history, it becomes easier to trace the sequence of changes.

OS and middleware configuration

If you are using EC2, the inside of the instance is also subject to configuration management. You should keep track of the OS settings, installed software, and running services.

The information managed for AWS resource settings and the OS/software within EC2 instances are different. Managing them separately makes it easier to pinpoint where changes or problems occur.

Main services and methods that can be used for configuration management on AWS

AWS configuration management utilizes multiple services and methods depending on the purpose. Typical examples include AWS Config, AWS CloudFormation, and AWS Systems Manager.

AWS Config | Record configurations and change history

AWS Config is a service that records the configuration of corresponding AWS resources and allows you to view the change history.

You can track when security group settings were changed, how the configuration of EC2 instances changed, and more. AWS Config Rules can also be used to evaluate whether each resource complies with predefined rules, helping to detect configuration problems.

AWS CloudFormation | Manage configurations with code

AWS CloudFormation is a service that allows you to define the configuration of AWS resources using templates and manage your infrastructure as code.

Since environments can be built from the same template, variations in manual configuration are reduced. Managing templates with Git or similar tools allows for a workflow where change history is recorded as code, reviewed, and then implemented.

While AWS Config tracks the state and changes of the real environment, CloudFormation is used to define and reproduce the desired configuration in code.

AWS Systems Manager | Manage the state of your operating system and software.

AWS Systems Manager is a service for managing managed nodes such as EC2. It can also be used for managing the internal configuration of servers, including the operating system and installed software.

Systems Manager Inventory allows you to collect information on applications, services, and more. State Manager helps maintain managed nodes and other elements in a defined state, minimizing configuration discrepancies.

How to continuously manage AWS configuration

AWS configurations change even during operation. We establish a process that defines what to manage and the criteria for management, understands the current situation, and allows us to check for discrepancies after changes.

1. Determine what to manage and what the ideal configuration should be.

First, determine what you want to manage and what state you want to maintain. Instead of managing everything at the same level of granularity, it's easier to prioritize resources that have a significant impact on security and availability.

For example, you can define criteria such as the scope of permissions allowed by security groups and whether or not encryption is used. AWS Config Rules and Conformance Packs can be used to evaluate whether each resource complies with those criteria.

2. Visualize the current configuration and change path.

Next, we'll check the resources and settings that are actually running. Even if design documents exist, they may not necessarily match the current environment.

In addition, we will organize where the changes are coming from, such as the AWS Management Console, CLI, IaC, and operational tools. Being able to compare configuration changes with operation history makes it easier to track the history of the changes.

3. Standardize and make reproducible configurations using IaC.

We will gradually migrate configurations that are currently managed manually to IaC such as AWS CloudFormation.

By managing code with tools like Git and establishing a process for changes after review, you can reproduce the same configuration while minimizing variations in setup methods.

You don't need to implement IaC (Infrastructure as Code) for everything from the start. It's more practical to start with resources that change frequently or areas where configuration errors have a significant impact, and then gradually expand the scope.

The following article also discusses configuration management of development environments using CloudFormation and other tools.

>> AWS Development Environment Building Guide | Explaining everything from service selection to security and costs

4. Detect unintended changes or configuration differences.

Even with Infrastructure as Code (IaC), if changes are made directly from the AWS Management Console or similar tools, discrepancies will arise between what is defined in the code and the actual environment.

CloudFormation drift detection allows you to see the difference between the settings specified in the template or parameters and the actual configuration for the corresponding resources. Using AWS Config Rules, you can also identify settings that deviate from predefined criteria. 

5. Evaluate the detected differences and use the results to make improvements.

Once a difference is detected, the intention behind the change is first confirmed. If it is a necessary change, it is reflected on the IaC side; if it is an incorrect change, the production environment is reverted to its original state.

For routine rule violations, you can automate the process using AWS Config's remediation actions. On the other hand, for changes with a wide-ranging impact, you should handle them differently depending on the issue, such as having a human review them.

Furthermore, we investigate why the discrepancies occurred and review the change procedures and management rules as needed. Our configuration management operations include ensuring that the same problems do not recur.

The following article provides a detailed explanation of how to review your entire AWS environment and use that information to improve its configuration and operations.

>> What can you learn from an AWS environment audit? Checklist and how to proceed with improvements.

Three key points for establishing AWS configuration management

To establish configuration management as an integral part of your organization's operations, you need to clearly define the rules for changes and where to reference information. When multiple people or AWS accounts are involved, you should establish common operational procedures rather than leaving decisions to individual users.

1. Define the change path and the rules for handling exceptions.

Even if configuration changes are standardized to be made via IaC, there are still situations where changes need to be made directly from the AWS Management Console, such as during incident response.

Therefore, it's important to decide on the usual process for making changes, including who will verify changes made directly and how they will be reflected in IaC. Establishing rules for handling exceptions also helps prevent discrepancies between the actual environment and the managed code from being left unaddressed.

2. Determine where to refer to configuration information and change history.

When configuration information is scattered across design documents, management sheets, and notes from the person in charge, it becomes difficult to determine which version is the most up-to-date.

Determine where to refer to each type of information, such as using AWS Config to check resource configurations and change history, and CloudTrail to track operation history. CloudTrail's event history allows you to view management events for the past 90 days in each AWS region. For longer-term recording, use the trail or CloudTrail Lake event data store. 

3. Apply common rules to multiple accounts.

As the number of AWS accounts increases, configuration management methods and standards tend to vary from account to account.

AWS Config Aggregators allow you to consolidate configuration and compliance data across multiple AWS accounts and regions. You can also combine AWS Organizations and Conformance Packs to deploy common AWS Config Rules across multiple accounts.

Instead of handling each account individually, applying common standards across the entire organization helps to minimize inconsistencies in management as the environment expands.

Summary

AWS configuration management involves using AWS Config, AWS CloudFormation, AWS Systems Manager, and other tools depending on the purpose, ensuring that configurations and changes can be continuously monitored.

In addition, by defining management standards, change routes, and information reference sources, you can reduce inconsistencies in operations across different personnel and accounts.

Instead of trying to manage everything from the start, it's best to begin with resources that have a significant impact on security and availability, and gradually expand the scope.