Recently I have been working with the AWS EC2 Image Builder service for an Innablr client who was in need of a variety of hardened (or “Golden”) Windows images for use across their organisation.
EC2 Image Builder is AWS’s competing service for Hashicorp’s Packer, with some nice features and some annoying ones too.
In this blog post I’ll explain how to use the EC2 Image Builder service to build a Windows Server 2019 image with components provided by AWS and a custom component to apply some CIS level one standards to the image.
A hardened image is an operating system image that has been configured in a more secure manner than what it is “out of the box”, the primary purpose is to turn off features or settings not relevant to your workloads which in turn reduces the security risk of the underlying operating system.
The task at hand
As stated earlier, the plan is to demonstrate how to use EC2 Image Builder to produce a Windows Server 2019 image or Amazon Machine Image, what I didn’t mention is some of the requirements surrounding that image.
To better demonstrate how EC2 Image Builder works, let’s run with the following requirements for the purpose of this blog post.
We want to use the full version of Windows Server 2019 for the base image
We also want to make sure it is up-to-date with the latest patches
We also will be using .NET core based applications, so we need the framework installed on the image ahead of time
The image must be built to operate in a secure manner and any known controls or configurations that should be turned off or altered are executed but not to the extent where it’s a detriment to it’s potential users
Getting started
For this demonstration of using EC2 Image Builder we will be using a single CloudFormation stack, and a PowerShell script to apply our desired CIS benchmark level one configurations to the Windows Server image.
The code repository for this demonstration can be found on Github here.
Note: Please feel free to raise issues or pull requests on the above repository if you discover any issues or improvements I will be happy to take a look.
What does the EC2 Image Builder service consist of?
Pipeline
The primary service provided by EC2 Image Builder is the pipeline.
A pipeline is a specifically targeted CI/CD system for building Amazon Machine Images which can be used with the AWS EC2 service. A pipeline typically defines;
What is to be built?
Where it is to be built?
Whom gets access to the final image?
How often to check for updates?
The pipeline also references the following other EC2 Image Builder features;
Recipe - What gets built/baked into the image
Infrastructure configuration - Where the EC2 instance is spun up, what size is the instance, what network it should run on, security group to use etc.
Distribution - This determines where you want your generated AMI copied to (from an AWS Account perspective)
You can find more information regarding EC2 Image Builder pipelines via AWS’s official documentation
Recipe
The recipe is exactly what you would expect it to be, it defines what operating system you wish to build on, what components you would like to install, as well as what tests you would like to run.
Once a version of an image recipe has been defined, it cannot be amended. If you need to make a change to an existing recipe you are generating a newer version of that recipe, you cannot amend the previous version.
Note: While it will be mentioned below, versioning is a necessary evil when utilising the EC2 Image Builder service. Using a global version value for your entire CloudFormation stack, while possible, it is recommended custom components (which also require a version number) either be provided a separate version number to the recipe version if contained in the same CloudFormation stack, or the custom component is not part of the existing CloudFormation stack.
You can find more information regarding EC2 Image Builder recipes via AWS’s official documentation
Infrastructure configuration
This is an optional feature of an EC2 Image Builder pipeline, while optional is subjective in this case. An infrastructure configuration is always needed, however you can choose to utilise all the default settings that AWS has defined rather than providing specific resources to use such as the subnet and security group you wish your instance to use during it’s build and test phases.
With the infrastructure configuration you can specify some of the following configurations that your EC2 instance will use when your image is being built.
Instance Size/Type - Depending on what components you are installing on the instance you may want to use a larger instance type for the build and test process.
Amazon will default to the m5.large instance type if this is not specified
Note: Based on experience usage of smaller instance types, especially for Windows based images is not recommended as it can cause some unintended issues during both the build and test phases
SNS Topic - This will let you specify an SNS topic EC2 Image Builder can send notifications about the status of pipeline execution, these notifications consist of both success and failure notifications
VPC - If you have a requirement or would prefer to run the EC2 instance within your VPC you can define it here
There is also a requirement to then set a subnet id and a security group which provides even further options for restriction of the instance being run and/or provides access to other AWS resources within the custom defined VPC to assist in building the image
Terminate instance on failure - This is a toggle item, it is recommended you don’t terminate the instance upon failure as it can be easier to troubleshoot on the instance versus reading log files when first starting out
Note: Generally it’s recommended to leave this unchecked when first building your pipeline etc. However, upon consecutive executions it can be turned on to avoid unintended costs
Important: Make sure you assign an existing key pair in your infrastructure configuration, otherwise you will not be able to login to the instance regardless of this setting being disabled
Key Pair - If you do wish to be able to access the instance post failure, you will need to set a desired key pair to facilitate remote access to the instance
Note: I personally only tested on a custom VPC and security group in which port 3389 was open, using AWS’s default setup may block your ability to access the machine remotely
Logs - This will write all of the build and test logs for a pipeline execution to an S3 bucket. It is highly recommended this be set as the logs can be extremely useful when needed to troubleshoot build or test issues.
You can find more information regarding EC2 Image Builder infrastructure configurations via AWS’s official documentation
Distribution
The distribution configuration allows you to specify the output name of your Amazon Machine Image, set launch permissions (allow other accounts to use your AMI), as well as allow you to copy the AMI to other accounts (if your role has access).
It also allows you to set important tags that can be carried over to other AWS accounts when the image is copied.
Based on my usage, our client had a substantial amount of AWS accounts, thus we went with adding the AWS account Ids to the launch permission configuration of the Amazon Machine Image that is output post a successful build and test. This was handled post successful build via a Lambda function, if you have an alternative method or AWS service which has the full list of your AWS accounts you could populate it via the CloudFormation stack natively rather than using a Lambda function post.
If you have a substantial amount of AWS Accounts you wish to copy the image to, this is not generally recommended as it will increase the runtime of your pipeline, it is recommended you simply set launch permissions for each AWS account against the single image.
You can find more information regarding EC2 Image Builder distrubution configurations via AWS’s official documentation
Building the CloudFormation stack
IMPORTANT: EC2 Image Builder deploys EC2 instances on a VPC to allow the service to build and test the image. In this demonstration we will not be defining a specific VPC subnet nor security group, from testing if you have a default VPC in the account you are running the CloudFormation stack in, this will be used if nothing is specifically defined in the AWS::ImageBuilder::InfrastructureConfiguration CloudFormation resource.
Parameters
We’re going to need four parameters to assist us with building this CloudFormation Stack.
BaseImageArn - This will be the baseline or starter Amazon Machine Image ARN we will be using to build our hardened image on top of. It is recommended you use the Image Builder base images as they will generally come pre-installed with the necessary tooling to function correctly
CisScriptS3Uri - This is the S3 bucket uri of the CIS level one PowerShell scripts for our custom component, leave it default
Description - A general description of the image. You can ignore this being a parameter if you would prefer to individualise the description for each resource type
ImageName - This will essentially be the name for your EC2 Image Builder resources as well as used for the output name of the image when it has successfully built
Version - This is necessary for the image recipe and user defined components. More specifics as to why will be touched on for each respective resource
CloudFormation parameters
---
AWSTemplateFormatVersion: 2010-09-09
Description: 'EC2 Image Builder Stack - Windows Server 2019'
Parameters:
BaseImageArn:
Type: String
Description: "The base AMI ARN to be used for your Image Recipe to build upon."
CisScriptS3Uri:
Type: String
Description: "The URI of the CIS level one scripts for the custom component"
Default: "innablr-demo-resources/image-builder-component/"
Description:
Type: String
Description: "A general description of your image resources"
Default: "Hardened Windows Server 2019 image"
ImageName:
Type: String
Description: "Desired name of your image resources"
Default: "windows-server-2019-hardened"
Version:
Type: String
Description: "The semantic version of the deployment. You must increment this value for every CloudFormation change"Fairly standard approach to parameters, all of these can use the default except the obvious Version parameter as you will need to change this for every deployment of your CloudFormation Stack.
Note: If you do not have a version control system available to your source control, you could hardcode this version with a default value, you will simply need to remember to increase the version number for every change you will be deploying to AWS.
Resources
Bucket
EC2 Image Builder allows you to store logs for every pipeline execution you run for your image.
While this is optional, it is highly recommended you setup a bucket to receive these logs as troubleshooting will be necessary as you begin to tweak and fine tune your Recipe
You can remove the lifecycle configuration if desired, this is simply to ensure you remove stored data in a timely manner to reduce costs.
SNS Topic
As mentioned earlier in the Infrastructure configuration section, we desire an SNS Topic that our pipeline can send status updates to for our pipeline executions. Pipeline success or failure notifications are generated and sent to this topic.
We won’t be setting any specific configurations for the topic within this demonstration.
Note: As an enhancement you could have a lambda function monitor these notifications and then generate alerts based on the notification message.
Image builder role and instance profile
EC2 Image Builder needs its own set of permissions to perform actions against the EC2 instance itself during the build and test phases as well as the ability to interact with the published Amazon Machine Image as well as the ability to send notifications to our SNS topic.
Some permissions in this role relate to requirements of some components (e.g. ec2:AttachVolume). This is to allow a test component to attach and detach an EC2 volume to the instance for testing purposes to ensure the image we are building supports this functionality.
The other managed roles are required to allow EC2 Image Builder to interact with the instance itself.
Latest image ARN
To make it easier to reference our Amazon Machine Image ARN within the build account, we will define the ARN in an SSM parameter. This is an optional CloudFormation resource and has no bearing on the outcome of your image being built.
CloudFormation resources stage #1
Resources:
Bucket:
Type: AWS::S3::Bucket
Properties:
AccessControl: Private
LifecycleConfiguration:
Rules:
- Status: Enabled
ExpirationInDays: 7
Topic:
Type: AWS::SNS::Topic
Role:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- "ec2.amazonaws.com"
Action:
- "sts:AssumeRole"
Policies:
- PolicyName: ImageBuilderPolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- "ssm:SendCommand"
- "ec2:CreateTags"
- "ec2:AttachVolume"
- "ec2:CreateTags"
- "ec2:CreateVolume"
- "ec2:DeleteVolume"
- "ec2:DescribeVolumes"
- "ec2:DetachVolume"
Resource: "*"
ManagedPolicyArns:
- "arn:aws:iam::aws:policy/EC2InstanceProfileForImageBuilder"
- "arn:aws:iam::aws:policy/AWSImageBuilderFullAccess"
- "arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM"
- "arn:aws:iam::aws:policy/service-role/AmazonSSMAutomationRole"
- "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
Path: "/"
Roles:
- !Ref Role
ImageArn:
Type: AWS::SSM::Parameter
Properties:
Name: !Sub "/${AWS::StackName}/image-arn"
Type: String
Value: !Sub "arn:aws:imagebuilder:${AWS::Region}:${AWS::AccountId}:image/${Name}/x.x.x"This concludes the first stage of our CloudFormation resource definitions.
Pipeline
This is where we point to our yet to be defined recipe, infrastructure configuration and distrubution configuration.
The main values to consider from a customisation standpoint are;
Description - A basic description of your pipeline. We are using the CloudFormation parameter to handle this.
Name - The name of your pipeline. We are using the CloudFormation parameter to handle this.
Schedule - The schedule defines how often and when EC2 Image Builder will run.
You can set this to run a build based on your cron expression regardless of updates or otherwise.
If using the EXPRESSION_MATCH_AND_DEPENDENCY_UPDATES_AVAILABLE setting as we are in this demonstration if the base image or any of the components have a version change EC2 Image Builder will begin a build upon its next cron run.
If you were wanting to keep in cadence with Microsoft’s Tuesday patch cycles you could opt to run a schedule once a week every Wednesday to ensure your Windows image has the latest patches
The cron expression used in this demonstration will check for updates every day at 12:00PM UTC / 10:00PM AEST
Infrastructure configuration
As explained earlier, the infrastructure configuration defines where EC2 Image Builder is to spin up the EC2 instance for the build and test phases.
The key properties we need to set for this demonstration are;
InstanceProfileName - This is the instance profile the EC2 instance will use when performing its build and test tasks, we defined this earlier
Logging - Here we define the bucket we will be sending our build and test logs to, along with the S3 structure we wish to utilise
SNS Topic - As explained earlier, we wish to have notifications provided to let us know if the pipeline has succeeded or failed.
As mentioned earlier, you can configure this resource to use a specific subnet and security group where desired, but for the purposes of this demonstration, we are letting AWS handle these aspects.
DistributionConfiguration
This is where we define where to share the image after it has been successfully built, this also determines what the Amazon Machine Image name will be along with a description etc.
For the purposes of this demonstration we are only providing launch permissions for the account we are building the image in.
You can amend the LaunchConfiguration property to include more AWS Account Ids to allow greater use across your AWS landscape. How you provide these additional AWS accounts is at your discretion.
LaunchPermissionConfiguration:
UserIds: [!Sub '${AWS::AccountId}']Recipe
This is where we define what happens to our image during our build and test phases.
The Components property is what we need to focus on. AWS provides a number of Windows based EC2 Image Builder components, before we reference our to be built custom components, we will define some of these AWS provided components.
Note: The order in which you define these components is the order in which EC2 Image Builder (and the AWSTOE service) will execute them. I recommend (unless necessary) that all build components are defined first with any tightened security controls executed last and then all your test components follow after that.
Based on our earlier defined components, we want to use the update-windows component, the dotnet-core-runtime-windows component and to assist in securing our image we will use the stig-windows-medium component. These are all the build phase components.
Note: You can find out more about STIG and EC2 Image Builder here
For testing we simply want to validate that we can attach, write to and detach and EBS volume from our EC2 instance, so we will be using the ebs-volume-usage-test-windows component which will run in the test phase
Note: You MUST have at least one build phase and test phase component defined
Based on the above our Components property now looks like the following;
- ComponentArn: 'arn:aws:imagebuilder:ap-southeast-2:aws:component/update-windows/x.x.x'
- ComponentArn: 'arn:aws:imagebuilder:ap-southeast-2:aws:component/dotnet-core-runtime-windows/x.x.x'
- ComponentArn: 'arn:aws:imagebuilder:ap-southeast-2:aws:component/stig-build-windows-medium/x.x.x'
- ComponentArn: 'arn:aws:imagebuilder:ap-southeast-2:aws:component/ebs-volume-usage-test-windows/x.x.x'You can find the ARNs by using the EC2 Image Builder web console.
Additionally if you wish to use specific versions of these components you can amend the semantic x.x.x to the desired version in this section or alternatively you can define a version parameter and sub that value in the ComponentArn.
Custom Component
As part of our requirements, we wish to apply CIS level one configurations to our Windows image. Unfortunately, AWS does not provide a component for this so we will need to create a custom component and use a PowerShell script to facilitate this.
I won’t be going into the specifics of the Data definition in this blog post for AWSTOE. You can find out more information about AWSTOE and EC2 Image Builder here
The CIS Level one configuration script is a modified version of a script found here by NVISO Security.
Modifications are specific to allowing remote access to the machine as by not doing so administration is impossible.
There are a few aspects of a custom component that are worth noting;
If you wish to use the component for multiple recipes, it is recommended the component not be part of a pipeline stack
The reason for this is versioning. In this demonstration, we are sharing the version number and CloudFormation can safely update both the recipe and the component version without conflict
However, if the custom component defined here is referenced by another recipe this CloudFormation stack does not have control over, you will be unable to update the component due to the previous version being in use. This also applies to recipes.
There are many different ways to write and provide scripts to your custom component, the approach taken in this demonstration is not necessarily “the best way”
Now that we have our custom component defined, we need to update our Recipe to include it. We achieve this by simply referencing the component resource
- ComponentArn: !Ref CisLevelOneComponentCloudFormation resources stage #2
Pipeline:
Type: AWS::ImageBuilder::ImagePipeline
Properties:
Description: !Sub "${Description}"
ImageRecipeArn: !Ref Recipe
InfrastructureConfigurationArn: !Ref InfrastructureConfiguration
Name: !Ref ImageName
DistributionConfigurationArn: !Ref DistributionConfiguration
Schedule:
PipelineExecutionStartCondition: EXPRESSION_MATCH_AND_DEPENDENCY_UPDATES_AVAILABLE
ScheduleExpression: "cron(0 0 12 1/1 * ? *)" # 12:00PM UTC / 22:00PM AEST Daily
InfrastructureConfiguration:
Type: AWS::ImageBuilder::InfrastructureConfiguration
Properties:
Description: !Ref Description
TerminateInstanceOnFailure: false
InstanceProfileName: !Ref InstanceProfile
Name: !Ref ImageName
Logging:
S3Logs:
S3BucketName: !Ref Bucket
S3KeyPrefix: !Sub "${ImageName}/Logs"
SnsTopicArn: !Ref Topic
DistributionConfiguration:
Type: AWS::ImageBuilder::DistributionConfiguration
Properties:
Name: !Ref ImageName
Description: !Ref Description
Distributions:
- Region: ap-southeast-2
AmiDistributionConfiguration:
Name: !Sub "${ImageName}-{{imagebuilder:buildDate}}"
Description: !Ref Description
AmiTags:
Name: !Ref ImageName
LaunchPermissionConfiguration:
UserIds: [!Sub "${AWS::AccountId}"]
Recipe:
Type: "AWS::ImageBuilder::ImageRecipe"
Properties:
Name: !Ref ImageName
Version: !Ref Version
ParentImage: !Ref BaseImageArn
Description: !Ref Description
Components:
- ComponentArn: "arn:aws:imagebuilder:ap-southeast-2:aws:component/update-windows/x.x.x"
- ComponentArn: "arn:aws:imagebuilder:ap-southeast-2:aws:component/dotnet-core-runtime-windows/x.x.x"
- ComponentArn: "arn:aws:imagebuilder:ap-southeast-2:aws:component/stig-build-windows-medium/x.x.x"
- ComponentArn: !Ref CisLevelOneComponent
- ComponentArn: "arn:aws:imagebuilder:ap-southeast-2:aws:component/ebs-volume-usage-test-windows/x.x.x"
Tags:
Name: !Ref ImageName
CisLevelOneComponent:
Type: AWS::ImageBuilder::Component
Properties:
Name: "CISLevelOneConfigurations"
Description: "Retrieves the custom CIS level one configuration PowerShell script and then executes it"
Platform: "Windows"
Version: !Ref Version
Data:
Fn::Sub: |
name: 'Download and execute a CIS configuration script'
description: 'Execution of CIS level one configuration script'
schemaVersion: 1.0
phases:
- name: build
steps:
- name: CreateTempDirectory
action: ExecutePowerShell
onFailure: Abort
maxAttempts: 3
inputs:
commands:
- New-Item -Path "C:\" -Name "temp" -ItemType "directory"
- name: DownloadSetupScript
action: S3Download
onFailure: Abort
maxAttempts: 3
inputs:
- source: s3://${CisScriptS3Uri}/cis_config_setup.ps1
destination: C:\temp\cis_config_setup.ps1
- name: DownloadConfigScript
action: S3Download
onFailure: Abort
maxAttempts: 3
inputs:
- source: s3://${CisScriptS3Uri}/cis_config.ps1
destination: C:\temp\cis_config.ps1
- name: ExecuteScript
action: ExecutePowerShell
onFailure: Abort
maxAttempts: 3
inputs:
commands:
- cd c:\temp
- .\cis_config_setup.ps1
- name: Cleanup
action: ExecutePowerShell
inputs:
commands:
- cd C:\
- Remove-Item -LiteralPath C:\temp -Force -RecurseThat’s it, our CloudFormation stack is ready for use!
Deployment
Given we have written this demonstration stack in a single CloudFormation template you may deploy it any way you chose, simply remember the Version parameter must be updated with any changes to the Recipe or Component, to save confusion I recommend incrementing the version number for every change regardless of which resource is amended.
You can name the CloudFormation stack anything you choose.
Using your new image pipeline
Go to EC2 Image Builder
Locate your pipeline and select it’s checkbox
Click the Actions button and select Run Pipeline
A banner will pop up, click the View Details button.
You will be then redirected to a page where you can monitor the status of the pipeline run
In addition you can monitor the EC2 Instance dashboard to see your build phase instance being provisioned
Logs will begin immediately populating in your S3 bucket and will contain information about the execution of each component
Once the pipeline has completed either successfully or has failed the SNS topic will receive a notification and the page you were redirected to when you first ran the pipeline should reflect the status of the pipeline run.
Success
Ensuring you have followed along and used the provided source code to build your own Windows hardened Amazon Machine Image, I hope you got your own version working!
EC2 Image Builder helps engineers focus on what they want their image to contain and how they wish it to be configured without having to worry about standing up a traditional CI/CD pipeline or place it in an environment to function.
AWS take care of monitoring when a newer image needs to be produced for you and the only costs associated with the service is the EC2 Instance used during the build and test phases, whereas with a traditional Packer on AWS architecture you would be paying for AWS CodeBuild or an equivalent service to monitor and manage the build process.
I hope you found this walkthrough useful and possibly insightful to the benefits of EC2 Image Builder
Innablr is a leading cloud engineering and next generation platform consultancy