AWS automation – CloudFormation bootstrapping early lessons – Part 1

Why bootstrapping

One of the empowering and disruptive characteristic of public cloud service such as AWS is that it levels the playing field. A startup has access to the same set of features and potential capabilities at the same cost level as a fortune 100 company has. However, what each company does with cloud service will largely depend on its business objectives, cloud architecture maturity, and its ability to innovate. The approach and outcome will likely differ dramatically.

Developing a DevOps framework and integrate seamlessly with AWS through automation is probably the biggest competitive differentiator among adopters. If you have been working with AWS, you probably noticed the role CloudFormation plays in automation, as a deployment launch pad, and often the glue to integrate with other tools such as scripts, code snippets, and external automation platforms like Chef and Puppet.

CloudFormation supports bootstrapping of instances, which is an essential building block of launching various services and applications. While there are several methods documented by AWS and increasingly more examples shared in the public domain, the initial learning curve still has quite a bit of research and experiments involved. Even in a relatively simple scenario, all the pieces need to fall into places for things to work. A thorough understanding of what happens at each step of the execution is probably fundamental to developing more sophisticated automation capabilities.

Logical Steps

The example used here is to have CloudFormation launch a windows instance bootstrapped with a PowerShell script loaded from S3. Seems simple? The logical sequence actually involves quite a few steps behind the scene:

1. CloudFormation creates stack and launches instance
2. CloudFormation passes meta data to launched instance, which is instructed to get a script from S3, and place it in the defined local directory
3. Using cfn:init, the instance invokes command defined by CloudFormation, in this case executing a PowerShell script
4. Upon completion of the script, the instance runs cfn:signal  to return status back to CloudFormation
5. Finally CloudFormation determines whether  operation successful  (if not, it will rollback and terminate the instance)

During development stage, things may not work all at once. In my experience, that turned out to be a good thing, since I learned a lot more from things not working.  It is often necessary to follow the execution sequence step by step, and pinpoint the deviation from desired outcome at each step.

Invoking PowerShell – an example

With various components at play, some symptoms may not be so obvious. It is critical to have an idea of where it fails, in order to look for clues and apply further investigation at the right places, thus pinpoint the root cause, apply fix, and progress incrementally.

Below shows an example of what happens when instance launches. The script fails to generate the expected outcome. Logging on to the instance to examine the cfn log file:

…

2014-04-12 06:26:04,578 [DEBUG] Running command 3-update

2014-04-12 06:26:04,578 [DEBUG] No test for command 3-update

…

Note the command attempts to run the script, however it did not show “succeeded” following execution. Although it did not provide further information, we can guess that something went wrong when command was invoked. To collect further evidence, have the script to write output to a log file. We saw nothing recorded, indicating the script did not run successfully. With that knowledge, we can also test run the script directly from a command line on the instance, and noticed the need to set execution policy. Running PowerShell script on bootstrapping requires the proper option setting, which can be defined via cfn:init command:

"command" : "powershell.exe -ExecutionPolicy Bypass -NoLogo -NonInteractive -NoProfile -File C:\\cfn\\yourscript.ps1"

Now recreating CloudFormation stack with the updated template, a fresh instance is started which complete the script successfully. We can then go on check the outcome of the script itself.

2014-04-12 15:36:25,568 [DEBUG] Running command 1-update

2014-04-12 15:36:25,568 [DEBUG] No test for command 1-update

2014-04-12 15:36:51,042 [INFO] Command 1-update succeeded

2014-04-12 15:36:51,042 [DEBUG] Command 1-update output:

2014-04-12 15:36:51,073 [INFO] Waiting 60 seconds for reboot

2014-04-12 15:37:51,102 [INFO] ConfigSets completed

2014-04-12 15:37:51,118 [DEBUG] Deleting Scheduled Task for cfn-init resume

2014-04-12 15:37:51,414 [DEBUG] Scheduled Task deleted

2014-04-12 15:37:51,664 [INFO] Starting new HTTP connection (1): 169.254.169.254

2014-04-12 15:37:51,680 [INFO] Starting new HTTPS connection (1): cloudformation-waitcondition-us-east-1.s3.amazonaws.com

I will summarize additional dev and troubleshooting lessons and share a complete working example in a follow up post.