This post is a reminder to scrub your images before making child images. It is not sufficient to only think about doing this when making public images. Today I ran into a situation where my child image did not properly boot due to stale data which was injected into the parent image at boot via cloud-init.
Configuring The Parent Instance For Child Image Creation
My goal was to launch a base VM, install custom software into it, and then create a child image from that VM. I could then launch that child image and know the software I needed would be installed and running in it.
The easiest way for me to do this was to write a small bash script that installed the software and started the services which I needed. I then launched a new VM with that script as user data. From there cloud-init ran it and my software was installed. Then I created a snapshot of that image for later use. It was a good plan…
To make a long a story short, I was using Ubuntu 10.04 which has cloud-init 0.5.10. In its default configuration there the user data script was stored to the image in such a way that it would be automatically run every time that VM was booted. Thus every time my child image was booted, the bash script I wrote to configure it strictly for its initial creation was run. Sadly my script was not idempotent and when run a second time it removed software and configuration files but not until the services I wanted to run started. Therefore I received some initial false positives that the run was successful only to later have everything fail. This was all due to the fact that I did not properly sanitize my image before saving it. Don’t live like me!
When making a snapshot of an image that was launched with user data and cloud-init please clean up the cloud-init logs and user data cache. I did not and I lost almost an entire day trying to debug my image!