Monday, 7 April 2014

Bootstrap EC2 instances with userdata from the AWS Console

This is another in a series of videos that I am producing for my YouTube channel HowDoYouDoStuff.


In this video, I am going to show you how to bootstrap EC2 instances with userdata from the AWS Console.


You can read the transcript below.


HTML 5 Player



Transcript

Hi guys. I'm Phil Stirpe.

In this video, I am going to show you how easy it is to bootstrap EC2 instances with user-data from the AWS Console.

Bootstrapping is an important aspect of provisioning instances in the cloud.

If you want to implement a truly elastic set of assets allowing you to bring up or discard servers at a moment’s notice, you need to be able to configure an instance at start-up.

This might involve installing some software at initial boot or configuring the machine to behave in a desired manner.

To bootstrap an EC2 instance in Amazon web services, you need to pass it user data at launch time.

User data can be up to 16 kB of data and typically takes the form of a script.

When you launch an EC2 instance it will be based upon an Amazon Machine Image or AMI.

These are typically LINUX or Windows server images. Although you can create any custom type that you wish.

If you choose an off-the-shelf AMI created by Amazon, it will contain one of 2 services. Cloud-Init for LINUX or EC2-Config for Windows.

These services run during the initial boot and check for any user data that you may have passed to the instance.

If your user data can be parsed as a valid script then it will be executed. If not it is ignored.

This is an example of a script we might pass to a Linux machine. It is a bash script that we could pass as user data. It would execute as the server was loading.

This is an example power shell script that we might pass to Windows server to bootstrap that.

Let’s look at the mechanics for passing user data to a launching instance.

I’m going to launch a Windows server as I have a useful power shell script that I can use here.

Rather than launch a micro, I’m going to choose a general-purpose m1.medium as my instance type.

I am now going to click on the Configure Instance Details button. There are a number of options that I can select here but the only one that I want is user data which is buried down in Advanced Details.

Now it is in this box that I can enter my script. Be it a power shell script or a bash script. But rather than enter it directly, I could choose it from a file.

So let me just click this button and then select this file and click Open.

Now you can’t actually see the contents of the script here, so I’ll just open it up in Notepad++ for you to take a look at.

This script downloads a number of files from an S3 bucket including a Chef Solo installer, installers for the likes of Notepad++ and a cookbook.

Once it has downloaded these assets and installed Chef Solo, it leaves Chef Solo to perform the rest of the installation.

So I will now click Review and Launch. I am happy with all of the details so I can go ahead and click on Launch and select my key pair.

Now this will take a couple of minutes to boot and then allow the user data script chance to run, so I will wait for a couple of minutes before trying to connect to the server.

[15 minutes later….]

Okay I have left it 15 minutes or so to give Chef Solo time to install all of the assets. Furthermore, when launching Windows servers for the 1st time you need to wait several minutes in order for the administrator password to be generated.

So I click the Connect button and then attempt to retrieve password. That’s fine. If the administrator password hadn’t been generated yet, I would have received a warning telling me that the password wasn’t ready yet.

I now need to pick the key file that I used when launching the server. This will allow me to decrypt the administrator password.

I need this password in order to connect to the remote server. So I will copy the password and then download the remote desktop file which I can use to connect to the server.

If I click on the Remote Desktop file, a dialog appears prompting me for the Administrator password. So I will enter the password that I copied earlier.

When I connect to this remote machine, I am expecting to see all of the assets that have been downloaded from S3 and that the installers have executed.

If I open up Windows Explorer, you will see that the root of C: contains some non-standard folders. For example a folder named Chef and a folder named opscode. There are also a couple of files that were generated by my script.

We can see that my script downloaded a number of assets and placed them in a folder called Chef. It then installed Chef Solo.

As my script was running, it was writing to this bootstrap file.

So this proves that my power shell script passed as user data was parsed by EC2-Config and executed at the launch of the server.

There is one last thing I would like to show you. Perhaps you would like to bootstrap your server but power shell scripts or bash scripts are not appropriate.

Perhaps you would like to implement some kind of business logic. For example, when this server launches I wanted to act as a video encoder or perhaps even an image thumbnailer.

In this situation I simply need to pass the server a code such as the number 1 or 2. Perhaps I might pass the words “video encoder” or “image thumbnailer” instead?

Then within the instance I may have a service of my own running which could look for this URL: http://169.254.169.254/latest/user-data

This URL is consistent for all instances.

You could use this URL to access user data passed into the launching instance and then act accordingly.

There you have it.

In this video, I have shown you how easy it is to bootstrap EC2 instances with user-data from the AWS Console.

Thanks for watching and please feel free to comment on my blog (www.philipstirpe.com) and Facebook page (www.facebook.com/philip.stirpe.tutorials). Perhaps you could suggest more video topics? Most of all, don't forget to subscribe to keep up with my videos as I release them.

Bye for now.



Flash Player








See you soon

Phil Stirpé
"I don't do average!"






8 comments:

  1. Where is the userdata script? Can I download it from somewhere?

    ReplyDelete
  2. Ebrahim

    I have enhanced the script somewhat since I recorded that video.

    It is much more powerful.

    I have just uploaded a sanitised copy here:

    http://1drv.ms/1lQKvZE

    You simply need to put the required assets into a bucket and then update the script to use that bucket name,

    You will also need to create and reference an SQS queue (if you want it for logging). Remove this code if not.

    Finally, you will need to launch the EC2 instance with an IAM role that has permission to read from S3 (and write to SQS queue if reqd.

    Good luck!

    ReplyDelete
  3. Thank you very much Philip! very useful and informative video. I'm still new with AWS and still learning. I really like your implementation of bootstrapping chef. Thanks again!

    ReplyDelete
  4. many thanks

    What about solution to use chef ? for Debian instances ?

    Thanks again

    ReplyDelete
  5. Your good knowledge and kindness in playing with all the pieces were
    very useful. I don’t know what I would have done if I had not
    encountered such a step like this.


    AWS Training in Chennai


    AWS Training in Bangalore


    AWS Training in Bangalore

    ReplyDelete
  6. Appreciating the persistence you put into your blog and detailed information you provide.

    It’s great to come across a blog every once in a while that isn’t the same out of date rehashed material. Fantastic read.

    AWS Training in Chennai

    ReplyDelete
  7. Thank you for any other informative web site. The place else may just I get that type of info written in such an ideal way? I have a venture that I am simply now running on, and I have been at the glance out for such info.Bootstrap Studio 4.1.2

    ReplyDelete
  8. I believe there are many more pleasurable opportunities ahead for individuals that looked at your site.

    Amazon Web Services Training in Chennai



    ReplyDelete