What fun Amazon EC2 is – Part 1: Instance Management

So as of late I have been doing a lot of programming to have a some what automated way to scale out indycar.com and indy500.com when the time calls for us to scale out. To scale out we are using Amazon EC2 with a great deal of success. Last year we successfully made it through a season with using EC2. The biggest problem we ran into was the time it took to scale out.

Scaling out the old way took almost an hour to do and was mostly a non-automated process. Now if you know me at all I automate any thing and everything that I can.

To have an efficent way to automate the instance management I setup a instance admin and support init.d script to help maintain the state of all the running instances.

To create the site I started with Zend Framework as the base for the site as it has everything that I need rolled in to one nice neat package. I created a module to handle the starting, stopping and listing of the instances. To make the start and stop work properly I had to have a bash script that would run on startup and shutdown to notify the instance admin the an instance was started or stopped.

#!/bin/sh
#
# chkconfig: 2345 99 00
# description: register the config with the ims home base servers

. /etc/rc.d/init.d/functions

# Source in our AWS Credentials
. /root/instance/bin/aws_credentials

# check for mysql to be installed
if [ ! -e /usr/bin/mysql ]; then
yum install -y mysql
fi

# Define some Variables

hostname=`hostname`
internal_ip=`curl -s http://169.254.169.254/latest/meta-data/local-ipv4`
get_address=`curl -s http://169.254.169.254/latest/meta-data/public-ipv4`
ip_convert=${get_address//./-}

my_meta=(
"instance-id"
"ami-id"
"local-hostname"
"local-ipv4"
"public-hostname"
"public-ipv4"
"instance-type"
"public-keys"
"placement/availability-zone"
)

for p in ${my_meta[*]}
do
data=( `/usr/bin/curl -s http://169.254.169.254/latest/meta-data/$p/` )
if [[ $data == *=* ]]; then
data=`echo $data | awk -F "=" '{print $2}'`
fi
sql_arg=( "${sql_arg[@]}" "$data" )
done

sql_conn="-u MYSQLUSER -pMYSQLPASS -h MYSQLSERVER"
assigned_ip=`mysql $sql_conn -B -N -e "select ext_address from cloud.instance where ext_host='$hostname';"`
eip_state=`mysql $sql_conn -B -N -e "select eip_state from cloud.instance where ext_host='$hostname';"`

start() {

if [ "$eip_state" != "1" ]; then

# we have an elastic ip so assign it
if [[ -n $assigned_ip && "$get_address" != "$assigned_ip" ]]; then
#we have an elastic ip
/root/instance/ec2/bin/ec2-associate-address --private-key $EC2_PRIVATE_KEY --cert $EC2_CERT $assigned_ip -i ${sql_arg[0]}
get_address=$assigned_ip;
fi

# send the data off to the instanceadmin to have it finish the process
URL_POST="inst=${sql_arg[0]}&image=${sql_arg[1]}&int_host=${sql_arg[2]}&keypair=${sql_arg[7]}&type=${sql_arg[6]}&zone=${sql_arg[8]}&ext_host=$hostname&ext_address=$get_address&int_address=$internal_ip"

curl -s -d $URL_POST http://INSTANCEADMIN/instance/start

touch /var/lock/subsys/ims_instance
exit 0
else
echo $hostname " has already been registred with home base"
exit 1
fi
}

stop() {
if [ "$eip_state" != "0" ]; then

# send the data off to the instanceadmin to have it finish the process
URL_POST="inst=&image=&int_host=&keypair=&type=&launched=&zone=&ext_host=$hostname&ext_address=$get_address&int_address="

curl -s -d $URL_POST http://INSTANCEADMIN/instance/stop

rm -f /var/lock/subsys/ims_instance
exit 0
else
echo $hostname " is already unregistred with home base"
exit 1
fi
}

restart() {
stop
start
}

case "$1" in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
*)
echo "Usage $0 { start | stop | restart }"
exit 1
;;
esac
exit 0

As you can see from the code it’s your basic init.d script that. Not this is meant to run on CentOS or RedHat EC2 instances. When the start is called is sends all the information about the instance to the instance admin where it stores it in the database and sends out an email. Stop does the same thing except it clears out the database (with exception of the hostname and the external ip, if it’s an elastic ip). Doing this makes it easier to see which instances are running because you can actually see a domain associated with an instance-id instead of just an instance-id.

That’s it for part 1. In Part 2, I will outline the processed I used to have an instance build it self.