I do have to say I learned a lot while doing the inital work and I’m happy that Wil pinged me to do it. If you want to check out the code currently it lives on github but it is on track to make the 1.8 release of Zend Framework.

Please leave a commet here or on the proposal for anything that I may have over looked or if you have any ideas on how I could make it better.

Splitting The Archive In To Pieces.

tar czPf - /dir_to_tar/ | split -b 200m -d - test_backup.tgz.

The code is very simple. First well tell tar to create (c), gzip (g), absolute names (P) and force (f) and the path that we want in the archive. It then gets piped to the split command where split is told to make no larger than 200mb files (b) and to put numeric suffixes on the files (d). Since we don’t have an input file the – represents the stream in from the tar command and we want it to output to test_backup.tgz. as the 00, 01, 02…. will be added to the end of that file.

After that is done running we just put the files on s3 using the s3Sync utility.

Combining The Pieces In To One Archive:

What happens when you want to extract the archive that you split.  You can just extract each file like you normally do.  This is how I do it but I’m always open for better ideas.

cat test_backup.tgz.* > test_backup.tgz

I must state that I’ve never had a split archive over 09 so i’m not sure how it will act when when you get more than that but like I said before if you have any better methods I’m open to new and better ways.

Scaling out the old way took almost an hour to do and was mostly a non-automated process. Now if you know me at all I automate any thing and everything that I can.

To have an efficent way to automate the instance management I setup a instance admin and support init.d script to help maintain the state of all the running instances.

To create the site I started with Zend Framework as the base for the site as it has everything that I need rolled in to one nice neat package. I created a module to handle the starting, stopping and listing of the instances. To make the start and stop work properly I had to have a bash script that would run on startup and shutdown to notify the instance admin the an instance was started or stopped.

#!/bin/sh
#
# chkconfig: 2345 99 00
# description: register the config with the ims home base servers

. /etc/rc.d/init.d/functions

# Source in our AWS Credentials
. /root/instance/bin/aws_credentials

# check for mysql to be installed
if [ ! -e /usr/bin/mysql ]; then
yum install -y mysql
fi

# Define some Variables

hostname=`hostname`
internal_ip=`curl -s http://169.254.169.254/latest/meta-data/local-ipv4`
get_address=`curl -s http://169.254.169.254/latest/meta-data/public-ipv4`
ip_convert=${get_address//./-}

my_meta=(
"instance-id"
"ami-id"
"local-hostname"
"local-ipv4"
"public-hostname"
"public-ipv4"
"instance-type"
"public-keys"
"placement/availability-zone"
)

for p in ${my_meta[*]}
do
data=( `/usr/bin/curl -s http://169.254.169.254/latest/meta-data/$p/` )
if [[ $data == *=* ]]; then
data=`echo $data | awk -F "=" '{print $2}'`
fi
sql_arg=( "${sql_arg[@]}" "$data" )
done

sql_conn="-u MYSQLUSER -pMYSQLPASS -h MYSQLSERVER"
assigned_ip=`mysql $sql_conn -B -N -e "select ext_address from cloud.instance where ext_host='$hostname';"`
eip_state=`mysql $sql_conn -B -N -e "select eip_state from cloud.instance where ext_host='$hostname';"`

start() {

if [ "$eip_state" != "1" ]; then

# we have an elastic ip so assign it
if [[ -n $assigned_ip && "$get_address" != "$assigned_ip" ]]; then
#we have an elastic ip
/root/instance/ec2/bin/ec2-associate-address --private-key $EC2_PRIVATE_KEY --cert $EC2_CERT $assigned_ip -i ${sql_arg[0]}
get_address=$assigned_ip;
fi

# send the data off to the instanceadmin to have it finish the process
URL_POST="inst=${sql_arg[0]}&image=${sql_arg[1]}&int_host=${sql_arg[2]}&keypair=${sql_arg[7]}&type=${sql_arg[6]}&zone=${sql_arg[8]}&ext_host=$hostname&ext_address=$get_address&int_address=$internal_ip"

curl -s -d $URL_POST http://INSTANCEADMIN/instance/start

touch /var/lock/subsys/ims_instance
exit 0
else
echo $hostname " has already been registred with home base"
exit 1
fi
}

stop() {
if [ "$eip_state" != "0" ]; then

# send the data off to the instanceadmin to have it finish the process
URL_POST="inst=&image=&int_host=&keypair=&type=&launched=&zone=&ext_host=$hostname&ext_address=$get_address&int_address="

curl -s -d $URL_POST http://INSTANCEADMIN/instance/stop

rm -f /var/lock/subsys/ims_instance
exit 0
else
echo $hostname " is already unregistred with home base"
exit 1
fi
}

restart() {
stop
start
}

case "$1" in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
*)
echo "Usage $0 { start | stop | restart }"
exit 1
;;
esac
exit 0

As you can see from the code it’s your basic init.d script that. Not this is meant to run on CentOS or RedHat EC2 instances. When the start is called is sends all the information about the instance to the instance admin where it stores it in the database and sends out an email. Stop does the same thing except it clears out the database (with exception of the hostname and the external ip, if it’s an elastic ip). Doing this makes it easier to see which instances are running because you can actually see a domain associated with an instance-id instead of just an instance-id.

That’s it for part 1. In Part 2, I will outline the processed I used to have an instance build it self.