imperialWicket

am i the only croquet-playing computer nerd?

Netflix Asgard Amazon Linux 201303 (Release 1.1.1.01)

2013-05-22

Netflix OSS: Asgard Amazon Linux (1.1.1.01)

US-East (VA): ami-17761e7e
US-West (OR): ami-1bfe6f2b

This is a pre-built, public AMI that provides Netflix OSS: Asgard on Amazon Linux. This AMI is free to use, and I will do my best to support and update it. For general Asgard/AWS questions, the Asgard email group or Asgard wiki are good starting points.

Getting Started

This AMI very closely matches the Ubuntu-flavored Netflix Asgard AMI that is available on Amazon Web Services. There is a lot of helpful information there regarding general connectivity and configuration options.

I will relay here that the default launch configuration automatically starts apache and tomcat, and requires access to port 443. The SSL certificate is generic (you'll need to confirm that it's ok), and the site is hidden behind basic authentication. The initial basic auth creds are stored in /etc/httpd/.htpasswd and are:

user: netflix
pass: netflix

SSH Connection

You can connect as follows:

ssh -i /path/to/aws/key.pem netflix@ec2-##-##-##-##.*.compute.amazonaws.com

Elasticsearch Logstash index backup, restore, rotate

2013-05-21

Here are a set of scripts for managing Logstash indices in elasticsearch. This set of scripts is inspired heavily by a previous collection. The general concept is that I do not want to keep all my logs in elasticsearch, when it is highly likely that I will only be searching for logs from the past couple of days. Nonetheless, I want to persist a much longer time period, and I want it to be easy to bring back a particular index or group of indices. With these scripts, I can. Backups and restores are handled independently, so if you merely want to get rid of an index after some period of time, you can just run the removal script. Backup/restore efforts default to using S3 with s3cmd, but you can hack around this if you prefer to maintain local-only (or scp, or whatever).

I tested these and they are working for me, but I'm not responsible if you delete your data.

My setup runs the backup script around 1 AM and backs up the index from yesterday. I run the remove script at 2 AM (plenty of time for a very nice tar command to complete in my case) and keep the most recent 14 indices around. I have a ch
atbot configured to restore based on date with the restore script. If something happens and I need to investigate logs from many weeks ago, I have the chatbot hit the elasticsearch node and run a restore for that date; these back-dated in
dices are automatically wiped the next morning when the remove script runs.

I'm happy to make changes if bugs are spotted, and if there is interest I can move these to a proper repo. Happy logstashing!


Netflix Asgard Ubuntu 12.04 LTS ami (Release 1.1.1.01)

2013-05-19

I am creating a number of Amazon Machine Images that include pre-configured Netflix open source software. Asgard is the first target, and this is the first release. I plan to support at least Ubuntu and Amazon Linux; ideally CentOS and Debian will also have prepared AMIs.

Netflix OSS: Asgard (Ubuntu 12.04LTS 1.1.1.01)

US-East (VA): ami-f7e28b9e
US-West (OR): ami-a5c25395

This is a pre-built, public AMI that provides Netflix OSS: Asgard. This AMI is free to use, and I will do my best to support and update it. For general Asgard/AWS questions, the Asgard email group or Asgard wiki are good starting points.

Getting Started

You need an AWS account; you can find a lot of helpful basics on getting started with AWS on this site and elsewhere. This runs on a micro instance, but not well. I strongly recommend a small instance or greater, depending on how much your managing within AWS. Configure your security group to allow access to port 443 (HTTPS), if you want to change any configuration or connect via SSH you will need access to port 22 (SSH).

By default, once this server is started you can access Asgard using the AWS public dns over HTTPS (and only HTTPS)\. Note that the AMI uses Ubuntu's default Apache certificate, you will need to acknowledge this certificate mismatch upon first visiting your AMI via web browser. If this is concerning (as it should be!), you should connect via SSH and either move to port 80 (default or no-auth) or add an appropriate certificate. If you update the hostname, you should add the desired ServerName to the appropriate Apache site configuration (though this is not strictly necessary).

The initial basic auth creds are stored in /etc/apache2/.htpasswd and are:

user: netflix
pass: netflix

SSH Connection

You can connect as follows:

ssh -i /path/to/aws/key.pem netflix@ec2-##-##-##-##.*.compute.amazonaws.com

The netflix user has passwordless sudo privileges.

Web Server config

This ami is configured to proxy via Apache 2.2 (80 or 443) to Apache Tomcat 7 (8080). As noted, the default site for Apache is SSL only, and has basic authentication enabled.

Clear the netflix basic authentication credentials:
> /etc/apache2/.htpasswd
Add basic authentication credentials:

** Keep a space at the beginning of this command **

 htpasswd -b /etc/apache2/.htpasswd username password

There are four pre-configured sites available for apache:

  • default-ssl - Enabled; SSL connection with basic authentication
  • no-auth-ssl - SSL connection with no authentication
  • default - HTTP connection with basic authentication
  • no-auth - HTTP connection with no authentication

Use a2ensite/a2dissite to enable/disable these server configurations.

Roadmap

The current plan is to create an initial Amazon Linux release, then finish full puppet modules/manifests for building the base AMI and the particular Netflix OSS AMIs. This allows community members a great deal of flexibility in building their own AMI and also in maintaining these releases.

This AMI uses OpenJDK, which is in line with my personal preferences. If there is great demand for Oracle Java 6/7, I'll add a script to switch JDKs.

I will have the initial puppet repos on github soon, please report any issues or feature requests via github or imperialwicket (at) gmail.


The Mob 0.8.01 release notes

2012-09-4

This is the initial release of The Mob, a pre-configured AMI with Chicago Boss, supporting databases, Nginx, a 'boss' user, and Erlang.

   __  __            __  ___      __   
  / /_/ /_  ___     /  |/  /___  / /_  
 / __/ __ \/ _ \   / /|_/ / __ \/ __ \ 
/ /_/ / / /  __/  / /  / / /_/ / /_/ / 
\__/_/ /_/\___/  /_/  /_/\____/_.___/  
                                      
         Version 0.8.01

Please check for specific version details on the main page for The Mob.


Chicago Boss: My CB deployment strategy

2012-07-12

Summary

This is the solution I am using for smaller services and sites where automatic scaling is likely overkill. You can easily enhance this to support a handful of servers, beyond which something like puppet/chef is probably in order.

At a glance:

  • EBS backed EC2 instance with key access for my user
  • Nginx - configured to serve static content, and reverse proxy to mochiweb for CB content
  • Separate user for Chicago Boss and the apps
  • Chicago Boss - cloned from Github for easy updates
  • CB Admin - cloned from Github for easy updates
  • My app - cloned from the server git repository, and updated with post-receive hook

AWS

Generally, I use a vanilla Amazon Linux server, and configure it with my own user for SSH connection via keys only. I update my user to have passwordless sudo permissions, and usually I disable the ec2-user entirely. You will need to disable 'Defaults requiretty' in /etc/sudoers for the git hook to work properly, you can just comment the line.

I explained basic SSH key authentication when describing more secure lamp on AWS. You can ignore pretty much everything but the terminal code for ssh keys and disabling password based authentication. There is no need to install MySQL, Apache, etc.; unless you plan to use MySQL as a database for Chicago Boss, but that's beyond the scope of this post.

I detailed a Chicago Boss installation on AWS EC2 and the only particular skipped in that article is that I installed all of these under a particular user, which I will call 'app' in this post. Nginx is detailed briefly in that post, and redirects to the helpful page from the CB wiki.

After spinning up your EC2 instance and creating your personal user, whom I will call 'me', make sure that you can connect via 'ssh 11.22.33.44' - where 11.22.33.44 is your Elastic IP (or public dns value, but you should always consider using an Elastic IP - it makes a lot of things easier).

Update the server, install git, and create a bare repo under your user's home directory.

sudo yum update
sudo yum install git
cd 
mkdir app.git
cd app.git
git init --bare

Users/Permissions

Chicago Boss does not require root permissions and we will install our application(s) under an app-specific user. You should have a 'me' user present, and that's who we'll always connect as.

Now we need to create an 'app' user, and add 'me' to the 'app' group. We'll also give the 'app' group write access to the 'app' user's home directory. Add the 'app' user to the 'me' group so that 'app' can clone from the git repo in /home/me/.

sudo su -
useradd app
usermod -Ag app me
usermod -Ag me app
chmod 750 /home/me
chmod 770 /home/app
exit

After pushing your CB project from a local machine to the bare repo we created on the server (this likely involves something like 'git remote add origin 11.22.33.44:/home/me/app.git' in your local repo, followed by 'git push'), you can clone your application, Chicago Boss, and cb_admin from Github and the repo on your server.

sudo su - app
cd /home/app/
git clone https://github.com/evanmiller/ChicagoBoss.git
git clone https://github.com/evanmiller/cb_admin.git
git clone ../me/app.git
exit

You'll need to execute 'make' in the ChicagoBoss directory, clean up configs, etc. All of these actions are detailed in the linked articles, and if you encounter any issues, add a comment and I'll do my best to update things or point you in the right direction.

Git post-recieve hook for Chicago Boss

This was a little tricky, primarily due to permissions/users/ownership that I wanted to enforce. Also, there is the fact that rebar really likes to run with the user in its directory. For example, as user 'app':

This doesn't quite work:

cd /home/app/
./app/rebar compile
./app/init.sh reload

But this does:

cd /home/app/app/
./rebar compile
./init.sh reload

I didn't spend a lot of time with this, and it may just be an update that's needed to rebar and init.sh, but for now, I just dealt with it in the post-receive script, which looks like this:

/home/me/app.git/hooks/post-receive:
#!/bin/sh
#
# Checkout updates in the CB app directory, compile, and  hot deploy.

# Set GIT_WORK_TREE and checkout
sudo -u app GIT_WORK_TREE=/home/app/app/ git checkout -f

# CB specific compile and reload
sudo su - app -c 'cd /home/app/app/; ./rebar compile'
sudo su - app -c 'cd /home/app/app/; ./init.sh reload'

This is overly complex, and it is most due to my desire to disallow direct login as my app user. A lot of the more intricate sudo behavior is solely to allow the command to be executed remotely as the 'me' user, and would be unnecessary if you simply connected to the server as the same user who will host your app. There are lots of other ways to handle it (using /var/www/html, or /opt, or whatever and handling permissions there, for example), but I think it's easy to understand what's happening at a glance with this config.

With that post-receive hook in place, you can push to your remote repository at 11.22.33.44:/home/me/app.git, and after each push, the checkout occurs in your /home/app/app directory, and git automatically issues './rebar compile' and './init.sh reload' as the appropriate user from the appropriate directory.

It's not the easiest setup to get running, but it is simple once you complete the config. A bonus is that if anything goes awry in a push, compile will likely fail, and the server will simply maintain the old compiled files, so end users see the same behavior that was live before your update - and you can take your time and address issues.

Happy deploying!