UPDATE: I moved these scripts to the elasticsearch-logstash-index-mgmt repository.

Here are a set of scripts for managing Logstash indices in elasticsearch. This set of scripts is inspired heavily by a previous collection. The general concept is that I do not want to keep all my logs in elasticsearch, when it is highly likely that I will only be searching for logs from the past couple of days. Nonetheless, I want to persist a much longer time period, and I want it to be easy to bring back a particular index or group of indices. With these scripts, I can. Backups and restores are handled independently, so if you merely want to get rid of an index after some period of time, you can just run the removal script. Backup/restore efforts default to using S3 with s3cmd, but you can hack around this if you prefer to maintain local-only (or scp, or whatever).

…

*** UPDATE: Be sure to check for updated AMIs. Some updates include newer Asgard versions, others include optimizations and enhancements to the AMI itself. ***

I am creating a number of Amazon Machine Images that include pre-configured Netflix open source software. Asgard is the first target, and this is the first release. I plan to support at least Ubuntu and Amazon Linux; ideally CentOS and Debian will also have prepared AMIs.

Netflix OSS: Asgard (Ubuntu 12.04LTS 1.1.1.01)

US-East (VA): ami-f7e28b9e US-West (OR): ami-a5c25395

…

The Mob is an Amazon Machine Image that demos a group of Chicago Boss applications showcasing basic functionality and database connectivity to PostgreSQL, MySQL, and MongoDB.

You can launch The Mob with ami-ef47f186, it is publicly available.

The applications are available on Github:

• Chicago Boss MongoDB App
• Chicago Boss MySQL App
• Chicago Boss PostgreSQL App

The Mob is configured with MongoDB, MySQL, and PostgreSQL apps pulled from the Github repositories, so that you can easily update a running instance if the demo apps are updated. However, I recommend launching the instance, and using it as a virtual playground and experiment area in Chicago Boss, without any of the overhead of configuring an environment.

…

This is the initial release of The Mob, a pre-configured AMI with Chicago Boss, supporting databases, Nginx, a ‘boss’ user, and Erlang.

Please check for specific version details on the main page for The Mob.

…

Summary

This is the solution I am using for smaller services and sites where automatic scaling is likely overkill. You can easily enhance this to support a handful of servers, beyond which something like puppet/chef is probably in order.

At a glance:

• EBS backed EC2 instance with key access for my user
• Nginx - configured to serve static content, and reverse proxy to mochiweb for CB content
• Separate user for Chicago Boss and the apps
• Chicago Boss - cloned from Github for easy updates
• CB Admin - cloned from Github for easy updates
• My app - cloned from the server git repository, and updated with post-receive hook

AWS

Generally, I use a vanilla Amazon Linux server, and configure it with my own user for SSH connection via keys only. I update my user to have passwordless sudo permissions, and usually I disable the ec2-user entirely. You will need to disable ‘Defaults requiretty’ in /etc/sudoers for the git hook to work properly, you can just comment the line.

…