imperialWicket

I will probably write about this in greater detail another time, but I finally updated my primary laptop from Fedora 14 to the latest CrunchBang Statler release.

My biggest frustrations had to do with driver issues, which is strange, as I have not experienced hardware compatibility issues in Linux since installing Dapper on an old Dell laptop. These issues were: broadcom wireless driver support (which is an issue with the 2.6.32 kernel release, not a distro-specific issue), and our title issue with system beeps when running with an Intel sound card.

If you are having issues with system beeps in CrunchBang (with a non-intel card), you should check framling's recommendations in the crunchbang forums. Those instructions removed some of my system beep issues, and in fact they work well for a most non-intel cards. You can also try nmk's simpler recommendation in that same thread, and it will likely resolve the issue for non-intel cards as well.

Still beeping at startup and shutdown after these configuration changes? My Dell (with the Intel sound card) was. Until I found these intel-specific notes in the Debian user forums. It seems that the intel drivers have a setting that overrides the pcspkr module, and the default is enabled/on. As enok notes in that thread:

# echo "options snd_hda_intel beep_mode=0" >> /etc/modprobe.d/alsa-base.conf
# echo "blacklist pcspkr" >> /etc/modprobe.d/blacklist.conf

Those two options are all you need on Intel cards. Note that if you attempted any of the techniques in the other posts, you likely already added the "blacklist pcspkr" line to your blacklist.conf file.

You will probably still get a system beep at the next shutdown, since the new beep_mode configuration won't be loaded. But the next time you boot the machine, you should enter a sweet, blissful, and beep-free environment.

Something of an aside: I am finally making some time for Erlang, and have a couple neat projects in the pipeline for which I think it will prove appropriate. I want to focus these projects around server APIs with thick client-side code. I am leaning toward Chicago Boss with Spine. Before I commit to that, I want to spend a little more time with Zotonic and I would like to get to know Nitrogen. On the front-end, Backbone is a possible alternative, and I may avoid the client-side focus altogether and use the templating available in the Erlang framework.

I am going to skip the instance launching steps; you can find those spattered about the web. I put a LAMP targeting AWS EC2 instance article together that would do nicely, and you don't need to run any of the installations, just launch the instance and make an ssh connection. You might want to run a yum update.

The Chicago Boss installation turned out to be relatively straight-forward, but like installing Erlang, there were a few time-consuming parts that I wanted to document. The basic steps are:

1. Install a reverse proxy server (Nginx)
2. Install Erlang
3. Install Chicago Boss
4. Create a test project in Chicago Boss

Install Nginx

If you are only testing Chicago Boss, and do not plan on immediate use in a production environment, there is no necessity for a proxy server. The proxy server just gives you easy access to Chicago Boss over port 80. If you skip that step, just be sure that you add port 8001 to your security group.

As long as you are ok with an older Nginx release (in this case, I am), you can install Nginx from the Amazon repositories.

sudo yum install nginx

The download, configure, make, make install procedure for a more recent Nginx version does not seem difficult, but I did not run through it and it is outside the scope of this effort. I will note that if you are using setting the IP explicitly in your Nginx configuration files, be sure to use the Private IP and NOT an Elastic IP. You will encounter bind errors like the following if you use an Elastic IP:

[emerg]: bind() to 11.22.33.44:80 failed (99: Cannot assign requested address)

As I said, just replace the Elastic IP with the Private IP, and Nginx will start properly.

You will want to configure Nginx to relay requests to port 8001, there are instructions available from the Chicago Boss wiki for this redirection. That's not a great production config, but it is more than adequate for getting started.

Install Erlang

Get the latest download path here, and these Erlang installation instructions still work. Blogs are handy sometimes.

Install Chicago Boss

Download the latest release, and make it.

wget https://github.com/downloads/evanmiller/ChicagoBoss/ChicagoBoss-0.7.2.tar.gz
tar xzf ChicagoBoss-0.7.2.tar.gz
cd ChicagoBoss-0.7.2
make

I wanted to install the admin interface, for which there are instructions in the cb_admin repo and also in the Chicago Boss GH wiki. However, I had issues getting either to work, so I will follow up with a cb_admin post.

Create a project with Chicago Boss

We are up and running now, and the next step is definitely to create a project and start pushing Chicago Boss to meet your needs. Rather than copying the existing docs or paraphrasing, I will redirect everyone to the Quickstart wiki page and the more verbose Tutorial pdf. I recommend both.

If you encounter issues or have questions, the mailing list seems to be the way to go.

I have to look this up everytime, and for some reason it is really difficult to find a good resource. So after wasting another hour of my life searching random forums, here is what worked for me.

Using an FTP connection (it worked for me through a local FTP client or through the godaddy Hosting Control Center -> Content -> FTP File Manager), edit the php5.ini in your html directory. The full path is something like '/home/content/12/34567890/html/php5.ini' with different numerical values. Open that file. GoDaddy creates it for you when the account is created, if php5.ini is missing, someone probably deleted it, and you just need to make a new one.

In the php5.ini file, add the line:

memory_limit = 128M

The default is 64MB, which is enough for a lot of sites. Usually this needs to increase for the purpose of using particular extensions/plugins/modules in a CMS. Often image-handling extensions strongly encourage or even require 96M or 128M. I have not tested to see what the max is, but 128M is a good first step when 64M is no longer cutting it.

Now add a php file to that same 'html' directory. We need to use this for executing phpinfo(), and validating the change. Create a 'temp.php' file and add the line:

<?php phpinfo(); ?>

If you load your new temp.php page in a browser (yourPrimaryDomain.com/temp.php), you should see the PHP version you are using and all the PHP info details. A couple of these are of primary interest:

• Loaded Configuration File
• memory_limit

Unless you got really lucky, "Loaded Configuration File" is probably "(none)" at this point. If so, then your memory_limit value is still "64M". The problem is that your server has a fastcgi process running, and hence the php5.ini contents were loaded before you made the change. In order to reload the new php5.ini file, you need to kill off the existent web processes. Go to your hosting control center, then Content -> System Processes, now use the 'End Web' button to kill the running processes. This does not impede your site loading, it just forces the server to spawn a new process for the next request.

If you refresh the phpinfo page, you should see updated values for Loaded Configuration File and memory_limit. The server should have loaded your php5.ini file, and the memory_limit now reflects the updated value.

A couple other notes:

• Be sure you edit the php5.ini file in your root 'html' directory. If you have subdomains or addon domains with their own directories, you still want to edit the file in your root 'html' directory.
• Lots of places have data about php.ini affecting PHP 4 and php5.ini affecting PHP 5. One would think that since GoDaddy is no longer offering PHP 4, you could use either php.ini or php5.ini. Nope. Stick to php5.ini.
• Don't try to use the php_value_memory_limit directive in your .htaccess file, this causes the server to throw errors.
• I see lots of accounts that indicate you must use GoDaddy's FTP File Manager to edit the php5.ini file in order for it to work properly. I can't confirm this, and I can only assume that this direction was coincidentally related to running processes for the users that reported the behavior.

Now delete the phpinfo() file; I called it 'temp.php'. You do not want to leave these laying about your server. While I'm offering suggestions, you should also consider transferring to another hosting provider.

One of my goals for 2012 is moving away from shared hosting and toward cloud and/or small VPS servers. For now, I am targeting AWS Micro instances for a lot of my efforts. Ultimately, I will probably move to Lighttpd or Nginx, but in the mean time I thought I would relay a configuration that is working well for me on low traffic sites.

For anyone seeing performance or stability issues on a micro instances (using LAMP) that get reasonable traffic, you are probably encountering something like:

• WSOD (White screen of death) - most often this is actually a out of memory error in PHP
• MySQL failing to start or getting killed - this is usually apache using all the memory and the kernel shutting down the MySQL server
• Actual output of Out Of Memory errors in PHP - really the same as above
• MySQL failing to start due to pthread create returned 11 - MySQL can't create a new thread, probably because apache is using too many.

The root cause of these problems is lack of memory. Before you shut down your micro instance in favor of a small instance, or go back to shared hosting, consider tuning apache down a few notches. The default apache configuration is actually quite inappropriate to a server with 1GB or less memory. The default changes a little from one Linux distribution to the next, but for reference I'll describe the Amazon Linux defaults.

Amazon Linux default apache config

This is all stored in the /etc/httpd/conf/httpd.conf file. There is a lot in this file, but most of what we want to discuss is near the beginning. Our biggest concerns have to do with KeepAlives and the prefork module. If you are using workers, you will also want to review the worker module configuration -- which is very analogous to the prefork module configuration.

Timeout

The default timeout is 60 (seconds). This is ridiculous. Have you ever waited 60 seconds for a site to load? I turn this down to the 20-30 second range.

KeepAlive

KeepAlive defaults to Off in Amazon Linux. Enabling KeepAlive can avoid generating/killing so many apache child threads, but if your users tend to only view a page or two, this makes no difference, because apache only persists the thread for a given connection. If you have high user engagement, you probably want KeepAlive on, but you should tune it down to keep your total apache thread count low. I have it enabled in my example config below, but for many of my sites, visitor engagement is relatively low, and I simply leave it disabled.

MaxKeepAliveRequests and KeepAliveTimeout

If you enabled KeepAlive on your low memory server, you should reduce the MaxKeepAliveRequests and KeepAliveTimeout. MaxKeepAliveRequests defaults to 100, which seems relatively low, but how many users do you actually have making 100 requests to your server per session? Related to this is the KeepAliveTimeout value, which defaults to 15 (seconds). In my mind, 15 seconds is longer than a short page view requires, and shorter than a long page view requires. That is, if someone hits my site and realizes the page is not what they want, they look at tags/categories/menu/whatever and decide to look at something else. The time required for this is usually more like 5-10 seconds. There is no reason to keep the thread alive for 15 seconds. The opposite end of this spectrum is a user who reviews an entire page, and then elects to review a previous or alternate article, in this scenario more than 15 seconds often elapses, and the user generates a new thread anyway. If you enabled KeepAlive, I recommend reducing KeepAliveTimeout to 5 or 10 seconds. MaxKeepAliveRequests probably won't be an issue, but 25 or 50 is likely to behave in the same way as 100, and under the low chance that something awkward is happening, the lower maximum will curtail the issue sooner.

Prefork Module

Prefork is where the real magic happens. This is where we can tell apache to only generate so many processes. The defaults here are high, limiting you to a max of 256 servers. Just to put this in perspective, let's say you get 10 concurrent requests for a php page, and that page happens to require around 64MB of RAM when requested (well within the default php memory_limit of 128MB on Amazon Linux). That's around 640MB of RAM usage on your 613MB micro instance. And that's with only 10 connections - apache is configured to allow 256! I scale these down a lot, usually sticking with 10-12 MaxClients. As we mentioned in our example, this is still a dangerous number because 10-12 concurrent connections would use all our memory. If you want to be really cautious, make sure that your max memory usage is less than 613MB (on an AWS micro instance - 512MB is common on VPS from other providers). Something like 64M php memory limit and 8 max clients keeps you under your limit with space to spare - this helps ensure that the kernel won't do something crazy like killing the MySQL process when your server is under load.

You can play with the MinSpareServers, MaxSpareServers, StartServers, and MaxRequestsPerChild; but within the context of less than 20 MaxClients, making changes like 3 StartServers vs 5 StartServers will have almost no visible impact.

Without further ado, here are some values to get you started. Tweak away, and have fun - welcome to httpd.conf (use apache2.conf if you're on debian).

/etc/httpd/conf/httpd.conf

Timeout 30
KeepAlive On
MaxKeepAliveRequests 50
KeepAliveTimeout 10


    StartServers          3
    MinSpareServers       2
    MaxSpareServers       5
    MaxClients            10
    MaxRequestsPerChild   1000


# if you are using workers, update the IfModule worker.c section similarly.

/etc/php.ini

memory_limit = 100M

I want open source days. Add those to the list: healthcare, dental, prescription, vision, vacation days, sick days, etc. and open source days.

Many devs are complaining about perks, or lack thereof. Alternatively, many employers are advertising ridiculous benefits. One thing some companies offer is x% of your time for R & D, personal projects, or open source. This is close to what I want, but it fills a different need. Anyway, the following would be quite appealing to me.

----------------------

Some Position

Description: Responsible for doing some stuff. Full stack, rockstar, ninja, ginsu, terminator-alpha-squadron-hacker extraordinaire geek, whatever other catchy new descriptors are 'in' right now. Oh, and the company is really awesome, so: open source, beer, caffeine, hardware choice, cow, some cash money, and this crazy-niche nerd-lore adult toy.

Benefits: Equity, healthcare, 401k, 5% required research and development time, flexible schedule, 3 weeks vacation, 10 annual sick days, 5 open source days.

----------------------
Disclosure: I probably wouldn't apply anywhere that uses ninja/rockstar-esque terminology.

Open source days?

I don't think I'm the only one who has occasional trouble with traction in the morning. Or on the occasional Wednesday. Or Monday. Or Friday. Or...

Sometimes I am just a little burned out. My candle is going at both ends, and occasionally I just need to revert to a vegetative state for a while to get those brain juices flowing again. This is what vacation and/or sick days are for (most people need to use these more). Other times I have the urge to be extremely productive, but I have no motivation to move forward on my current task. These are the times when I'd like to be able to take an 'open source day'. I might want to work on a personal project that I have open sourced, I might want to support an existing open source project, I might just want to take a few open source solutions for a test drive. All of these are productive in their own right, but an employer is perfectly justified when highlighting that these activites are not offering (or rarely offer) a short-term and noticeable boon to the current task at hand. Solution: use an open source day!

Alas, I don't have open source days. On my 'off-days', rather than simply diverting my productivity to some other project (where I might be at peak productivity), I dick around fixing bugs at a snail's pace, reviewing reddit and hacker news obsessively, deleting spam comments from my blog, or maybe going through the xkcd catalog (again).

Some of the employers or productivity gurus out there might be concerned about this tactic. You might suggest that these are the days when you should employ productivity technique x, or that you've found technique y in combination with list-type z to work under similar scenarios. I only remind you that I said 5 days in a year. I'm not talking about resolving each and every Tuesday-lull by donating 8 hours (or so) of time to a random project. But about once a quarter I think some dedicated and endorsed open source involvement would be rejuvinating and productive for all parties involved. The employer gains some open source credibility, and possibly some employee education both for the individual and others with 'what are you working on?' and water-cooler-type insight. The employee gains experience on something that may or may not be directly in line with their position (both of which are productive). The open source project gains whatever it gains - evangelism, bug fixes, code cleaning, documentation, user support, whatever. And all this is gained at the expense of 8 man hours.

It's worth noting that the 8 man hours you exchanged were probably more like 2 man hours, because I wasn't likely to accomplish much that day anyway. Not only that, but the next day's 8 hours are more like 12, because I'm likely to be good and recharged after such a welcomed mental diversion.

Some may also say that an open source day is a perfectly good use of R & D time, or that this is a justifiable usage for vacation/sick time. While I can get behind both of these ideas, I am talking about an offering that would appeal to me. If an employer or potential employer announced that I get 5 open source days to use at my own discretion (usage within reason, of course - no open source weeks immediately prior to a launch date), it would offer a notable plus for said employer. I see this as an employer going out of their way to say a few things:

1. Open source is important to us.
2. We want open source to be important to you.
3. We trust your judgment, if today would be more productive for you spent on open source project x, we support that decision.
4. We know that whole-heartedly diverting your attention can sometimes promote epiphanies, strokes of genius, or just light bulbs with resolutions to bugs that you'd otherwise waste days trying to fix.

I doubt we'll see this offering (or maybe I'm just looking in the wrong places?), but I think it would be grand to take a proper open source day.