For a long time I've wanted to learn Apple's Cocoa and Objective-C. I've also been very interested in the completely unrelated Amazon Simple Storage Service (S3). Last night I embarked upon a quest to teach myself whatever I could of those by building a rather simple, but cool Ruby on Rails application. What I should have been doing was reviewing a chapter from an upcoming Ruby on Rails book but this was an itch I had to scratch.

Instead of teaching myself Objective-C however I decided to use the wonderful RubyCocoa library. To get at the Amazon S3 service I used the also very cool library that is appropriately namespaced AWS::S3. Removing Objective-C from the equation both hindered and helped. It is a pretty tough language to read so there was a good deal of trial and error when trying to understand some Objective-C code and translate it to Ruby and RubyCocoa.

My inspiration for this tangent comes from Panic, or more specifically their Coda application. It's an awesome tool and I was really impressed by its ability to generate thumbnails of your website or project. Creating website thumbnails isn't new, Alexa at least has been doing it for years. The way Coda does it is inspiring however. They paint nice icons that look like sheets of paper curling up with a well defined shadow, giving it a believable 3D look. It's a small thing, and it's the small things that matter.

My goal isn't to reproduce Coda's thumbnails, I don't have the time or skills to do that yet. My goal is to simply create decent full or partial page screen captures of active websites. The image of the Coda web site here is a result of my efforts. Using a simple web app I enter a URL, the viewport size (default is 1024x768) and the resized version (default is 320x240). If you leave the height off it'll go full height and get entire web page without cropping. Likewise if you leave the resize height off it'll scale it in proportion to the width.

In the end I finally have a good understanding of S3 and I know that I have a long long way to go with Cocoa. Especially if I want to get into Core Image and Quartz.

Here are a few more examples:

Just a few questions that I can't seem to find any answers for.

1. Why do people still use CVS or Subversion when there are tools like Darcs, Mercurial, or Git?
2. Why are people still developing in and for Windows?
3. Why are people still declaring variable types ahead of time?
4. Why isn't Objective-C more popular?
5. Why did I learn about S-expressions so late in life?

It's been a rough day. I hate trying to figure out how to get a project set up in Visual Studio when all you have are source files.

I'm aggravated at the fact that Subversion is such a piece of crap, but to use anything better you really have to get away from a crappy Windows command prompt and into something more useful, like csh or bash. The droppings left by Subversion in .svn directories (oops, sorry, they are _svn in Windows, WTF?) are super-duper annoying.

I'm also tired of the excessive typing (pun intended) required to develop in C++ or C#. Why can't I just say "x = y" without declaring the type of x; if y is already known. Even worse, why do I have to write code like this (in C#):

XmlDocument x = new XmlDocument();
x.load("file.xml");

foreach (XmlElement e in x.SelectNodes("//foo[@title='bar']") 
{
    doSomethingUsefulWith(e);
}

When all I really need is this (in Ruby):

x = XmlDocument.new
x.load("file.xml");

x.each do |e| 
  doSoemthingUsefulWith(e) 
end

I know, silly arguments. Especially since C# 3.0 brings in Linq and var and a whole slew of other new features to make it more tolerable.

But you know what? You still have to create a bloody class with a public version of the static method main in order to execute. And you still have to wait for 3.0 to come out and stabilize. And then you have to pay for it if you want VS.NET. That's just not right. Yet people continue to pay for it. Suckers? Nah, they just wont listen.

I've done this at least a dozen times in VMware, OpenVZ, and Virtuozzo virtual servers. Every time I do it I start from scratch. It's silly. This time I'm writing down the steps so I don't forget.

Getting Started

Let's start with a Virtuozzo powered VPS running a clean installation of CentOS 4.4. We begin by removing a few things that we just don't need right now.

• Fetchmail
• SWSoft's Apache build
• SWSoft's Sendmail build
• The Pop3 server that we don't need

Remove what gets in the way

rpm -e fetchmail hspc-wwwroot
rpm -qa | grep httpd | xargs rpm -e
rpm -qa | grep sendmail | xargs rpm -e
rpm -qa|grep ^apr|xargs rpm -e
rpm -e popa3d
service xinetd restart

Install yum

In the virtual server I am on right now, yum is not installed. Odd. Let's install it:

rpm -i http://mirror.centos.org/centos/4.4/os/i386/CentOS/RPMS/yum-2.4.3-1.c4.noarch.rpm \
       http://mirror.centos.org/centos/4.4/os/i386/CentOS/RPMS/python-elementtree-1.2.6-4.2.1.i386.rpm \
       http://mirror.centos.org/centos/4.4/os/i386/CentOS/RPMS/python-sqlite-1.1.7-1.2.i386.rpm \
       http://mirror.centos.org/centos/4.4/os/i386/CentOS/RPMS/sqlite-3.3.3-1.2.i386.rpm \
       http://mirror.centos.org/centos/4.4/os/i386/CentOS/RPMS/sqlite-devel-3.3.3-1.2.i386.rpm \
       http://mirror.centos.org/centos/4.4/os/i386/CentOS/RPMS/python-urlgrabber-2.9.8-2.noarch.rpm

The CentOS-4 Testing repository

Now we want to add the c4-testing yum repository so we will install the latest Ruby rpm's. Or we can install from sources, but no point in doing that right now since c4-testing has Ruby 1.8.5.

cat << EOF > /etc/yum.repos.d/CentOS-Testing.repo
[c4-testing]
name=CentOS-4 Testing
baseurl=http://dev.centos.org/centos/\$releasever/testing/\$basearch/
enabled=1
gpgcheck=1
gpgkey=http://dev.centos.org/centos/RPM-GPG-KEY-CentOS-testing
EOF
yum -y update

Install Ruby, and Rails

As mentioned previously, you can install Ruby from sources or use the RPM provided in the CentOS repository. I use the CentOS built Ruby simply because it's already at the latest version (1.8.5 at this time) and it saves a little time.

Ruby

yum install -y ruby ruby-libs ruby-devel ruby-rdoc ruby-irb ruby-ri

Ruby Gems

wget http://rubyforge.org/frs/download.php/11289/rubygems-0.9.0.tgz
tar zxf rubygems-0.9.0.tgz
cd rubygems-0.9.0
ruby setup.rb

Rails, Mongrel, and Sqlite3-ruby

Rails, mongrel, and sqlite3-ruby:

gem install -y rails
gem install -y mongrel
gem install -y sqlite3-ruby

Apache and Subversion

And now we get to the tougher stuff. We need subversion if we plan to freeze_edge, which I always do during development. I also like to use Trac. We wont cover the installation of Trac today, but we'll make sure our environment is Trac friendly. More on that another time perhaps.

So let's get prepared for Subversion. We'll need the following items installed before we can build it the way we need it:

• Apache Portable Runtime (apr, apr-util
• The Apache Web Server (httpd)
• Neon
• SWIG

apr

Starting with apr and apr-util, we need to download, configure, build, and install from sources:

wget http://www.ip97.com/apache.org/apr/apr-1.2.8.tar.bz2
bzip2 -dc apr-1.2.8.tar.bz2 | tar xf -
cd apr-1.2.8
./configure && make && make install

apr-util

wget http://www.ip97.com/apache.org/apr/apr-util-1.2.8.tar.bz2
bzip2 -dc apr-util-1.2.8.tar.bz2 | tar xf -
cd apr-util-1.2.8
./configure --with-berkeley-db --with-apr=/usr/local/apr/bin/apr-1-config && make && make install

Apache Web Server

wget http://apache.downlod.in/httpd/httpd-2.2.3.tar.bz2
bzip2 -dc httpd-2.2.3.tar.bz2 | tar xf -
cd httpd-2.2.3
./configure -C \
            --prefix=/usr/local \
            --with-apr=/usr/local/apr/bin/apr-1-config \
            --with-apr-util=/usr/local/apr/bin/apu-1-config \
            --datadir=/var/www/home \
            --sysconfdir=/etc/httpd/conf \
            --enable-dav \
            --enable-suexec --with-suexec \
            --with-suexec-caller=apache \
            --with-suexec-docroot=/var/www \
            --with-suexec-logfile=/var/log/httpd/suexec.log \
            --with-suexec-bin=%{_sbindir}/suexec \
            --with-suexec-uidmin=500 --with-suexec-gidmin=500 \
            --with-devrandom \
            --enable-cache --enable-disk-cache --enable-mem-cache --enable-file-cache \
            --enable-ssl --with-ssl \
            --enable-deflate --enable-cgid \
            --enable-proxy --enable-proxy-connect \
            --enable-proxy-http --enable-proxy-ftp \
            --enable-proxy-balance && make && make install

Neon

wget http://www.webdav.org/neon/neon-0.25.5.tar.gz
tar zxf neon-0.25.5.tar.gz
cd neon-0.25.5
./configure --with-ssl=openssl && make && make install

SWIG

wget http://prdownloads.sourceforge.net/swig/swig-1.3.31.tar.gz
tar zxf swig-1.3.31.tar.gz
cd swig-1.3.31
./configure && make && make install

Subversion

wget http://subversion.tigris.org/downloads/subversion-1.4.2.tar.bz2
bzip2 -dc subversion-1.4.2.tar.bz2 | tar xf -
cd subversion-1.4.2
./configure --with-apxs=/usr/local/bin/apxs \
            --with-swig=/usr/local \
            --with-ssl \
            --with-neon=/usr/local \
            --with-apr=/usr/local/apr/bin \
            --with-apr-util=/usr/local/apr && make && make install

Good to go

At this point we're good to go. I usually install Vim7 right about now because I prefer to do all my coding in vi. CentOS 4 only has Vim6; Vim7 has newer rhtml and rb syntax files. Adding PostgreSQL or MySQL will complete your stack for a production environment. Sqlite3 is great for getting the ball rolling. Now it's time to be a Really Useful Engine and get working on your shiny new Rails app.

Good luck!

For the past 14 months I've been working for [Site5 Internet](http://engineering.site5.com). At least 90% of the work was [Ruby on Rails](http://www.rubyonrails.org). It has been the best experience in my fifteen years as a software engineer. It's rare to find a company like Site 5, composed of really smart people who don't have their brains locked away in a jar somewhere.

To everyone at Site5, thank you for the pleasant time. It's been fun, it's been rewarding, and I've learned a lot. I don't know that I'll ever get a grasp on my habit of over engineering things to death, but hopefully there will be more Adams and Davids to keep me in line wherever I go. Thanks guys and gals, it's really been a pleasure.

Today I accepted a new job at a local place here in Michigan. I was really surprised at how the job market treated me over the past three weeks. While I expected to find very little, I actually had to turn down two other offers. It was hard. I was reminded of an Iron Maiden song:

Can I play with madness?
The prophet looked at his crystal ball.
Can I play with madness?
There's no vision there at all.
Can I play with madness?
The prophet looked and he laughed at me. Haha!!
Can I play with madness?
He said "You're blind, too blind to see."

Last night I made the decision and I feel good about it. So that's a good sign. My madness has crept away, I'm no longer blind to the vision but instead quite focused.

I don't know when I start the new job yet. I should learn that pretty soon. It'll be interesting to be meddling in big datasets again. The job is about Data Warehousing and ETL tools. I like messing with gobs of data but I'll sure miss Ruby. I'll miss Rails. I'll have to force myself to work on something Railsy.

Maybe I'll finally sit down and teach myself Lisp. I can't believe after all these years I haven't made that move yet. I think it's emacs. I hate emacs. I've been a vi / Makefile programmer for as long as I can remember, every time I try to mess with emacs I lose my mind. Once again reducing myself into a state of madness.

Whatever I do, it'll be fun. There are at least twenty million ideas I have floating around in my head. Did I say twenty million? I meant fifty million. Something like that.

My hat's off to Matt, Rod, Adam, Todd, Kevin and the rest of the Site5 gang that I just can't list here without the risk of running out of disk. There are a lot of really good people over there and the future looks really bright for them. I plan to keep in touch as much as possible and beg forgiveness for my lack of performance over the last three weeks. I was busy playing with madness.

Every now and then you run into a problem that can be resolved in only one way.  A kludge.  A kludge is generally what happens when you are forced to work around problems you can’t control.  It happens a lot more frequently than it should, and it almost always comes back to bite you in the ass.

A few months ago was such a time.  In the paragraphs that follow I’ll present to you a kludge that was perfectly functional but gave me some serious headaches.  I am not entirely sure how my associates felt about this particular kludge but it was both discomforting and annoying.  It should also have never been an issue, but I’ll elaborate on that further down.

The Stage

Allow me a few more moments and then I’ll get to the point.

• The project: Flashback at Site5
• The platform: Ruby on Rails
• The back-end: MySQL, Subversion, Linux (CentOS 4)

In a nutshell, Flashback is a versioning system that automatically sweeps and stores changes for Site5’s customers web space.  The key function is to provide rapid undo capabilities for simple or massive changes allowing a customer to restore individual files or even their entire web space to any point in time, with just a few clicks.

The Problem

Flashback originally used Subversion as the underbelly for the version control system.  Subversion has a nasty habit of storing its meta-data in .svn directories that happen to be right there within the data being versioned.  For anyone who has tarred up their work area while using Subversion (or CVS for that matter) this is evident, and it generally isn’t annoying; though in our case it was a major road block.  You see, we can’t be polluting users home directories with a bunch of Subversion meta-data without their consent.  Even with their consent it would be inconsiderate at best, and destructive at worst.  We just couldn’t allow this behavior to persist.  Unfortunately Subversion doesn’t provide an alternative location for storing this meta-data and so our adventure begins with finding a work-around.

Phase 1 – Research

Many ideas were tossed around, from using a union file system to staging the directories prior to versioning them, to patching Subversion directly.  I am pretty sure each and every member of the Site5 Engineering Team had an idea or two, but I rather unfortunately forget who had what ideas so I am unable to provide credit where credit is due.  At one point I think we even discussed using something other than Subversion!

Phase 2 – Trial and Error

I worked diligently in an attempt to get a working UnionFS, hacked just right, so we could hide the .svn directories in a completely different place transparent to the customer.  I am pretty sure David Felstead started working on the idea of staging the data, and even had a good deal of success with that.

Phase 3 – Aha!

With almost any tricky problem comes a tricky solution.  This was no exception.  I had a bright idea and I got to work immediately.  I didn’t even bother to tell anyone about it until it was about 90% done, I knew it would work.  All good kludges work, no matter how ugly they are.  If it doesn’t work it isn’t a kludge, it’s a catastrophe.

The Kludge

So you want to know what it was do ya?  It was almost too simple.  You see, Linux (and UNIX in general) has this great facility in ld.so(8) that allows you to pre-load a set of dynamic libraries before the executable loads its own shared libs.  This is invoked by setting the LD_PRELOAD environment variable prior to execution.  So with a bit of strace(1) magic I set out to write my own shared library whose entire purpose was to intercept all file and directory related calls made during an invocation of Subversion (either by library call-out or svn executable) and rewrite the file paths, relocating every instance that contained  “/.svn/” to a different directory tree.

Here is a small sample of how the code looked for the fopen(3) intercept function:

001   /** 
002    * intercept fopen
003    */
004   FILE *fopen(const char *path, const char *mode)
005   {
006     static FILE *(*orig_func)();
007     if (!orig_func) {
008       orig_func = (FILE*(*)()) dlsym(RTLD_NEXT, "fopen");
009     }
010   
011     IF_DOT_SVN(path) {
012       char *new_name = adjusted_filename(path);
013       FILE *ret = orig_func(new_name, mode);
014       free(new_name);
015       return ret;
016     }
017   
018     return orig_func(path, mode);
019   }

This fopen function is loaded before the application loads its own libraries (including the standard C library).  Since this version of fopen is loaded first, it trumps any that is loaded later.  The call to dlsym on line 8 is how we find the original from the standard C library.  So now, any time the application makes an fopen function call, the path is rewritten to point to an internal directory tree, outside of the users web space prior to calling the true fopen function.  In all there were 24 different functions that had to be intercepted in order for Subversion to be completely covered.  Lucky for us Subversion delegates most of these tasks to the Apache Portable Runtime which is pretty easy to mine.  Using strace (or truss) is nice to detect system calls, but you still have to figure out where those calls originate.  GDB can be extremely useful in this case, I highly recommend it.

Post Mortem

Ultimately this strategy proved to work very well.  We were able to separate the Subversion meta-data from the users data.  This went into production, and there were no problems caused by pre-loading our custom “fix_dotsvn.so”.  It was easier than installing a UnionFS.  It was faster than staging all of the data.  But it did come with its own set of baggage.  We had to be absolutely certain that the kludge was in place or we could easily corrupt a repository beyond repair.  It’s a very easy mistake to make and not one that is easily fixed.  This is a high price to pay.

This is why this is a Load-Bearing Kludge.

Load-Bearing Kludge

There is no real definition that I am aware of.  The term actually comes from Eugene Szedenits, Jr., an individual I regard quite highly whom I worked with at Clareos previously to joining the Site5 Engineering Team. I will attempt to provide the definition as I see it, and with Gene's blessing:

Any kludge you cannot remove without causing the entire application to die a horrible death.

That about sums it up.

What now?

Our Load-Bearing Kludge no longer exists.  Thanks to David Felstead’s incredible work we were able to supplant Subversion with a versioning system that he wrote in pure Ruby.  For our application it blows the doors off of Subversion in both speed and reliability.  You have to read this entry in his blog to get the full picture.

At the beginning of this article I mentioned that I’d explain why I don’t think we should have ever been put in this place to begin with.  The explanation is simple:  Subversion should not be polluting your source code with meta-data directories.  There are better ways.  It isn’t likely going to change any time soon, and I’m not likely to start using any of the alternative systems out there because in reality Subversion is the best I’ve seen.  I just don’t like it when applications dirty up my source trees.

It was just a few months ago, when I read an article about Ruby on Rails that I realized my entire life was about to change.  How you ask?  It all happened pretty quickly, so listen closely…

It was a rainy spring day…  Okay, I don’t really remember the weather or even if it was spring yet, but that isn’t really important.  So there I was, busy reading through Slashdot (as I often do while running large builds) when I saw it; an article posted that referenced Ruby on Rails and Zope.  I had been actively pursuing knowledge in Zope and learning bits of Python here and there over the past two years.  Instead of reading the article, I just bounced right to Ruby on Rails to see what this thing was.

After that things are a bit hazy.  I remember seeing spots and swirls of red while my head spun to a stop.  Moments later I reoriented myself enough to just barely stop myself from falling out of my chair.  Too much drama?  Perhaps, but believe me when I say this isn’t far from the truth.  Now let me explain why I was so impressed so quickly, but first:

• It wasn’t the speed at which you could develop useful applications.  That is merely a nice bonus.
• It isn’t the fact that Ruby is the language of choice.  That is merely a nice bonus.
• It has nothing to do with how nice the markup system is.  That is merely a nice bonus.

The real reason:  It Makes Sense.

That’s right.  For the first time in my life I went through an entire web framework and didn’t ask myself “Why the hell did they do that?”  Instead I found myself saying “Why didn’t I think of that?”  These are the things that make it make so much sense for me:

• Test driven development
• Modern design principles
• No rediculous cosmic-class-heirarchy to memorize
• It just works

That’s it.  It makes perfect sense.  Nothing else that I have ever encountered has made so much sense.  That my friend is a moment of clarity that just cannot be ignored.

So here I am today.  A Sr. Software Engineer with Site5, one of the worlds leading web hosting organizations, ready to change the web hosting industry forever.  The Site5 engineering team is made up of some extremely bright people and there is no doubt in my mind that we’ll make your head spin in the coming months and years.

Now.  Let’s get on with changing the world.  I only plan to live for 430 years so time is gold.