Anatomy of a Search Engine

EveKnows.com is 100% Linux powered. The free (as in speech!) system has proved to be absolutely perfect for our needs. It’s fast, stable, and customizable–exactly what you look for in a platform for running fresh, cutting-edge applications such as EveKnows.

One of the harder tasks I’ve had is tuning disk access. The search engine is currently running on a Debian 4.0 system with SATA hard drives. The UNIX utility top reports 10-20% IO usage (which is a good indicator of disk access) almost all the time. When I turn on the Caroline search spider, that usage spikes to 50%. At the moment this isn’t really a big deal, but as the site’s popularity continues to grow, it will eventually become a bottleneck and severely limit performance.

Thus, I’ve been trying to learn about profiling disk access on Linux systems. Maybe I’ve just been looking in the wrong places, but I haven’t been able to find any tools which can show me which applications are causing the heavy IO load. Some digging revealed that dmesg can report individual IO calls when /proc/sys/vm/block_dump is set to 1, but that raw information is essentially useless. To that end, I wrote a small Perl script which totals all of the IO statistics and displays a pretty table of results. If anyone is interested in using it themselves, the code is below.

Update: HTML tends to screw up Perl code, so copying/pasting the below code probably won’t work; if you just want to download the script for your own use, you can find it here.

#!/usr/bin/perl
#
# Copyright 2007 Aidan Trent 
# Released under the terms of the GNU GPL                                                                                                                                                                             

# Usage: SCRIPT_NAME 
# The optional  parameter tells the script how many
# minutes it should spend gathering IO statistics. The
# default is 5.                                                                                                                                                                                                       

use strict;
use warnings;

my $sleep_time = 60 * 5; # 5 minutes
if ($ARGV[0]) {
    $sleep_time = 60 * int ($ARGV[0]);
}
`echo 1 > /proc/sys/vm/block_dump`;
sleep ($sleep_time); # 5 minutes
`echo 0 > /proc/sys/vm/block_dump`;

`dmesg > /tmp/io_stats.temp`;
open (FD, ‘/tmp/io_stats.temp’) or die;
my (%total, %read, %write, %dirtied);
while () {
    if (/(.*)\(\d+\):\s+(dirtied|READ|WRITE)/i) {
        my $name = $1;
        my $type = $2;
        print “$name - $2\n”;
        if (!$total{$name}) {
            $total{$name} = 0;
        }
        $total{$name}++;
        if (!$read{$name}) {
            $read{$name} = 0;
        }
        if ($type =~ /read/i) {
            $read{$name}++;
        }
        if (!$write{$name}) {
            $write{$name} = 0;
        }
        if ($type =~ /write/i) {
            $write{$name}++;
        }
        if (!$dirtied{$name}) {
            $dirtied{$name} = 0;
        }
        if ($type =~ /dirtied/i) {
            $dirtied{$name}++;
        }
    }
}
close (FD);

print “Name\t\tTotal\tRead\tWrite\tDirtied\n”;
foreach my $key (sort {$total{$b} <=> $total{$a}} keys %total) {
    my $tab = ”;
    if (length ($key) < 7) {
        $tab = "\t";
    }
    print "$key:$tab\t$total{$key}\t$read{$key}\t$write{$key}\t$dirtied{$key}\n";
}

unlink ('/tmp/io_stats.temp');