Anatomy of a Search Engine

Sorry for the recent slideshow outage. The bug has been fixed, so your XXX photo slideshows should be back in all their naked glory at EveKnows.com ;)

Today some very exciting features were added to everyone’s favorite porn search engine. I think the coolest by far is the new photo slideshow (more on that in a minute!), but here’s the full list so you can judge for yourself.

Quality of Results
The gallery ranking system was drastically overhauled and now offers significantly better results. A gallery’s date plays an important role in the new algorithm, so it should tend to keep fresh results near the top of the listings. It also helps to break up the tendency of ‘grouping’ search results, i.e. searching for ‘redhead’ would return lots of results from a single site, then a bunch of results from a different site, then results from a third, etc. Now the results will be mixed together much more appropriately.

More Thumbnails
New galleries will have three thumbnail images instead of only one, so you can get a better idea of whether the gallery contains the type of photo’s you’re interested in. Hover your mouse over the gallery and the thumbnail image will rotate, displaying each different photo for a few seconds. Please note that JavaScript must be enabled for this feature to work properly.

Photo Slideshow
New galleries now support a photographic slideshow of their images! Yes, that’s right: EveKnows automatically run through a complete slideshow of a gallery’s photos, all with a single click of your mouse! Not all galleries support this feature, and not all browsers are compatible with it. Notably, Safari will not be able to display most galleries as slideshows (although it does work with a few). Firefox, Internet Explorer, and Opera, however, all work properly. If a search result has an associated slideshow, you may view it by clicking the компютри втора употребаSlideshow link; galleries without slideshows will not have this link. If your web browser blocks pop-up windows, you will need to either press the ‘Play’ button to start the slideshow or turn of the pop-up blocker for EveKnows.com.

As always, feel free send me any comments or criticisms!

Today the EveKnows.com search suggestion module got an update. Previously it had been based on the words detected in galleries; while this seemed to make sense, it didn’t work out well in practice. I’m using aspell to handle the dictionary lookup, so words weren’t weighted based on their popularity. The new system still uses aspell, but it draws its dictionary from previous user queries. The basic premise here is that more people will search for the proper spelling of a porn star or phrase than for any particular misspelling, which will keep the desired results on top. If anyone notices any problems, please leave a comment explaining the issue.

Recently I made some adjustments to EveKnows, resulting in release 1.3. The changes have mostly been behind the scenes, but should lead to more accurate searches. If you notice any problems with the site, please leave a comment explaining the issue so that it can be resolved.

Also, I’ve added a Recent Searches feature, allowing you to view the 1,000 latest queries. They’re sorted chronologically, not by popularity.

Today I updated EveKnows to release 1.2. This update adds a Report Bad Gallery feature. If you find a gallery which is no longer available, inaccurately described, installs spyware, or has any other problems, please use this link to easily report the problem.

Other changes include some small user interface fixes. If you notice any problems with the site, please leave a comment explaining the issue so that it can be resolved.

EveKnows.com is 100% Linux powered. The free (as in speech!) system has proved to be absolutely perfect for our needs. It’s fast, stable, and customizable–exactly what you look for in a platform for running fresh, cutting-edge applications such as EveKnows.

One of the harder tasks I’ve had is tuning disk access. The search engine is currently running on a Debian 4.0 system with SATA hard drives. The UNIX utility top reports 10-20% IO usage (which is a good indicator of disk access) almost all the time. When I turn on the Caroline search spider, that usage spikes to 50%. At the moment this isn’t really a big deal, but as the site’s popularity continues to grow, it will eventually become a bottleneck and severely limit performance.

Thus, I’ve been trying to learn about profiling disk access on Linux systems. Maybe I’ve just been looking in the wrong places, but I haven’t been able to find any tools which can show me which applications are causing the heavy IO load. Some digging revealed that dmesg can report individual IO calls when /proc/sys/vm/block_dump is set to 1, but that raw information is essentially useless. To that end, I wrote a small Perl script which totals all of the IO statistics and displays a pretty table of results. If anyone is interested in using it themselves, the code is below.

Update: HTML tends to screw up Perl code, so copying/pasting the below code probably won’t work; if you just want to download the script for your own use, you can find it here.

#!/usr/bin/perl
#
# Copyright 2007 Aidan Trent 
# Released under the terms of the GNU GPL                                                                                                                                                                             

# Usage: SCRIPT_NAME 
# The optional  parameter tells the script how many
# minutes it should spend gathering IO statistics. The
# default is 5.                                                                                                                                                                                                       

use strict;
use warnings;

my $sleep_time = 60 * 5; # 5 minutes
if ($ARGV[0]) {
    $sleep_time = 60 * int ($ARGV[0]);
}
`echo 1 > /proc/sys/vm/block_dump`;
sleep ($sleep_time); # 5 minutes
`echo 0 > /proc/sys/vm/block_dump`;

`dmesg > /tmp/io_stats.temp`;
open (FD, ‘/tmp/io_stats.temp’) or die;
my (%total, %read, %write, %dirtied);
while () {
    if (/(.*)\(\d+\):\s+(dirtied|READ|WRITE)/i) {
        my $name = $1;
        my $type = $2;
        print “$name - $2\n”;
        if (!$total{$name}) {
            $total{$name} = 0;
        }
        $total{$name}++;
        if (!$read{$name}) {
            $read{$name} = 0;
        }
        if ($type =~ /read/i) {
            $read{$name}++;
        }
        if (!$write{$name}) {
            $write{$name} = 0;
        }
        if ($type =~ /write/i) {
            $write{$name}++;
        }
        if (!$dirtied{$name}) {
            $dirtied{$name} = 0;
        }
        if ($type =~ /dirtied/i) {
            $dirtied{$name}++;
        }
    }
}
close (FD);

print “Name\t\tTotal\tRead\tWrite\tDirtied\n”;
foreach my $key (sort {$total{$b} <=> $total{$a}} keys %total) {
    my $tab = ”;
    if (length ($key) < 7) {
        $tab = "\t";
    }
    print "$key:$tab\t$total{$key}\t$read{$key}\t$write{$key}\t$dirtied{$key}\n";
}

unlink ('/tmp/io_stats.temp');

Today EveKnows was upgraded to version 1.1. Changes include:

• Pop-up menus to allow more data to displayed for each search result
• Find Similar Sets command, which executes a search for similar galleries
• View More from this Site command, which displays every gallery in the EveKnows database from the same site
• Miscellaneous user interface fixes

The EveKnows.com database is now approaching 600,000 unique porn galleries. The vast majority of these have been pulled by our own web spider, Caroline. I’d like to invite every TGP/MGP owner to submit their own sites for indexing. Within one week your galleries will begin to show up in EveKnows.com’s search results, resulting in more traffic for your site. Caroline will also regularly re-index your site to pick up any new galleries you’ve posted.

Today marks the release of EveKnows.com 1.0, the Internet’s first Web 2.0 porn search engine! EveKnows is different than the other sex search engines–rather than searching adult sites in general, it focuses solely on free picture and movie galleries. We aim to revolutionize porn searches the way Google changed mainstream searching, with an emphasis on clean, quality results and absolutely no pay-for-placement galleries. This means you, the user, find exactly the porn you’re looking for, rather than the same, tired content from companies with large advertising budgets.

EveKnows.com utilizes the latest Web 2.0 technologies, making it a true joy to use. These include:

• Advanced Search Algorithm: Much like Google, EveKnows tracks which sites link to each gallery, giving preference to galleries displayed on multiple popular sites.
• Search Suggestions: Start typing a model’s name, and similar queries will automatically be displayed
• Custom RSS Feeds: Use your web browser’s RSS reader to watch feeds of your favorite models. The feeds are updated in real-time, so you’ll always see their latest published galleries.
• Popular Search Cloud: See what everyone else is searching for! The more popular a model’s name is, the darker the link will be.
• Automated Indexing: Our web robot, Caroline, is crawling the Internet for fresh porn 24 hours each day. Every clean gallery she finds is added to the site in real-time, resulting in an enormous database of the absolute newest porn anywhere!
• Standards-compliant Interface: EveKnows uses standard XHTML, CSS, and JavaScript, so it will work properly and look fantastic on a wide range of devices, from desktop PCs and Apple notebooks, to PSPs and iPhones.
• Open Development: The Anatomy of a Search Engine blog details the development of EveKnows.com, eliciting comments and suggestions for improvements from the site’s users.

More information can be found on the About Eve section of the site, along with advanced usage instructions. TGP owners and gallery submitters are welcome to request their own sites be indexed by our web robot Caroline.

So go ahead–try it out and see what you think!

One of the most frequent questions I’m asked is, “What sort of servers are required to run a porn search engine?”. Everyone seems a bit surprised by my answer: thus far, nothing special. From EveKnows.com’s testing launch in March until early July, in fact, the site ran on a single Athlon XP 2200 machine with 1GB of RAM and a standard IDE hard disk. Nothing special indeed. That configuration hasn’t had any trouble handling 60,000 page-views daily. Of course, I’m running EveKnows on a highly-tuned Linux server with custom-written software; everything has been optimized to make the most of the available resources, but the point still stands: it doesn’t take much hardware to handle the current traffic load.

Last week I finally upgraded to a dual-core Athlon 64 X2 4200 with a SATA hard drive. The new site design will go live sometime this week, marking the release of EveKnows.com 1.0. With luck, I’ll get a little bit of publicity and increase the daily traffic. The extra power is designed to insure against any possible spikes; I’d hate to get an influx of new visitors and watch the server melt under the load. At the moment, though, the load is sitting between 0.03 and 0.08. At least I can recompile kernels faster than ever before ;)

Also, I’ve added a Hardware category to this blog. I’ll keep everyone up to date on how the new server performs and how things scale as the site increases in popularity.