W00t! Finally finished the SQL optimizations I started working on a couple of days ago. Searching single terms is now twice as fast as it was; coupled with the previous change, this equates to a four-fold speed increase over the past week! EveKnows.com is now serving up the hot, naked chicks faster [...]
I added a new modifier to the searching algorithm for EveKnows.com — site:
The site: modifier allows you to restrict your porn search to a particular website. For instance, searching Ariel Rebel site:myarielgalleries.com will only return Ariel Rebel galleries hosted on http://myarielgalleries.com. It’s a nifty little tool for finding similar porn once you’ve found [...]
As the EveKnows.com database grows ever larger, search times have been going up. Yesterday I tried some more SQL optimizations to alleviate this, and cut most search times in half. The problem seems to be disk access with large result sets, especially broad, single-word queries such as teen, babe, sex, etc. Finding [...]
I’ve added a new site search feature to EveKnows.com. Now TGP owners can include a search box on their site and have the search results only include galleries their TGP links to. It’s a simple way to let your surfers search through your gallery database!
The service is completely free, just go to http://eveknows.com/about.html to get [...]
I just made an update to the search result weighting algorithm. To keep the newest galleries on top, sites which were first indexed within the past three days are now given some extra weight. I’m a little worried that the extra processing will slow down the searching algorithm too much, but we’ll try [...]
So with Caroline running full-bore, I noticed a frightening thing–the EveKnows.com server was bottle-necked by disk IO. Between the spider downloading thumbnails and inserting galleries and SQL server fetching them and the Apache server handing out web pages, my machine was slowing to a crawl. Since the search database is far too large [...]
With the new threaded model for Caroline, I started to notice memory usage getting out of control. Under the previous forking system, I could expect a typical run to go through 200 MB, but the new model was easily topping 1GB and then dying when the system ran out of physical memory. After [...]
Today I took a break from tweaking the engine database and gave the interface an overhaul. I know search engines these days are mostly simple affairs, but I figured a decent design couldn’t hurt :p
It’s all valid XHTML + CSS, and should degrade cleanly in older browsers. Let me know if anyone has [...]
Caroline is the name of the spider used to build the EveKnows.com database. Today I hacked together version 0.4, which moved to the threading model I mentioned earlier. Running multiple processes was a pain and relied on having a few different starting-points; now I can run the spider on a single URL and [...]
Wow. Never underestimate the power of a left join. After watching the gallery database grow to 60,000 galleries with 2.5 million indexed words, SQL query times were getting pretty bad. A search for ‘Sexy Teens’ took around 10 seconds; if it’s that bad at 60,000 galleries, imagine how bad it would be [...]