Search Engine Goliath, Meet Search Engine David
In making this post, I feel like some guy in 1996 going “there’s this little website called eBay, it’s like an auction but it’s all over the world at once, right now it’s just got a few old lunchboxes and some busted electronics, but I think it could be huge someday!”
First, some background. Nobody I know of is happy with the current search monopoly situation. You search one place because they have the best results, nobody else even comes close. But when those results suck, what can you do? Nothing.
This problem is particularly acute with adult blogs, which often perform poorly in the search engines despite having some of the most detailed textual discussions extant of many sexual issues. There’s presumably a complex of reasons for this, the biggest of which is probably spam. So many porn spammers attempt to game the search engines to promote their sex sites, that the search engine “immune systems” (filters and controls) are quick to kick in when adult search terms are present. There’s also some evidence (endlessly blogged about elsewhere) that the search engines, being corporate, are fairly hostile to sexual material, or at best indifferent to the quality of adult searches.
Anyway, what can be done? The paradigm of automatic crawling plus automated anti-spam filters yields a functional index, but in adult areas the subjective quality of the results often seems low. And, in my experience, the more I know about a search term, the worse the results look to me — there are too many “where is that site, it should be here?” and “that site is just a slick-looking front end for spammy scraped-and-morphed RSS” moments.
Enter Jimmy Wales, the guy (love him or hate him) who was instrumental in making Wikipedia what it is today. He’s been working for quite awhile on using wiki-style user interaction to create better search results than anything available today. It’s an insanely ambitious concept, because the easier you make it for folks to “improve” search results, the easier you make it for them to game them, spam them, and crap all over them in wild orgies of sheerest vandalism. Is it possible for the crowdsourced wiki magic to overwhelm the forces of spam, or at least to fight them to a useful draw? Right now, it seems unlikely. But everybody thought Wikipedia could never be useful, either. That turned out to be dead wrong; for all its manifest flaws, Wikipedia is insanely useful on many topics.
If — please join me in my pipe dream — if only the new Wikia Search (re-released today in open Alpha with, for the first time, useful user-editing features) could produce a user experience that’s competitive with the current search behemoth, wouldn’t that be awesome? It doesn’t have to win or be better — it just needs a fan base and an integer percentage of total search volume, enough to trigger some concern and competition from the corporate search providers. We all know that internet users search for adult stuff (including, but not limited to, porn) a whole lot. Right now, those search outcomes are poor, and nobody in corporate America seems much interested in improving them.
I am hoping that Wikia Search offers a way forward. Why not check it out, play around with the very intuitive tools for improving the results of the searches that you do, see if it isn’t fun to use and fun to improve? (I got sucked in on my first visit; before I knew it, I’d deleted tons of spammy results from several searches and fixed the ugly “snippets” for several favorite sites. I even added a few worthy sites that weren’t showing up. It’s addictively fun, and much easier than working on a Wikipedia article.)
We know the spammers — including the porn spammers — are going to be all over this if it gets any traction. In my (metaphorical) pipe dream, I’m imagining the non-spammy adult web people getting there first, to help build and defend useful search results for adult terms. Idealistic, I know, and pointless if this turns out to be a failed experiment. But imagine the fun if it succeeds!
Here’s the TechCrunch article where I learned of today’s relaunch, and there’s also a short video there explaining how to use the user-modification tools to improve the search results:
Today, Wikia Search is beginning to suck a lot less. It has only indexed 30 million Websites, but it is finally rolling out a set of editing features that lets searchers reorder, add, remove, rate, annotate, and comment on results. It also makes it easier for anyone to try to game the search results. Although, as with Wikipedia, an spammers can be banned by the community. We should see some fierce edit wars on this one.
Here’s Jimmy himself, in Forbes:
Participation in Wikia search has the same incentive as anything online–it’s something people enjoy doing. People edit wikis not because it’s a charity, but because they have common interests and because it’s fun. Also, we’re making the barriers to participation very, very low. If you search for something and find a result that’s not relevant, it’s gone with a single click, and you’ve made the search results a little better.
…
Right now search is a closed box, and there are some plausible reasons for that, like preventing people from gaming the algorithm for commercial gain and keeping out malicious players. But can we create something that’s as open and transparent as possible and publicly accountable? That’s what we’re shooting for.
Two years from now, people may point at this post and go “LOL, whut?” But I’m hoping, instead, that everybody goes “You mean, search used to be done by brainless robots trying to follow clever rules? Wow, that must have really sucked.”
Because, it does.
Shorter URL for sharing: https://www.erosblog.com/?p=2283
true that. brainless robots following clever rules as opposed to user interaction and selectivity…it’s a no contest. :)
yay for the web-head!
I dunno how well it’s going to work out. You have to keep in mind that just as Google has bots that comb the interwebs, so do the spam folks have bots that pollute them. Especially since it works via AJAX it would be trivial to write a bot that would just intercept search queries from somewhere (like, say, spyware for instance or even from publicly-available sources) then spam Wikia for those search queries with a spam site.
Google is definitely the search engine Goliath, but they’re still no match for a well organized militia of zombie computers infected by a clever virus, like the Storm botnet.
The cycle completes.
The first web search engine was populated by people, well, a person. Back in the old days (very early 90’s) all web sites world wide were collected in a text list to be shared around so people could find them. That became untenable.
Then followed the indexes. Yahoo was originally a list compiled by hand of as many sites as possible. There was some automation to find new sites, but these were checked before being added. Growth was such that that also became untenable.
Then the limited automation became the full automation of latter search engines. It implied a level of trust in what was published. That too also worked for a while, and still does to some extent. Google is a good example, but only one of many. There are some technical limitations due to the stupendous data size, that is one of the challenges involved.
This is just bringing it back a level to the by hand indexes of before. Cheap too, as this time the people compiling the site lists are unpaid. Being unpaid however will make it just as subject to gaming as the other techniques.
I find it funny that they’re using the old net-nanny crawler.
http://en.wikip...gine)
I’m not sure that the result are actually bad. I think the problem is that the larger culture doesn’t really have the words needed to accurately describe what we’re looking for most of the time.
To me, it’s the opposite of “Eskimo words for snow” problem. For example I may be looking for something like the “Strong And Intense” photo. What would I type in? I’ve always loved the “turnabout bondage” films and photo shoots but I’ll be danged if I can find a search term that’s even useful there and it’s not like the porn spammers are even touching the word “turnabout”. There’s a whole genre there with no word describing it. “Harmony Philosophy of Love Bondage” could well descibe an entire subculture but I’d have a devil of a time searching for adult content that matched that phase that came from anywhere other than Harmony.
And the words we do use are often subcultured and not always easy to find. Why do we use “Shibari” instead of “Kinbaku” to the extent that we do?
I really don’t think it’s the fault of the search engines. We simply don’t have the words to search for yet and the words we do have still haven’t crossed from subculture to mainstream.
A picture may be worth a thousand words, but we don’t have the thousand words yet, just a heck of a lot of pictures.
I wish I had run across this yesterday. As a (very) recent blogger in the adult catagory, it took me a while to find live blogs through Google’s Blog beta. Since then I have ceased searching entirely and only use blog roll after blog roll searching for those still active and quite interesting blogs. But searching from a computer-indexed list rather than something intelligent leads to some not-so-intelligent results.
I’m not so sure I understand Wikia, but hopefully the concept will work out. I loved wikipedia, and often edited and started articles that went along with my expertise, and dearly enjoyed finding other people editing and adding to what I had written. Hopefully this will turn out well.