"AII" does not find the blog comment I wrote containing that phrase. "Justin James" only finds one item, the T1000 review I wrote. "Justin +James" does not bring up anything I wrote at all. "J.Ja" does not bring up any results, despite it appearing in everything I ever wrote for this site except for articles. "Justin AND James" does not return results. ""Justin James"" brings up results, but none for me. The results it brings up all simply have the word "James". "Usability" does not bring up a single one of my blog posts or discussion items or blog comments, despite it being something that I am constantly writing about. Relevancy should more heavily weight the words in the query as the query words nearly always decrease in importance from right to left in the searcher's mind; instead, all words in the query are weighted equally. As another commenter already pointed out, you also need to be using a soundex algorithm and/or a spell checker to be catching mispellings, typos, and mistakes.
This search does not work right at all, where I sit. It actually pulls up less items than the current search system.
Tag search is worthless, for the following reasons:
* Tags are not standardized. I may use "software development" as a tag, the search may look for "programming."
* No automatic thesaurus for tagging, with a proper weighted graph of meaning. "Software development" would be just as important as "programming" if the user searches for "programming", but a more obscure synonym for "software development" would be less important. Without the thesaurus, and standardized tags, the system lacks all use if it prioritizes tags over all else.
* Limited tags. TechRepublic limits the length of the tag line, so I cannot even manually type in all of the applicable synonyms into the tags.
* Emphasis on filtering full text search by the frequency of occurance is bad. If I write an article about "Timothy Walters" and refers to him throughout the article as "Mr. Walters" with the exception of the initial paragrah saying "Timothy Walters," your new search system will not find my article about "Timothy Walters" because that name only appears once, despite my whole article being about him.
As such, the system MUST be using full text search, not as a failover, but it must be using full text search with tagging to add additional weight, provided the conditions I specify above are met (and that is just the tip of the iceberg!).
"Folksonomy" as an idea is sexy, hip, cool, and has the aura of "democracy." In reality, it is nearly useless in terms of helping users. There is plenty of research out there to support that statement.
I heartily recommend the following article about vector-based search systems before attempting to write any search engine: http://www.perl.com/pub/a/2003/02/19/engine.html I also highly recommend a THOROUGH reading of Jakob Neilsen's research into search habits (www.alertbox.com). He has put in countless hours getting detailed statistical models of how users actually use search. Search Engine Watch is another good resource as well.
Attempting to shoehorn quality search within the confines of standard SQL is nearly impossible. I do not recommend it in the slightest. I can almost imagine how the search SQL is written, too, because I wrote something along these lines a few months ago, but at least the data was not actual content, and therefore tricky to search, it was mostly numerical and address searching. Trying to use SQL for something like this is hopeless. You probably want to be using a functional programming language such as Lisp, Scheme, O'Caml, etc. to implement a search, or one that has a lot of functional programming elements in it like Perl or Ruby. Trying to write search in an object oriented language like Java is like trying to move a pile of sand with a pair of tweezers, when there is a shovel two feet away that you are not allowed to touch.
Also, the results returned are hard to read. Much better than giving the first few lines of an item is giving the word/phrases itself, surroudned by context, which lets the searcher more easily know if the content is really what they need. Sometimes we only need one sentence, or only remember one phrase, the phrase that we are looking for.
A number of years ago, I wrote a basic search system (for an ecommerce package I was sort of working on) that could at least parse out Boolean operators (both symbolic and using words). It is in Perl, but should translatable fairly easily into another language. I would be happy to send it to you guys to use.
J.Ja
