My latest article on Apache Solr, title “What’s New with Apache Solr” is now available over at IBM developerWorks.  It covers some of the new features like spell checking, Data Import Handler, distributed search, editorial results placement (a.k.a. “paid placement”), SolrJ and a variety of other pieces.

Hope it is helpful…  Feel free to give me any feedback.

First off, is SOLR-651, which implements what I am calling a Term Vector Component. The basic gist of it is that Solr can now serve up term vectors from Lucene.  For those not initiated, term vectors store the term, term frequency and, optionally, position and offset information in a document-centric way in Lucene (as opposed to the inverted index storage used for searching.)  Term Vectors are often useful for doing things besides search like highlighting, machine learning, document-document similarity.  This component can provide:

1. Term
2. Term Frequency
3. Position (based on analysis)
4. Offset (character based)
5. IDF – Inverse Document Frequency

Combining all of these things, plus a couple of other features, I think, can really enable Solr to act as a more general Text server (which is what Taming Text is going to show.)  For instance, the Analysis Request Handler can act as a Document Analyzer server, and the Luke Request Handler can provide all kinds of corpus statistics.  And I haven’t even mentioned search, faceting and spell checking yet.  Nor have I mentioned the other thing I am working on:  adding search-result and document clustering to Solr.  This is taking place on SOLR-769.  The basic implementation I have now does search result clustering using the Carrot2 open source project.  After that, I plan on adding in Mahout for document based clustering.  I also know that Tom Morton, for Taming Text, has added in OpenNLP‘s Named Entity Recognition into Solr.  Some point in the near future, I’ll put up a link to that code.

Bottom line: Solr ain’t just for search anymore!

The author of this nice plugin for WordPress contacted me today about his Lucene based WordPress plugin, so I thought I would give it a try, as I’m obviously a big fan of Lucene and also never much cared for MySql’s search (in)capabilities.

The plugin is easy enough to install, only thing that struck me as a little odd was the need to set 777 on the permsissions.  Presumably, this is so it can write the index, but perhaps it would be better to store the index outside of the plugin infrastructure.  Of course, I don’t know what’s involved with writing plugins, etc. so not sure if that makes sense or not.

Indexing and enabling search was a snap, and I do think the results are good, even if my sites don’t have a ton of posts.  Indexing on my “main” site (http://www.grantingersoll.com) took slightly longer than here, but I do have more posts and comments there.

Only minor suggestion I would have is that the default boosts for title and content aren’t all that great.  I think the title boost was 1.8 and the content boost was 1.3.   I changed mine to be title: 5 and content: 2.  The way boosting works at indexing time, it has only 8 bits of granularity, there isn’t too much difference between 1.8 and 1.3 and I tend to think title matches are much more important.  Thus, I made them greater.  Still, very cool that the author has hooked field boosting in to begin with.

Things I would love to see:

1. Highlighting
2. Spell checking

All in all, seems to be a great little plugin.  And now, I can “eat my own dogfood” too!

What this means, is that one can now get back search results for the given query, and get spelling suggestions at the same time, pretty much like Google’s “Did You Mean” functionality (but probably not the same quality, as they have a much bigger corpus and probably use user feedback as well.)

For details on how to use it, try out the Solr example in the source distribution and see the Wiki docs.

Also note, that it allows one to plug in their own spell checker (or a commercial one) or use the Lucene spell checker.