Drew, Tom and I are feverishly working away on finishing up Taming Text.  We are currently in the process of addressing the feedback we got from our final review and should have updates up soon.  I have also posted all of the book’s source code up on Github under the Taming Text user.  The source includes, amongst other things, a simple Question Answering system using Solr and OpenNLP, as well as analyzers for Lucene that use OpenNLP for sentence detection, part of speech tagging and Named Entity Recognition.  As with most books, these examples are meant to be just that, examples.

Going forward, there should be several new algorithms going in post 0.1, as Isabel Drost has added some code for Winnow and Perceptron implementations and I believe Karl Wettin has some work on hierarchical clustering in place.  I also believe Ted Dunning and Jeff Eastman are working on Dirichlet clustering.  Finally, Sean Owen is always rocking on Taste’s collaborative filtering capabilities, so there will no doubt be more goodness in that regard.  As for me, I’m working on integrating clustering (Carrot2 and Mahout) into Solr and will be writing a chapter on doing text clustering in Taming Text.

Tika’s interface is very similar to SAX, so it is easy to think about extraction in terms of receiving SAX events just like you do with XML, which is also nice because it is thus streaming and doesn’t require you to load a whole document into memory before dealing with it.

I’m now in the process of incorporating Tika into Solr (see SOLR-284) and I think we’ll eventually see it hooked into the Data Import Handler (DIH) in Solr, too, such that one could easily get content from a DB or a URL with the DIH, and then extract it if it is a binary object.

Ultimately, Tika is one more piece to the puzzle when it comes to dealing with content and it fits well with my personal vision (i.e. removing my Lucene PMC hat) of what Lucene is and should become.  Namely, as we move forward beyond just search (since search is a commodity these days, thanks to Lucene), it is important to have a whole suite of tools to bring to bear on the problem of dealing with structured and unstructured data.  Thus, things like Lucene, Solr, Carrot2, UIMA, Mahout, Tika, OpenNLP and other tools all should be easily usable by text tamers (riffing on my “Taming Text” theme…)  in creating intelligent applications.  As Lucene continues to develop and grow, it should become easier and easier to build things using the Lucene Stack which should spur a new wave of ideas and opportunities for those paying attention.

Had a lot of fun at my recent talk at the Charlotte JUG.  They’ve got a good core of people and there was a lot of good discussion about the topic. Even managed to give away some free eBooks of “Taming Text“.  Wish I would have had time for demos, but as it was there was a lot to cover.  At any rate, the link has a copy of the slides I presented.

I would also be remiss if I did not mention the food we had at at Cajun Queen before hand…  Could have stayed there longer and enjoyed the music and a few frothy brews…  Next time…

First off, is SOLR-651, which implements what I am calling a Term Vector Component. The basic gist of it is that Solr can now serve up term vectors from Lucene.  For those not initiated, term vectors store the term, term frequency and, optionally, position and offset information in a document-centric way in Lucene (as opposed to the inverted index storage used for searching.)  Term Vectors are often useful for doing things besides search like highlighting, machine learning, document-document similarity.  This component can provide:

1. Term
2. Term Frequency
3. Position (based on analysis)
4. Offset (character based)
5. IDF – Inverse Document Frequency

Combining all of these things, plus a couple of other features, I think, can really enable Solr to act as a more general Text server (which is what Taming Text is going to show.)  For instance, the Analysis Request Handler can act as a Document Analyzer server, and the Luke Request Handler can provide all kinds of corpus statistics.  And I haven’t even mentioned search, faceting and spell checking yet.  Nor have I mentioned the other thing I am working on:  adding search-result and document clustering to Solr.  This is taking place on SOLR-769.  The basic implementation I have now does search result clustering using the Carrot2 open source project.  After that, I plan on adding in Mahout for document based clustering.  I also know that Tom Morton, for Taming Text, has added in OpenNLP‘s Named Entity Recognition into Solr.  Some point in the near future, I’ll put up a link to that code.

Bottom line: Solr ain’t just for search anymore!

I will be speaking at the Charlotte Java Users Group on Oct. 15th, covering things like Lucene, Solr, OpenNLP and Mahout, amongst other things.  Basically, a high level talk on my book.

If you ever read papers in this field, you quickly notice they all have this nice theory and they make all these nice (grand?) claims, but at the end of the day, 99% of the interested population can’t reproduce them because they don’t have the software to do it.  As Dr. Pedersen also points out, often times the creator can’t even reproduce it.  Even worse, they do publish the software, but it is abandoned, or undocumented, and who knows what settings are required: the Professor has moved on (“I got a new grant!”), the Grad students have moved on (“I got my PHd.”) and most importantly, the funding has moved on (Gov’t Program Manager: “I had another successful project, time for a bigger budget!”).  Sorry for the cynicism…

In fact, I think one thing Mahout can really offer the likes of Researchers is that you can focus on the big ideas, and we’ll take care of making sure your prototype is scalable, documented and maintained!  Besides, wouldn’t you rather be a part of seeing people actually use your work instead of it just living on some piece of paper or locked away in your hard drive collecting cosmic dust?

Scary…  I guess it is real!