Like Spans, payloads involve the position of terms, but go one step further. Namely, a Payload in Apache Lucene is an arbitrary byte array stored at a specific position (i.e. a specific token/term) in the index. A payload can be used to store weights for specific terms or things like part of speech tags or other semantic information.

via Lucid Imagination » Getting Started with Payloads.

Tika’s interface is very similar to SAX, so it is easy to think about extraction in terms of receiving SAX events just like you do with XML, which is also nice because it is thus streaming and doesn’t require you to load a whole document into memory before dealing with it.

I’m now in the process of incorporating Tika into Solr (see SOLR-284) and I think we’ll eventually see it hooked into the Data Import Handler (DIH) in Solr, too, such that one could easily get content from a DB or a URL with the DIH, and then extract it if it is a binary object.

Ultimately, Tika is one more piece to the puzzle when it comes to dealing with content and it fits well with my personal vision (i.e. removing my Lucene PMC hat) of what Lucene is and should become.  Namely, as we move forward beyond just search (since search is a commodity these days, thanks to Lucene), it is important to have a whole suite of tools to bring to bear on the problem of dealing with structured and unstructured data.  Thus, things like Lucene, Solr, Carrot2, UIMA, Mahout, Tika, OpenNLP and other tools all should be easily usable by text tamers (riffing on my “Taming Text” theme…)  in creating intelligent applications.  As Lucene continues to develop and grow, it should become easier and easier to build things using the Lucene Stack which should spur a new wave of ideas and opportunities for those paying attention.

The basic concept is that term frequency is the number of times the term occurs in a document while the IDF is the inverse of the number of times the term in question occurs in all of the documents.  Thus, the more often a term appears in a document, the more important the document.  The IDF, then acts as a counterbalance to the term frequency by saying that the more documents the term appears in, the less important it is overall in determining the importantce of the term and the containing document.  Hence, I usually explain TF-IDF as the “Yin and Yang of Search”, and this seems to resonate well with my students, as it pretty clearly demonstrates how the opposing forces work to creating meaningful results for end users.  Of course, as sometimes happens with opposing forces,  one outweighs the other leading to bad results.

For more on the yin yang, see Yin and yang – Wikipedia, the free encyclopedia.

My latest article on Apache Solr, title “What’s New with Apache Solr” is now available over at IBM developerWorks.  It covers some of the new features like spell checking, Data Import Handler, distributed search, editorial results placement (a.k.a. “paid placement”), SolrJ and a variety of other pieces.

Hope it is helpful…  Feel free to give me any feedback.

This year, it is a 2 day training, but for those who want to, they can sign up for the first day of Lucene Boot Camp, and then attend Solr Boot Camp on the second day.  I am going to structure the class such that the first day is more high-level stuff (but still with some coding) while the second day will be much more hands-on.  Thus, for managers, CTOs, non-programmers, who just want to understand Lucene and Solr, they can easily plan on attending 1 day of each.

First off, is SOLR-651, which implements what I am calling a Term Vector Component. The basic gist of it is that Solr can now serve up term vectors from Lucene.  For those not initiated, term vectors store the term, term frequency and, optionally, position and offset information in a document-centric way in Lucene (as opposed to the inverted index storage used for searching.)  Term Vectors are often useful for doing things besides search like highlighting, machine learning, document-document similarity.  This component can provide:

1. Term
2. Term Frequency
3. Position (based on analysis)
4. Offset (character based)
5. IDF – Inverse Document Frequency

Combining all of these things, plus a couple of other features, I think, can really enable Solr to act as a more general Text server (which is what Taming Text is going to show.)  For instance, the Analysis Request Handler can act as a Document Analyzer server, and the Luke Request Handler can provide all kinds of corpus statistics.  And I haven’t even mentioned search, faceting and spell checking yet.  Nor have I mentioned the other thing I am working on:  adding search-result and document clustering to Solr.  This is taking place on SOLR-769.  The basic implementation I have now does search result clustering using the Carrot2 open source project.  After that, I plan on adding in Mahout for document based clustering.  I also know that Tom Morton, for Taming Text, has added in OpenNLP‘s Named Entity Recognition into Solr.  Some point in the near future, I’ll put up a link to that code.

Bottom line: Solr ain’t just for search anymore!

Lucene Boot Camp (http://www.lucenebootcamp.com) is scheduled this year for ApacheCon US on November 3 and 4th in New Orleans.  This year, I am doing a two day event, as I felt the one day event was just not enough time to get in all the goodness that is Lucene (not that two days is either, but…)  Additionally, by having two days, the class will have more time to explore and develop the examples with the class and more one on one time.  There is also a fair amount of new material in Lucene these days, so I am going to be updating the examples, etc. on the website and changing things up a bit.

This class has traditionally filled up pretty fast.  In Amsterdam this past April, there was even a waiting list, so I recommend signing up early.

Also, for those coming from the US, note that the class is scheduled over on Tuesday, Nov. 4, which is election day.  Not sure who decided that one, but, if you’re planning on voting and attending my class, make sure you get an absentee ballot.  Which reminds me…

The author of this nice plugin for WordPress contacted me today about his Lucene based WordPress plugin, so I thought I would give it a try, as I’m obviously a big fan of Lucene and also never much cared for MySql’s search (in)capabilities.

The plugin is easy enough to install, only thing that struck me as a little odd was the need to set 777 on the permsissions.  Presumably, this is so it can write the index, but perhaps it would be better to store the index outside of the plugin infrastructure.  Of course, I don’t know what’s involved with writing plugins, etc. so not sure if that makes sense or not.

Indexing and enabling search was a snap, and I do think the results are good, even if my sites don’t have a ton of posts.  Indexing on my “main” site (http://www.grantingersoll.com) took slightly longer than here, but I do have more posts and comments there.

Only minor suggestion I would have is that the default boosts for title and content aren’t all that great.  I think the title boost was 1.8 and the content boost was 1.3.   I changed mine to be title: 5 and content: 2.  The way boosting works at indexing time, it has only 8 bits of granularity, there isn’t too much difference between 1.8 and 1.3 and I tend to think title matches are much more important.  Thus, I made them greater.  Still, very cool that the author has hooked field boosting in to begin with.

Things I would love to see:

1. Highlighting
2. Spell checking

All in all, seems to be a great little plugin.  And now, I can “eat my own dogfood” too!

Jason Rutherglen has been up to some interesting things with Lucene lately concerning real time search.  This has always been one of those parts of Lucene that has been needed over time by some people, but has never reached the critical mass whereby someone tackles it.  Looks like Jason finally has, and I am glad to see it.  I have, on occassion, met with clients, or users who truly have needed it, but most have usually been able to live with a little bit of a delay and the associated workarounds.  Now, it seems, they won’t have to.

Jeff’s Search Engine Caffè
Copyright and distribution issues
Let’s say for a minute that a web search track is interesting. A major barrier to improvements in academic and open source web search is the lack of large-scale (hundreds of millions or even billions of pages) test collections that evolve over time. GOV2 is a static crawl of 25 million government documents and can therefore be distributed without too much hassle. Not to mention there is little to no spam. However, there’s a problem: commercial documents are copyrighted! Is it possible to create a large-scale test collection of web documents that can be shared freely? I don’t know the answer to that question. Could could that volume of data even be distributed?

Right, we are not going to get into the distribution/copyright game.  We are going to focus on using collections that are freely available.  Each user would just be told what to download.

For example, we could do something like:

Have the user download a static version of Wikipedia from a specific date, index them however they see fit, then run a set of queries we develop and then rate the top 10 or 20 and post their results, including their actual implementation, which is always lacking other than the usual hand waving of saying “we did stemming and relevance feedback”.  We have the advantage in that we can say EXACTLY what we did, no question on implementation, so, gasp, others can repeat the exact experiments, like any good scientist does, before going on to improve it.   Then, when the next person comes along, they do the same thing.  If they disagree about the judgments for the same run, we have a discussion and one person convinces the other and we move on.   Next, someone will come along with a scoring improvement and post those results, and now people will know the current “best” algorithm for this set of data.

Lather, rinse, repeat for other collections, developed over time.  Any engine can submit, anybody can participate.  Open source at it’s best!