Grant Ingersoll Talks with Peter Keegan | Enterprise Search support for Apache Lucene and Solr by Lucid Imagination.

Peter has been a long time contributor to Lucene and is doing some cool things with Lucene’s payload functionality as well as many other Lucene features.

I will be speaking November 19, 2007 at the Triangle Java Users Group on Lucene and Solr.   The talk will be an introduction to the features and capabilities of both Lucene and Solr, as well as some basic compare and contrast information.

Talk covered term vectors, using various query types and Lucene performance tips and tricks.

I will post the latest version soon, but there is very little difference between this version and the latest.

Topics covered include Lucene performance, term vectors and query tips and tricks.

Feedback is always welcome

So, you might be asking yourself, what exactly are payloads good for?  Naturally, the answer is a lot!  For example, in the “Anatomy of a Search Engine” by Brin and Page (discussed here) see section 4.2.5 on Hit Lists where they discuss how they store information about the term in the index, such as font, capitalization, etc. which then get factored into the scoring algorithm later.  You could also store things like part of speech, or per term weights and then do things like score noun matches higher than verbs or use the encoded weight as part of the score calculation.  Another option would be to store synonyms of the words, or the synset from Wordnet.  In NLP applications, it could be useful for storing co-references or other types of linkage and then use a graph ranking strategy such as PageRank or TextRank (discussed here.)  Another option is to store XPath information or other metadata which are often stored in separate fields and require stitching the information back together.

In a sense, payloads open up a lot of new avenues for search in Lucene.  Open questions remain as to how much data should be stored and still have good performance.  Also note, that the current API is still considered experimental and may change, although I doubt it will drastically change.

What ideas do you have for payloads that I missed?  Let me know or update the planning page on the Lucene wiki.