Grant's Grunts: Lucene Edition » term vectors

Some New Features in Solr

grant_ingersoll — Thu, 23 Oct 2008 12:41:08 +0000

I’ve had a chance recently to work on some things in Solr that I think that can, in the right circumstances, really enhance Solr.

First off, is SOLR-651, which implements what I am calling a Term Vector Component. The basic gist of it is that Solr can now serve up term vectors from Lucene. For those not initiated, term vectors store the term, term frequency and, optionally, position and offset information in a document-centric way in Lucene (as opposed to the inverted index storage used for searching.) Term Vectors are often useful for doing things besides search like highlighting, machine learning, document-document similarity. This component can provide:

Term
Term Frequency
Position (based on analysis)
Offset (character based)
IDF – Inverse Document Frequency

Combining all of these things, plus a couple of other features, I think, can really enable Solr to act as a more general Text server (which is what Taming Text is going to show.) For instance, the Analysis Request Handler can act as a Document Analyzer server, and the Luke Request Handler can provide all kinds of corpus statistics. And I haven’t even mentioned search, faceting and spell checking yet. Nor have I mentioned the other thing I am working on: adding search-result and document clustering to Solr. This is taking place on SOLR-769. The basic implementation I have now does search result clustering using the Carrot2 open source project. After that, I plan on adding in Mahout for document based clustering. I also know that Tom Morton, for Taming Text, has added in OpenNLP‘s Named Entity Recognition into Solr. Some point in the near future, I’ll put up a link to that code.

Bottom line: Solr ain’t just for search anymore!

Lucene goodness

grant_ingersoll — Sat, 03 Nov 2007 01:31:59 +0000

Lots of good things happening in Lucene land lately, all of which should benefit users with faster indexing and searching capabilities. Most notably, Lucene 2.3 (hopefully released this quarter) has some major changes in indexing memory management and performance. I have personally clocked indexing using release 2.2 at about 400 rec/s (single threaded, Mac Pro dual CPU/dual core, using the contrib/benchmark indexing.alg) to over 2,100 records/s on 2.3-dev (the latest trunk). It also features easier control of the indexing process by specifying how much memory to give it, instead of the confusing maxBufferedDocs factor.

Other work being undertaken should speed up reopening IndexReader’s. There also are a number of smaller changes including a faster StandardTokenizer (the tokenizer most people use) and faster term vector access.

Of course, with that comes more testing and a greater need to make sure the next release is rock solid and backwards compatible. So, if you are a Lucene user, I would encourage you to give trunk a try on some of your non-production indexes, etc. and help us test it out.

Advance Lucene slides from ApacheCon Europe 2007

grant_ingersoll — Mon, 07 May 2007 14:31:12 +0000

The latest version of my slides for “Advanced Lucene” are located at http://www.cnlp.org/presentations/present.asp?show=conference

Talk covered term vectors, using various query types and Lucene performance tips and tricks.

ApacheCon Europe “Advanced Lucene” slides

grant_ingersoll — Thu, 03 May 2007 13:18:29 +0000

My (slightly old) slides for ApacheCon Europe are now available in the conference proceedings available at http://eu.apachecon.com/downloads/materials.zip

I will post the latest version soon, but there is very little difference between this version and the latest.

Topics covered include Lucene performance, term vectors and query tips and tricks.

Feedback is always welcome