Hope to see you in Boston!

That saying often comes back to me often these days when I hear again and again why some programmer chose to do something a certain way in Apache Lucene (and to some extent Solr, but less so since it takes care of most of the details of Lucene) , despite all documentation and community saying don’t do it that way.  The usual case for this involves paging deeper into the results, but I’ve seen it in many other areas as well, such as:

1. Loading everything into RAMDirectory instead of just relying on O/S caching because “it’s faster”, even though they don’t quantify it
2. Faceting implementations
3. Overriding defaults without testing
4. Blindly using defaults without testing (stemming in particular!)
5. Using a very large JVM Heap, thus choking off memory for the O/S, because “more memory is better”

In the paging case, the programmer thinks it is too expensive to execute the query a second or third time, so they go and retrieve 10 or 20 pages worth of results, if not much, much more, stuff them in a cache and then return the top ten.  There are many problems with this, the first being that most people don’t go beyond page one or two so all that work is wasted anyway.  The second is simply that Lucene is super fast at executing the search, never mind the fact that the Operating System probably cached everything it needs to do the search anyway and by you caching all that info you may have forced some of those O/S caches out of memory, thus slowing down subsequent searches.  Third, it creates excessive garbage which can lead to major collections, thus grinding the app to a halt.  Fourth, materializing the actual documents from disk for the search results is often expensive because doing so usually involves random seeks on disk.  In this case, the developer “assumes” Lucene would be slow at something, but didn’t bother to actually measure it to really know.  Often times what comes out of all this “premature optimization”, is a whole slew of code that now needs to be maintained, thus further complicating the application and making it harder for new developers to participate and make the application better, all while costing the company time and money.  Thus, assumptions in Lucene and Solr (and elsewhere) are Considered Harmful.  See Lucene’s Best Practices page and Solr’s Performance Factors, amongst other resources like “Lucene in Action” (2nd edition), for more info on how to do it right (or to challenge our assumptions!)

This isn’t just a Lucene/Solr phenomenon, and of course, I am not without sin, as I often catch myself or get caught by others, too.  Writing this is as much a reminder to me to be pragmatic and to test my assumptions as it is to anyone else.  Of course, one of the best things about a community like Lucene and Solr is having people who can objectively challenge your assumptions as opposed to more traditional development models where it is often the case that the most experienced or most senior developers rule the roost and developers are shackled by the “one right way” to do things.

Lucid Imagination » Understanding Lucene Performance – Free online workshop.

Daniel Tunkelang has written up an interesting post on the new Open Relevance Project that me and a few other Lucene people are starting up and I thought I would respond here:

Little late to the conversation, but I think maybe we should back up a little bit.   I like a lot of the comments and wish they were actually made on general@lucene.apache.org where we are discussing the merits of the undertaking (see http://www.lucidimagination.com/search/document/76d7cdeed4882397)  not that I expect that to happen given the way blogs work. At any rate, I’d like to add my two cents as the one who started the thread on general@lucene.apache.org.

First off, the ORP is VERY early stage brainstorming.  ORP really doesn’t warrant much attention at this point and it is premature to even speculate about how it relates to TREC, Google, Yahoo!, Microsoft or anything else.   I’m not even sure it has enough support to be a viable Lucene subproject! For now, I think most of us who are actually working on the genesis of the project are merely looking for a means to improve Lucene (and also Solr, Nutch and Mahout), despite what Otis says in his blog post about having grander notions for comparing across engines.

So, to the background…

This (ORP) is something I’ve been thinking about for a long time now and have discussed with a number of people in the past.    The motivation comes from my frustration over the years in not being able to obtain data that everyone on Lucene can use without limitations, since I’ve almost always worked in places that had little money to spend on this kind of thing.
See http://www.lucidimagination.com/search/document/656d5ca50c8c9242, http://lucene.grantingersoll.com/category/trec/ and http://lucene.grantingersoll.com/2008/09/18/opening-up-academic-research-on-ir-and-machine-learning/ for background.  The second motivation is simply to have practical, real world data driven by actual users.

In the past, I have talked with both NIST and Sheffield to try to work out terms by which the Lucene community could obtain TREC resources, but the licensing terms simply prevent a totally free redistribution.  (BTW, this is not NIST/Sheffield’s fault, but the company that allows them to use the data.  NIST/Sheffield are doing the best they can given their constraints.)  I have also talked with a few commercial companies that redistribute data (blogs, etc.) all to no avail (it’s usually the copyright that kills it.)  If the ASF were to buy the dataset, we could distribute to the committers on Lucene according to the licensing terms, but not to the broader community and we’d have to maintain a list of who has it, etc.  See the terms here, for instance.  Since many of the best ideas come from the community in Open Source and you never know when and where they come from, I deemed this unacceptable and decided not to pursue it even though the ASF authorized me to go forward with it (i.e. spend the money) if I wanted to.  After all, it is only a few hundred bucks.

To me, it is vital that there be an open and FREE means for doing relevance tests that the Lucene community can use to improve itself.  If others can benefit, so be it.  Much like Lucene developed a benchmarking tool for people to share performance tests (both speed and relevance) in a straightforward way (see the contrib/benchmark section of the Lucene distribution), so to is there a need for us (speaking, unofficially, for Lucene) to talk about relevance in a public way so we can compare notes just as any two researchers buried in the bowels of a commercial company might compare notes.   Many, many people have used Lucene to do TREC (in fact, I have), but it is a showstopper when the other person you are discussing relevance with can’t just pick up the exact same bits (corpus, queries, judgments) and run the exact same tests.  In other words, the goal is not to compare competing offerings, IMO, (although it will likely happen b/c that is human nature) it is to give Lucene users a common way of evaluating and talking about relevance.

As anyone familiar with Lucene knows, ORP will be driven by the people that show up and volunteer to contribute to it, as are all Apache projects.  Thus, the slate really is clean.  If anyone (and I truly mean anyone, not just Lucene users, even though that is the preliminary focus) is interested, please show up and discuss over at general@lucene.apache.org.   We’d welcome the ideas and, moreover, any efforts.

With the Data Import Handler, for example, one can now easily hook Solr up to any database that supports JDBC and have it index the database for search.  Since most databases aren’t particularly good at text search, this will be a really attractive option for a lot of people who want to index a database.

The new distributed search support is also quite nice in that it allows developers to work with even larger collections without having to roll their own complex solution in low level Lucene.  While it currently requires some operations support to be bullet proof (think load balancers, etc.), it is a great first step and I know it is already used in production by several large installations.

As for spell checking, it used to be that one had to issue a separate query to get spelling results through the SpellcheckRequestHandler, now, thanks to Solr’s pluggable SearchComponent architecture, Google-style “Did you mean” results are available in the response with a query (assuming you turn the component on).

We also are now publishing maven artifacts (but note, I forgot to do it w/ the original distribution process, so they may not show up yet).

For all the features, fixes, etc. see the announcement on the Solr homepage and the link to the release notes.

Text Processing: Why Servers Choke : Beyond Search
If you’ve been wondering how slow Lucene is, this paper gives you some metrics. The data seem to suggest that Lucene is a very slow horse in a slow race.

Are we reading the same paper?  This hardly says Lucene is a slow horse in the race.  What it says, is that Lucene’s StandardTokenizer is slow in comparison to the papers approach for this one particular piece.  Quite a leap to say that Lucene overall is slow, which just doesn’t hold water with most people’s experience.  It also doesn’t compare it to other search engines.  Most notably, the authors fully admit that their comparison “is not strictly an apples to apples comparison” because Lucene’s StandardTokenizer does other things to produce tokens that are actually useful for the user further down the stream, like identifying email and web addresses, etc.  Don’t get me wrong, I’m not saying SpeedyFX isn’t interesting and worthwhile, just saying people shouldn’t infer something in a paper that isn’t there.

Note also that Lucene 2.3 has much improved indexing speed and this paper was written against 2.2, and many of the speedups focus on tokenization during the indexing process (i.e. object creation, object reuse, etc.).  We also upgraded our grammar to use JFlex, which we found to be much faster than JavaCC.  Can’t say what the numbers are in relation to this paper, but it would be interesting to see.  Perhaps the SpeedyFX people can share their code so we can all see.  I know, I know, researchers don’t like to do that, but to me it’s always a big gaping hole in these kinds of papers.

FWIW, StandardTokenizer is just one of many approaches to tokenization that Lucene provides. Furthermore, it is often not the long pole in the tent when it comes to indexing speed.

Still, the ideas are worth looking into.  Lucene’s always open to improvements, and all can benefit from them, as is the beauty of Open Source.

Congrats to the Hadoop team!  Score one for Open Source!

Jeff’s Search Engine Caffè
Copyright and distribution issues
Let’s say for a minute that a web search track is interesting. A major barrier to improvements in academic and open source web search is the lack of large-scale (hundreds of millions or even billions of pages) test collections that evolve over time. GOV2 is a static crawl of 25 million government documents and can therefore be distributed without too much hassle. Not to mention there is little to no spam. However, there’s a problem: commercial documents are copyrighted! Is it possible to create a large-scale test collection of web documents that can be shared freely? I don’t know the answer to that question. Could could that volume of data even be distributed?

Right, we are not going to get into the distribution/copyright game.  We are going to focus on using collections that are freely available.  Each user would just be told what to download.

For example, we could do something like:

Have the user download a static version of Wikipedia from a specific date, index them however they see fit, then run a set of queries we develop and then rate the top 10 or 20 and post their results, including their actual implementation, which is always lacking other than the usual hand waving of saying “we did stemming and relevance feedback”.  We have the advantage in that we can say EXACTLY what we did, no question on implementation, so, gasp, others can repeat the exact experiments, like any good scientist does, before going on to improve it.   Then, when the next person comes along, they do the same thing.  If they disagree about the judgments for the same run, we have a discussion and one person convinces the other and we move on.   Next, someone will come along with a scoring improvement and post those results, and now people will know the current “best” algorithm for this set of data.

Lather, rinse, repeat for other collections, developed over time.  Any engine can submit, anybody can participate.  Open source at it’s best!

In TREC, the queries and the judgments are provided for free, but one has to pay for the data, or at least most of it, since it is usually owned by Reuters or some other organization.  It isn’t expensive or anything, but it is a barrier none the less, especially for an open source project.  Furthermore, the whole notion of paying for data in this day and age of open source and Creative Commons just doesn’t sit right with me.   Don’t get me wrong, I’m a big fan of TREC, having participated in the past, it provides a valuable service to the proprietary/academic IR community.

So, what does this have to do with Lucene?  When I say I am trying to get my hands on TREC data, I don’t mean just for me, I literally mean obtaining TREC data for Lucene.  That is, I want the data to be made available, ideally, for all Lucene (and, for that matter, all open source search engine) users to use and run experiments on so as to spur on innovation in Lucene’s scoring algorithms, etc.  Now, I know the copyright owners will never allow this, as I have asked.  So, my next thought was let’s just get it for internal use by committers at Apache.  So, I went back to TREC and we have an agreement to do this, more or less.  The problem, however, is that they say we can only use the data on ASF (Apache) machines.  Not a big deal, right?  Kind of.  The ASF doesn’t really have the hardware to run TREC style experiments.  We pretty much have one Solaris “zone” alloted us (a “zone” is a virtual machine guest image running.)  Furthermore, the ASF is pretty much an all volunteer, worldwide distributed organization.  We do almost all of our work on our own machines as VOLUNTEERS.   Practically speaking, the best way for any of us to take advantage of the data is to have it locally, which I am told, isn’t going to happen.

So, what’s the point?  I think it is time the open source search community (and I don’t mean just Lucene) develop and publish a set of TREC-style relevance judgments for freely available data that is easily obtained from the Internet.  Simply put, I am wondering if there are volunteers out there who would be willing to develop a practical set of queries and judgments for datasets like Wikipedia, iBiblio, the Internet Archive, etc.  We wouldn’t host these datasets, we would just provide the queries and judgments, as well as the info on how to obtain the data.  Then, it is easy enough to provide simple scripts that do things like run Lucene’s contrib/benchmark Quality tasks against said data.

Practically speaking, I don’t think we even need to go as deep as TREC.  I think we would find the most use in making judgments on the top 10 or 20 results for any given query.

So, what do others think?  Am I off my rocker?  Are there any volunteers out there?  I think we could do this pretty simply through some scripts, and the effective use of a wiki.  I don’t think our goal is, in the short run, to be scientifically rigorous, but it should be over time.  Instead, I think our goal is to run a practical relevance test like any organization should when deploying search: take 50 (top) queries and judge them, as well as 20 or so random queries and judge them.  (I wonder if Wikipedia would give us there top 50 queries, or maybe it is already available.)  Over time, we can add queries, and refine judgments using the web 2.0 mentality of the wisdom of crowds.

FWIW, there is probably some alignment with the Wikia search project.

I did a FeatherCast today with Rich Bowen.  Dang, he is quick at editing…