Hope to see you in Boston!

CFP is now live for talks in Prague this May.

I’m looking forward to my first visit to Prague and also looking forward to talking with all the great Lucene users over in Europe.

Hope to see you there,

Grant

Lucene/Solr Meetup, June 3

http://www.meetup.com/SFBay-Lucene-Solr-Meetup/

Join us for an evening of presentations and discussion on
Lucene/Solr/Nutch/Mahout (and the rest of the Lucene ecosystem), the Apache Open Source Search Engine/Platform, featuring:

-Erik Hatcher, Apache Lucene/Solr PMC: Solr power
your data: How to get up an running in 20 minutes or less
-Grant Ingersoll, Apache Lucene/Solr PMC: New in Apache Solr 1.4 — faster performance, better replication, and more
-Additional topics to be posted at the URL shortly

We’d also like to have 15 minute lightning talks where people present their uses of Lucene/Solr/Tika/Mahout/Nutch/Droids.

We’ll have some food and beverages.

RSVP — seats are limited — at http://www.meetup.com/SFBay-Lucene-Solr-Meetup/

Sponsored by: Lucid Imagination

Please email questions of list to talks@lucidimagination.com

I’ll be at BarCampRDU tomorrow.  I proposed two sessions, one on Hadoop and Mahout and one on Lucene and Solr.  I don’t think I really want to do both, but I would like to do at least one, so we’ll see what other people are interested in.

If you’re around and you want to talk about any of these things, track me down.

In TREC, the queries and the judgments are provided for free, but one has to pay for the data, or at least most of it, since it is usually owned by Reuters or some other organization.  It isn’t expensive or anything, but it is a barrier none the less, especially for an open source project.  Furthermore, the whole notion of paying for data in this day and age of open source and Creative Commons just doesn’t sit right with me.   Don’t get me wrong, I’m a big fan of TREC, having participated in the past, it provides a valuable service to the proprietary/academic IR community.

So, what does this have to do with Lucene?  When I say I am trying to get my hands on TREC data, I don’t mean just for me, I literally mean obtaining TREC data for Lucene.  That is, I want the data to be made available, ideally, for all Lucene (and, for that matter, all open source search engine) users to use and run experiments on so as to spur on innovation in Lucene’s scoring algorithms, etc.  Now, I know the copyright owners will never allow this, as I have asked.  So, my next thought was let’s just get it for internal use by committers at Apache.  So, I went back to TREC and we have an agreement to do this, more or less.  The problem, however, is that they say we can only use the data on ASF (Apache) machines.  Not a big deal, right?  Kind of.  The ASF doesn’t really have the hardware to run TREC style experiments.  We pretty much have one Solaris “zone” alloted us (a “zone” is a virtual machine guest image running.)  Furthermore, the ASF is pretty much an all volunteer, worldwide distributed organization.  We do almost all of our work on our own machines as VOLUNTEERS.   Practically speaking, the best way for any of us to take advantage of the data is to have it locally, which I am told, isn’t going to happen.

So, what’s the point?  I think it is time the open source search community (and I don’t mean just Lucene) develop and publish a set of TREC-style relevance judgments for freely available data that is easily obtained from the Internet.  Simply put, I am wondering if there are volunteers out there who would be willing to develop a practical set of queries and judgments for datasets like Wikipedia, iBiblio, the Internet Archive, etc.  We wouldn’t host these datasets, we would just provide the queries and judgments, as well as the info on how to obtain the data.  Then, it is easy enough to provide simple scripts that do things like run Lucene’s contrib/benchmark Quality tasks against said data.

Practically speaking, I don’t think we even need to go as deep as TREC.  I think we would find the most use in making judgments on the top 10 or 20 results for any given query.

So, what do others think?  Am I off my rocker?  Are there any volunteers out there?  I think we could do this pretty simply through some scripts, and the effective use of a wiki.  I don’t think our goal is, in the short run, to be scientifically rigorous, but it should be over time.  Instead, I think our goal is to run a practical relevance test like any organization should when deploying search: take 50 (top) queries and judge them, as well as 20 or so random queries and judge them.  (I wonder if Wikipedia would give us there top 50 queries, or maybe it is already available.)  Over time, we can add queries, and refine judgments using the web 2.0 mentality of the wisdom of crowds.

FWIW, there is probably some alignment with the Wikia search project.

I did a FeatherCast today with Rich Bowen.  Dang, he is quick at editing…

This guy’s comment system wasn’t working at the moment, so I will leave my comment here. This won’t make much sense without reading the post first:

It’s funny you mention Wikipedia as an example, since they are running Lucene. As is Technorati and the Internet Archive. As is IBM Omnifind Yahoo! Edition. Are those big enough for you? If not, then choose any of them at http://wiki.apache.org/lucene-java/PoweredBy
And that list is just the companies who are public about it.

Speaking for myself (and not the ASF), as a Lucene developer, I would love to see us using it at Apache. It is something we are well aware of and have discussed. However, Lucene, like all Apache projects is VOLUNTEER and our volunteer infrastructure team is already loaded providing support to the actual products in terms of Subversion, JIRA, Confluence/MoinMoin, countless mailing lists, guarding against security attacks, creating new projects, etc. Simply put, it requires resources and time. Perhaps I can find some time between my day job and the volunteer work I do actually making the code better, supporting the community and occasionally administrating the nightly builds on our virtual servers, etc. to find time to deploy and maintain Nutch (which, mind you would do just fine for the job, just ask, aw never mind, we’ve been down that road) in a 24/7 high volume website. Even Google or your ISP has people working in operations to make sure even the most stable things are running and not being attacked/spammed/you name it, so Apache would be no different.

And, just so we are clear, every developer of Lucene “eats the Lucene/Nutch/Solr dog food”, we just don’t necessarily do it at Apache.org. I use it my day job. I use it in pet projects, I recommend it to clients, etc. I even use it in things that 5 years ago I would never have thought I would use it for (object stores, etc.) If that isn’t eating my own dog food, than I don’t know what dog food tastes like.

Finally, I don’t think our priority is to be squeaky clean. My personal one is to make sure Lucene is as good as it can be within my personal limitations. Just go look at our JIRA installation or our mailing lists to see all of the dirt. We aren’t hiding it.