Comments on: Open Source Search Engine Relevance

By: Jeff

Jeff — Thu, 22 May 2008 04:21:27 +0000

Grant - Interesting ideas!

What use cases should the collections be designed for? Things that come to mind: enterprise search, web search, product search, etc…

Also, what about interactive retrieval scenarios? This is one of the major drawbacks to TREC-like evaluations. Pooling also has its limitations (see the recent Terabyte Track summaries).

Could a possible solution be a platform like Alexa’s Web Search Platform? The documents would be hosted on the cluster and not distributed (getting around copyright and distribution problems), but could be processed to create search indexes. You could even create a working service and collect real queries. Who knows, you could even before A/B relevance tests on the best systems using live traffic.

See my blog for more thoughts.

By: grant_ingersoll

grant_ingersoll — Tue, 20 May 2008 10:57:28 +0000

I agree, Ian, but again, not sure if we need that data. As I understand TREC, they have analysts who read documents, etc. and come up with queries knowing somewhat that there are answers available.

I think we could have people come up with queries and then we make human judgments just as any organization would do in house.

The bigger question to me now, is this something people are interested in doing? That is, would people be interested in starting/joining a “Lucene Relevance” project as a subproject of Lucene? We can figure out the details from there.

By: Ian

Ian — Tue, 20 May 2008 04:24:19 +0000

I don’t think it is the ‘data’ component that makes this difficult, but the analysis of which results are better.

Ideally you would need to use some kind of crowdsourcing to judge which result is more relevant. (ie 95% of users click on X when they search for ‘moo’).

The problem then becomes one of privacy. you need to approach a organization to produce a anonymous version of their query logs (which is hard).

If you could convince wikipedia to just release the search query, and the result clicked you could then use this as a basis of judgement.

By: grant_ingersoll

grant_ingersoll — Mon, 19 May 2008 19:53:34 +0000

I don’t know that we need to use TREC data after the eval. I think I am proposing something that is completely open and repeatable by anyone using freely available data. Download X corpus, get queries from the project site, run them and make judgments on top 10-20 and post them/edit a wiki, lather, rinse, repeat.

It isn’t fully thought out as to how it all works. I’m just saying there is a need for some type of open source relevance project. It goes beyond Lucene, in my mind, but I’m happy to start it here.

I am no doubt sure there are smarter people out there than me that can figure out the details of how to make this be rigorous. I just want to see if I can kickstart the discussion and get something going.

By: Bob Carpenter

Bob Carpenter — Mon, 19 May 2008 17:27:08 +0000

How do you plan to use TREC data after the eval? The data consists of evaluations of relevant/irrelevant/maybe-relevant for the top K (usually K = 100 or 200) results for each of the participants. This gives you exact precision-at-K for submitted systems, but doesn’t let you evaluate recall or even precision-at-K for systems developed after the eval.

By: grant_ingersoll

grant_ingersoll — Mon, 19 May 2008 13:45:39 +0000

The restriction, as I understand it, is one of distribution. Taking the docs off of a “ASF machine” implies distribution and is a violation of the terms of the agreement. We can copy it to machines where the ASF is responsible for setting up accounts, or the accounts are based on the ASF credentials

By: Ken Krugler

Ken Krugler — Mon, 19 May 2008 13:25:08 +0000

Hi Grant,

Is the “only on ASF machines” restriction a data storage restriction, or a data processing restriction?

If it’s the former, then there would be obvious work-arounds.

And in either case, yes it would be great to have data to use for relevance research with Lucene and other open source search engines.

– Ken

By: Daniel Naber

Daniel Naber — Sun, 18 May 2008 18:55:28 +0000

Grant, sounds like a great idea! Wikipedia data is available at http://stats.grok.se/, although that seems to be the page titles, not the user’s original queries.