Comments on: Opening up Academic Research on IR and Machine Learning

By: Copying TREC is the Wrong Track for the Enterprise | The Noisy Channel

Copying TREC is the Wrong Track for the Enterprise | The Noisy Channel — Tue, 19 May 2009 03:23:20 +0000

[...] http://lucene.grantingersoll.com/category/trec/ and http://lucene.grantingersoll.com/2008/09/18/opening-up-academic-research-on-ir-and-machine-learning/ for background. The second motivation is simply to have practical, real world data driven by [...]

By: Hannes Carl Meyer

Hannes Carl Meyer — Sat, 20 Sep 2008 12:47:34 +0000

P.S.: Just bought your book and looked into the early access papers, congratulations!

By: Hannes Carl Meyer

Hannes Carl Meyer — Sat, 20 Sep 2008 11:59:41 +0000

I didn’t do any research in Mahout yet, but I think Apache UIMA framework is a good example in how to provide a common platform.

We started our development on a niche text processing software 5 years ago. Thats when we got into IBM’s UIMA framework and integrated it into our software right in the beginning.
Beside integrating our own text analysis methods we were looking a lot into university paper publications which were sometimes either hard to understand and not even practical at all.

This changed couple years ago when I saw more and more folks on universities starting to work with UIMA as a platform. They even started exchanging components each other. On a congress about UIMA last year I saw 8 project presentations of different UIMA projects, 6 of them from university, 2 of them from the industry (1 from the netherland, one from germany)

Over here in germany it looks like the university research in text processing is already ahead of the industry and the industry have to get into to benefit from research.

I will do my Mahout homework this weekend.

By: grant_ingersoll

grant_ingersoll — Fri, 19 Sep 2008 19:18:24 +0000

Yeah, I totally agree on your points, Bob. The data problem is particularly true. It’s interesting, because if I was a physicist, I would be required to publish the steps I took in order to reproduce an experiment. In our world, we just put up a formula or two, maybe a nice description and some pseudocode and then wave our hands and magically we get a nice, pretty, well-formatted table of results showing a 10% increase in mean average precision or some great F-measure. Pay no attention to the man behind the curtain.

Sharing the code and configuration is a good starting point.

By: Bob Carpenter

Bob Carpenter — Fri, 19 Sep 2008 16:39:14 +0000

The tools that are easy to use get widely re-used. For instance, Adwait Ratnaparkhi’s POS tagger or Collins’ parser or the Stanford NE tagger, which present usable tools out of the box that run fro the command line. Then there are tools like Joachims’ SVMLight or Bottou’s SGD that are easy to use from the command line.

There’s a big problem in recreating all the features, because papers usually only contain sketches of what’s used. And that’s where the accuracy comes from on actual tasks. And there’s also a big problem recreating heuristic pre- and post-processing. And then there’s patching together whole pipelines using instances of various resources like Wikipedia, Wordnet, and so on.

The big problem for sharing is scale. If I run over a terabyte of web data, or even over 50 GB, how do I share that? TREC has done things like distribute disk drives on their “large” scale tasks, which aren’t even large scale these days.