Grant's Grunts: Lucene Edition » Solr

Berlin Buzzwords 2012

grant_ingersoll — Wed, 18 Jan 2012 13:33:40 +0000

In case you haven’t heard, and are in Europe this June (or want to be), you should check out the Berlin Buzzwords conference. It’s a great conference for all things related to Lucene, Solr, Hadoop, Mahout, NoSQL and generally scaling. The CFP is open now through March 11.

Taming Text Update

grant_ingersoll — Tue, 27 Dec 2011 13:45:39 +0000

Drew, Tom and I are feverishly working away on finishing up Taming Text. We are currently in the process of addressing the feedback we got from our final review and should have updates up soon. I have also posted all of the book’s source code up on Github under the Taming Text user. The source includes, amongst other things, a simple Question Answering system using Solr and OpenNLP, as well as analyzers for Lucene that use OpenNLP for sentence detection, part of speech tagging and Named Entity Recognition. As with most books, these examples are meant to be just that, examples.

Mahout and Other News

grant_ingersoll — Fri, 05 Aug 2011 20:41:35 +0000

After some time away, I’m happy to have had some time recently to work on Mahout again. Lots of goodness all over the place happening there that I’ll leave to others to explain while I focus in on a few recent things I’ve been doing.

First off, I was doing a fair amount of work calculating document similarities across whole collections using, at first, the RowSimilarityJob and later a map-side simplification I wrote that uses the distributed cache called the VectorDistanceSimilarityJob. Both of these come in handy when one wants to calculate pairwise-similarity between all (or most) items in a collection. The original Mahout implementation was focused on providing recommendations, but as outlined in the Elsayed, Lin and Oard paper, it is quite useful for text as well in cases where one wants to precompute “more like this” for all documents. As for the need for two similar approaches, see the discussion at http://www.lucidimagination.com/search/document/40c4f124795c6b5/rowsimilarity_s#42ab816c27c6a9e7. In essence, it boils down to I didn’t need a fully generic implementation that was a bit slower on larger matrices since I mainly wanted to compare all my vectors in HDFS against a subset of “core” vectors that fit into memory. That being said, Sebastian is already hard at work on making the more generic version perform better when certain distance measures are used while still offering the full suite of capabilities of the existing RowSimilarityJob. See MAHOUT-767 for more info on that work.

Now, I’m looking into some more pruning techniques via MAHOUT-688. After that quick patch, I think I’m going to dig in a bit more to recommendations as well as run some tests on the ASF mail archives I posted a while back (see below for an update).

Also, I’ve switched to using Git and Github for managing my Mahout changes (as well as other work), so if you want to see what I’m up to, check out my Github account.

It’s not complete yet, but the ASF Public Mail archive I put up last September on Amazon AWS is getting a fresh new version. The interim solution is available at https://s3.amazonaws.com/asf-mail-archives-7-18-2011/index.html, but look for it to be a Public Data Set hosted by Amazon soon. The September version of this data contained roughly 6.7M emails sent to the public mailing lists at the Apache Software Foundation, so I suspect this version has somewhere in the 7M+ item range, but I haven’t counted them. At any rate, I hope it is useful to people.

Finally, on a personal note, I’m back at Lucid Imagination after a brief move elsewhere, this time in a new role as Chief Scientist. Lucid is a company I co-founded and helped build up for the past 4 years. I’m looking forward to be back working closely with Lucene and Solr again and a top notch technical team. I’m also looking forward to working on Mahout more, as well as other technologies like Hadoop, Pig, HBase and the like, especially as they relate to search and recommendations.

Stump the Chump

grant_ingersoll — Sun, 24 Apr 2011 06:09:39 +0000

I’m on the hot seat for “Stump the Chump” this year at Lucene Revolution, so if you have questions you want me to tackle, please either show up at my talk or email them to info@lucenerevolution.org. See Session Abstracts | Day 1 | www.lucenerevolution.org for more information.

Apache Lucene 3.1.0 and Apache Solr 3.1.0

grant_ingersoll — Thu, 31 Mar 2011 18:37:35 +0000

I just sent out to the mailing lists the official release announcements for Lucene and Solr 3.1.0 as well as posted over on the Lucid Imagination site the release announcement, etc.

Lucid Imagination » Apache Lucene 3.1.0 and Apache Solr 3.1.0.

Thanks to everyone for all their hard work in making these releases happen.

Lucene, Solr and SXSW

grant_ingersoll — Thu, 17 Mar 2011 12:16:42 +0000

Finally back in the saddle from SXSW, where a good time was had by all, AFAICT. It was my first time, so it was a bit overwhelming. So many people, so much hype, so much to do and see. Suffice it to say, most of the hype was about social media. Can you say Bubble? Sure there are some winners in the bunch, but there’s a whole lot of people whose only idea is simply to copy some other venture funded company who has yet to make a profit or to create Yet Another Social Media Monitoring Solution (YAMMS). From the engineering side of these companies, this stuff is incredibly cool and I totally get the excitement. What’s not to like about working on massively scalable problems with incredible update rates using cool new technology like Hadoop, Cassandra, Mahout, etc? From the business side, color me cynical.

Regardless of the hype, I had a good time. Obviously I’m biased, but I think the talk RC Johnson and I gave on using Solr in NoSQL situations went well, was well attended and well received. RC and I split duties during the talk. I covered the big picture around Solr as a NoSQL solution and RC covered how Bazaarvoice uses Solr to serve billions of searches/lookups per month. A good chunk of the audience was already familiar with NoSQL and Lucene/Solr, so it was a pretty technical audience, which is exactly what we hoped for. We also had some good questions during the talk as well as afterward. The slides are up on SlideRocket.

Lucid Imagination » Introducing Unlimited Developer Access to LucidWorks Enterprise

grant_ingersoll — Wed, 15 Dec 2010 14:36:06 +0000

Lucid Imagination » Introducing Unlimited Developer Access to LucidWorks Enterprise.

Our latest version of LucidWorks Enterprise is now available for general access.

Next up, ApacheCon!

grant_ingersoll — Sat, 09 Oct 2010 11:30:04 +0000

If you missed the Lucene Revolution conference, especially the training on Apache Solr and Lucene, don’t worry. You won’t have to wait long for the next round, both Erik Hatcher (co-author of Lucene In Action) and I will be giving trainings at ApacheCon in Atlanta in November.

To learn more and sign up, see:

There will also be almost two days of talks on Lucene, Solr, Mahout and friends.

Upcoming Apache Lucene and Solr public trainings

grant_ingersoll — Mon, 30 Aug 2010 15:33:02 +0000

Erik Hatcher and I are once again offering our Lucene and Solr training classes, but this time there are two opportunities to participate. The first will be at Lucene Revolution on October 5 and 6. The second is on Nov. 1st and 2nd at ApacheCon NA 2010. Both classes are designed to get people up to speed on either Lucene or Solr as quickly as possible. If you have any questions, feel free to drop me an email at trainer@lucenebootcamp.com.

Lucid Imagination » Free Webinar: How Cisco’s Pulse uses Lucene/Solr to put Social Networks to Work

grant_ingersoll — Fri, 18 Jun 2010 12:25:34 +0000

Lucid Imagination » Free Webinar: How Cisco’s Pulse uses Lucene/Solr to put Social Networks to Work.

‘Nuff said.