Just a few more days until the next Triangle Hadoop User’s Group meeting.  Get the details and sign up via Triangle Hadoop Users Group, TriHUG Next Meeting featuring Josh Patterson of Cloudera set for Oct. 11.

So, if you are so inclined to do the whole crowdsourcing thing, please go vote for my talk at SXSW 2012 – Apache Mahout: Bringing Intelligence to Your App and then maybe I will see you at SXSW in 2012.

First off, I was doing a fair amount of work calculating document similarities across whole collections using, at first, the RowSimilarityJob and later a map-side simplification I wrote that uses the distributed cache called the VectorDistanceSimilarityJob.  Both of these come in handy when one wants to calculate pairwise-similarity between all (or most) items in a collection.  The original Mahout implementation was focused on providing recommendations, but as outlined in the Elsayed, Lin and Oard paper, it is quite useful for text as well in cases where one wants to precompute “more like this” for all documents.  As for the need for two similar approaches, see the discussion at http://www.lucidimagination.com/search/document/40c4f124795c6b5/rowsimilarity_s#42ab816c27c6a9e7.  In essence, it boils down to I didn’t need a fully generic implementation that was a bit slower on larger matrices since I mainly wanted to compare all my vectors in HDFS against a subset of “core” vectors that fit into memory.  That being said, Sebastian is already hard at work on making the more generic version perform better when certain distance measures are used while still offering the full suite of capabilities of the existing RowSimilarityJob.  See MAHOUT-767 for more info on that work.

Now, I’m looking into some more pruning techniques via MAHOUT-688.  After that quick patch, I think I’m going to dig in a bit more to recommendations as well as run some tests on the ASF mail archives I posted a while back (see below for an update).

Also, I’ve switched to using Git and Github for managing my Mahout changes (as well as other work), so if you want to see what I’m up to, check out my Github account.

It’s not complete yet, but the ASF Public Mail archive I put up last September on Amazon AWS is getting a fresh new version.  The interim solution is available at https://s3.amazonaws.com/asf-mail-archives-7-18-2011/index.html, but look for it to be a Public Data Set hosted by Amazon soon.  The September version of this data contained roughly 6.7M emails sent to the public mailing lists at the Apache Software Foundation, so I suspect this version has somewhere in the 7M+ item range, but I haven’t counted them.  At any rate, I hope it is useful to people.

Finally, on a personal note, I’m back at Lucid Imagination after a brief move elsewhere, this time in a new role as Chief Scientist.  Lucid is a company I co-founded and helped build up for the past 4 years.  I’m looking forward to be back working closely with Lucene and Solr again and a top notch technical team.  I’m also looking forward to working on Mahout more, as well as other technologies like Hadoop, Pig, HBase and the like, especially as they relate to search and recommendations.

Wei Wei on Practical Hadoop Security and Me on Hadoop and Lucene and Solr.

For more info and to RSVP, see Triangle Hadoop Users Group.

We are starting with some introductory talks, but we will no doubt get much deeper very quickly.  If you’d like to give a talk or sponsor or just attend, drop on over to the TriHUG website for more information.

Hope to see you there!

First off, special thanks to all the speakers (Jason Rutherglen, Michael Busch, Erik Hatcher and all the lightning talks.)  We had a lot of excellent talks ranging from low level Lucene details on payloads and real time search to high level discussions on new feature in Solr and best practices for working on stopwords and relevance.  Also had intros to Mahout, Tika and the new Open Relevance project at Lucene.  I’ll post the slides on the Meetup site when they are available (I am still waiting to get them from the speakers.)

Second, I really enjoyed engaging with so many people about what they are working on in Lucene/Solr.  It is always fun to hear all the different ways people are (ab)using Lucene/Solr to do cool things, etc.   It was especially good to meet some fellow Mahout committers (Ted Dunning and Jeff Eastman) for the first time, as well as one of Mahout’s Google Summer of Code student David Hall, who is working on adding Latent Dirichlet Allocation.

Finally, I look forward to doing more of these.  Right now, I’m looking for interest in Raleigh, NC, but I know we’ll likely have another one in the Bay Area again soon.

Nice writeup on Hadoop in the NYT today.  Of course, Hadoop is often used to power machine learning, too, which is the premise behind using it on Apache Mahout.

Surprise and Coincidence – musings from the long tail: Real-time decision making using map-reduce