com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
link failure
Last packet sent to the server was 4467745 ms ago
...

com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1074)
	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2985) 	at
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2871) 	at

In my code, I loop over a JDBC ResultSet and add the records to Solr per the Solr field schema, mapping columns to fields, etc.  This would happen after getting through something like 9M+ records.  After some tracking down, hypothesizing, talking with others, we came to the conclusion that the issue was a combination of having the autocommit value in Solr set and MySQL timing out the ResultSet, such that when Lucene had to do a large merge (even in the background), Solr had to wait for said merge to finish, thus keeping the ResultSet open too long w/o activity.  Now, these large merges can take some time.  They can happen in the background, but Solr can’t refresh it’s IndexReader until the merge finishes, AIUI.  Thus, we’re stuck in the middle of a ResultSet loop, holding the cursor open past MySQL’s default setting (600 seconds, more on that later), causing MySQL to kill the connection, and rightfully so.  On the MySQL side of things, we are streaming the results, since it’s JDBC driver does not support setFetchSize() (ugh!).  As it turns out MySQL has a Streaming timeout value named netTimeoutForStreamingResults (see http://dev.mysql.com/doc/refman/5.0/en/connector-j-reference-configuration-properties.html) which defaults to 600 seconds.

Long story short, I have at least two options:

1. Turn off autocommit, meaning user’s won’t be able to see documents as soon as they may like
2. Increase the netTimoutForStreamingResults value.  This is great for MySQL and I have verified it works, but is not a generic value for other DBs, which our code supports

I am still deciding on what to do, and also thinking of some other options that can decouple DB retrieval from the indexing process.  At any rate, I wanted to post the cause of my seeing this exception, because I did not see anyone else with this exception whose cause was due to a timeout during ResultSet processing and hopefully it will save them some time.

Nice article on how the Lucene/Hadoop/Solr stack was used to solve a really big problem.  Someday, I hope (when we have actual code),  they can add Mahout to the equation and do even more interesting things with the data.

Well stated response to a criticism on Map Reduce.  Adding my own two cents, I once used Hadoop, a free open source implementation of Map Reduce (M/R) in a proof of concept implementation, to automatically translate (as in machine translation) a large (in my terms) collection of documents from one language (Arabic) to another (English).  It’s something that would be really hard to do in a database.  Besides, I had a bunch of dumb old machines laying around, while I didn’t have a $1 million plus license of Oracle laying around.

Other things M/R is nice for: crawling (see Nutch) and parallel indexing for search engines; log analysis,  machine learning, etc.

Finally, I first started doing parallel programming a fair number of years ago (remember the CM-5 from Thinking Machines?) and we used the Message Passing Interface APIs (MPI) amongst others.  As the author of the article above stresses, M/R is good for SOME large scale programs (see the new Mahout project at Apache, for some examples).  There are some problems that are really large and just don’t fit in the M/R model.  As with anything you do in life, take the time to figure out which one is right for you.  You may have to rise above yourself and learn something new.