Solr, an open source search server, is looking for a new logo.  See the link above for details.

Just a reminder that I would appreciate it if I could talk w/ the Solr  person just to hear more about the system and possibly plan for future  upgrades.

Aargh, curse these modern spelling variations!  Curse these fingers that type faster than the brain can think!

Lucene Boot Camp (http://www.lucenebootcamp.com) is scheduled this year for ApacheCon US on November 3 and 4th in New Orleans.  This year, I am doing a two day event, as I felt the one day event was just not enough time to get in all the goodness that is Lucene (not that two days is either, but…)  Additionally, by having two days, the class will have more time to explore and develop the examples with the class and more one on one time.  There is also a fair amount of new material in Lucene these days, so I am going to be updating the examples, etc. on the website and changing things up a bit.

This class has traditionally filled up pretty fast.  In Amsterdam this past April, there was even a waiting list, so I recommend signing up early.

Also, for those coming from the US, note that the class is scheduled over on Tuesday, Nov. 4, which is election day.  Not sure who decided that one, but, if you’re planning on voting and attending my class, make sure you get an absentee ballot.  Which reminds me…

The author of this nice plugin for WordPress contacted me today about his Lucene based WordPress plugin, so I thought I would give it a try, as I’m obviously a big fan of Lucene and also never much cared for MySql’s search (in)capabilities.

The plugin is easy enough to install, only thing that struck me as a little odd was the need to set 777 on the permsissions.  Presumably, this is so it can write the index, but perhaps it would be better to store the index outside of the plugin infrastructure.  Of course, I don’t know what’s involved with writing plugins, etc. so not sure if that makes sense or not.

Indexing and enabling search was a snap, and I do think the results are good, even if my sites don’t have a ton of posts.  Indexing on my “main” site (http://www.grantingersoll.com) took slightly longer than here, but I do have more posts and comments there.

Only minor suggestion I would have is that the default boosts for title and content aren’t all that great.  I think the title boost was 1.8 and the content boost was 1.3.   I changed mine to be title: 5 and content: 2.  The way boosting works at indexing time, it has only 8 bits of granularity, there isn’t too much difference between 1.8 and 1.3 and I tend to think title matches are much more important.  Thus, I made them greater.  Still, very cool that the author has hooked field boosting in to begin with.

Things I would love to see:

1. Highlighting
2. Spell checking

All in all, seems to be a great little plugin.  And now, I can “eat my own dogfood” too!

I’ll be at BarCampRDU tomorrow.  I proposed two sessions, one on Hadoop and Mahout and one on Lucene and Solr.  I don’t think I really want to do both, but I would like to do at least one, so we’ll see what other people are interested in.

If you’re around and you want to talk about any of these things, track me down.

Boy, this could really come in handy in Open Source, especially projects like Mahout, Nutch and distributed Solr.  I find my biggest personal challenge on Mahout is access to computing resources.  I personally don’t have the financial backing to buy much time on Amazon EC2.  I have been scraping by, here and there, but find myself constantly wanting access to more capabilities.

Sigh.  Maybe I should put more ads on this site and use the funds for buying EC2 time.

com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
link failure
Last packet sent to the server was 4467745 ms ago
...

com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1074)
	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2985) 	at
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2871) 	at

In my code, I loop over a JDBC ResultSet and add the records to Solr per the Solr field schema, mapping columns to fields, etc.  This would happen after getting through something like 9M+ records.  After some tracking down, hypothesizing, talking with others, we came to the conclusion that the issue was a combination of having the autocommit value in Solr set and MySQL timing out the ResultSet, such that when Lucene had to do a large merge (even in the background), Solr had to wait for said merge to finish, thus keeping the ResultSet open too long w/o activity.  Now, these large merges can take some time.  They can happen in the background, but Solr can’t refresh it’s IndexReader until the merge finishes, AIUI.  Thus, we’re stuck in the middle of a ResultSet loop, holding the cursor open past MySQL’s default setting (600 seconds, more on that later), causing MySQL to kill the connection, and rightfully so.  On the MySQL side of things, we are streaming the results, since it’s JDBC driver does not support setFetchSize() (ugh!).  As it turns out MySQL has a Streaming timeout value named netTimeoutForStreamingResults (see http://dev.mysql.com/doc/refman/5.0/en/connector-j-reference-configuration-properties.html) which defaults to 600 seconds.

Long story short, I have at least two options:

1. Turn off autocommit, meaning user’s won’t be able to see documents as soon as they may like
2. Increase the netTimoutForStreamingResults value.  This is great for MySQL and I have verified it works, but is not a generic value for other DBs, which our code supports

I am still deciding on what to do, and also thinking of some other options that can decouple DB retrieval from the indexing process.  At any rate, I wanted to post the cause of my seeing this exception, because I did not see anyone else with this exception whose cause was due to a timeout during ResultSet processing and hopefully it will save them some time.

Congrats to the Hadoop team!  Score one for Open Source!

Jason Rutherglen has been up to some interesting things with Lucene lately concerning real time search.  This has always been one of those parts of Lucene that has been needed over time by some people, but has never reached the critical mass whereby someone tackles it.  Looks like Jason finally has, and I am glad to see it.  I have, on occassion, met with clients, or users who truly have needed it, but most have usually been able to live with a little bit of a delay and the associated workarounds.  Now, it seems, they won’t have to.

What this means, is that one can now get back search results for the given query, and get spelling suggestions at the same time, pretty much like Google’s “Did You Mean” functionality (but probably not the same quality, as they have a much bigger corpus and probably use user feedback as well.)

For details on how to use it, try out the Solr example in the source distribution and see the Wiki docs.

Also note, that it allows one to plug in their own spell checker (or a commercial one) or use the Lucene spell checker.