Like Spans, payloads involve the position of terms, but go one step further. Namely, a Payload in Apache Lucene is an arbitrary byte array stored at a specific position (i.e. a specific token/term) in the index. A payload can be used to store weights for specific terms or things like part of speech tags or other semantic information.

via Lucid Imagination » Getting Started with Payloads.

My latest article on Apache Solr, title “What’s New with Apache Solr” is now available over at IBM developerWorks.  It covers some of the new features like spell checking, Data Import Handler, distributed search, editorial results placement (a.k.a. “paid placement”), SolrJ and a variety of other pieces.

Hope it is helpful…  Feel free to give me any feedback.

This year, it is a 2 day training, but for those who want to, they can sign up for the first day of Lucene Boot Camp, and then attend Solr Boot Camp on the second day.  I am going to structure the class such that the first day is more high-level stuff (but still with some coding) while the second day will be much more hands-on.  Thus, for managers, CTOs, non-programmers, who just want to understand Lucene and Solr, they can easily plan on attending 1 day of each.

Lucene Boot Camp (http://www.lucenebootcamp.com) is scheduled this year for ApacheCon US on November 3 and 4th in New Orleans.  This year, I am doing a two day event, as I felt the one day event was just not enough time to get in all the goodness that is Lucene (not that two days is either, but…)  Additionally, by having two days, the class will have more time to explore and develop the examples with the class and more one on one time.  There is also a fair amount of new material in Lucene these days, so I am going to be updating the examples, etc. on the website and changing things up a bit.

This class has traditionally filled up pretty fast.  In Amsterdam this past April, there was even a waiting list, so I recommend signing up early.

Also, for those coming from the US, note that the class is scheduled over on Tuesday, Nov. 4, which is election day.  Not sure who decided that one, but, if you’re planning on voting and attending my class, make sure you get an absentee ballot.  Which reminds me…

The author of this nice plugin for WordPress contacted me today about his Lucene based WordPress plugin, so I thought I would give it a try, as I’m obviously a big fan of Lucene and also never much cared for MySql’s search (in)capabilities.

The plugin is easy enough to install, only thing that struck me as a little odd was the need to set 777 on the permsissions.  Presumably, this is so it can write the index, but perhaps it would be better to store the index outside of the plugin infrastructure.  Of course, I don’t know what’s involved with writing plugins, etc. so not sure if that makes sense or not.

Indexing and enabling search was a snap, and I do think the results are good, even if my sites don’t have a ton of posts.  Indexing on my “main” site (http://www.grantingersoll.com) took slightly longer than here, but I do have more posts and comments there.

Only minor suggestion I would have is that the default boosts for title and content aren’t all that great.  I think the title boost was 1.8 and the content boost was 1.3.   I changed mine to be title: 5 and content: 2.  The way boosting works at indexing time, it has only 8 bits of granularity, there isn’t too much difference between 1.8 and 1.3 and I tend to think title matches are much more important.  Thus, I made them greater.  Still, very cool that the author has hooked field boosting in to begin with.

Things I would love to see:

1. Highlighting
2. Spell checking

All in all, seems to be a great little plugin.  And now, I can “eat my own dogfood” too!

com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
link failure
Last packet sent to the server was 4467745 ms ago
...

com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1074)
	at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2985) 	at
com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2871) 	at

In my code, I loop over a JDBC ResultSet and add the records to Solr per the Solr field schema, mapping columns to fields, etc.  This would happen after getting through something like 9M+ records.  After some tracking down, hypothesizing, talking with others, we came to the conclusion that the issue was a combination of having the autocommit value in Solr set and MySQL timing out the ResultSet, such that when Lucene had to do a large merge (even in the background), Solr had to wait for said merge to finish, thus keeping the ResultSet open too long w/o activity.  Now, these large merges can take some time.  They can happen in the background, but Solr can’t refresh it’s IndexReader until the merge finishes, AIUI.  Thus, we’re stuck in the middle of a ResultSet loop, holding the cursor open past MySQL’s default setting (600 seconds, more on that later), causing MySQL to kill the connection, and rightfully so.  On the MySQL side of things, we are streaming the results, since it’s JDBC driver does not support setFetchSize() (ugh!).  As it turns out MySQL has a Streaming timeout value named netTimeoutForStreamingResults (see http://dev.mysql.com/doc/refman/5.0/en/connector-j-reference-configuration-properties.html) which defaults to 600 seconds.

Long story short, I have at least two options:

1. Turn off autocommit, meaning user’s won’t be able to see documents as soon as they may like
2. Increase the netTimoutForStreamingResults value.  This is great for MySQL and I have verified it works, but is not a generic value for other DBs, which our code supports

I am still deciding on what to do, and also thinking of some other options that can decouple DB retrieval from the indexing process.  At any rate, I wanted to post the cause of my seeing this exception, because I did not see anyone else with this exception whose cause was due to a timeout during ResultSet processing and hopefully it will save them some time.

Patches welcome…  I know that is an old saw, but that is the only way it’s going to get better.

There are some good points in here, and some stuff that is a bit dramatic.

We do try to keep adapting Lucene and make it better, but in some respects we are damned if we do, damned if we don’t.  The whole abstract vs. interface debate has been going on for a long time on Lucene.  If we switched to interfaces, then people would be complaining constantly about how we break their code everytime we do a release if we introduce new methods.  If we leave things as abstract classes, then people like Cedric complain that Lucene is hard to extend.

As for the “final” declarations, the reason it works that way, is that we can’t see the future.  Often times, the things that are final now are legacy from back in the early days when we couldn’t imagine some of the uses that Lucene is now being used for, or as another commenter on the thread said, to avoid unintended consequences.  That’s why people submit patches and things get improved.  Sorry we can’t all work on Lucene nonstop all day.  If Lingway wants to hire me to do that, you know how to reach me! (at least, as a contractor, I’m not interested in leaving my current employment, but I do offer consulting.)

As for SpanQueries, yes they can be slower.  I’d love to have a discussion with someone like you who is a heavy user to see how to improve them.  Please send your profiling info ASAP to the java-dev mailing list!  Please don’t let all those hours you spent go to waste.  Even a half baked patch is a starting point.

I do agree about scoring being pluggable, but I gotta tell ya’, scoring is hard and not for the faint of heart.  It’s a whole other layer and doing it right means being fast and accurate and doing it wrong means deep, dark, scary rabbit holes where you don’t see light for days.  One of the simplest ways, however, to improve scoring is to change the length normalization.

As for some of the other “higher” features, like crawling/clustering, those are nice, but they don’t belong in the core of Lucene b/c not everyone needs them, although the number is increasing.  How many people have collections that go beyond 10-20M documents?  What would be nice, however is a contrib module or a layer above Lucene that provides all those nice things you want (you know, you can embed Solr in no time, by the way).  Lucene is meant to be really fast on one machine and to also play nice when you put in the appropriate distributed pieces.  It’s unfortunate that no one has donated the distributed piece yet (although Solr does now have it, thanks to Yonik!)

At any rate, thanks for the ideas.  Hope to see your patches soon!

Nice article on how the Lucene/Hadoop/Solr stack was used to solve a really big problem.  Someday, I hope (when we have actual code),  they can add Mahout to the equation and do even more interesting things with the data.

I hardly think Lucene is creating an isolationist culture, nor do we think our project is perfect.  What we do agree on is that our time is better spent on figuring out how to make Lucene better, not how to spend our time doing UNIX administration in a virtual server environment.  As I said before, we would love to do it, we just need a volunteer.  Since n8han is so concerned about us and our use of  Google in our little template search box that comes with Forrest and is turned on by default, maybe he would like to volunteer his time to set it up to use Nutch.  And don’t give the cop-out that you work on other projects, you don’t buy our “we have other things to do” argument, so why should we buy yours?

I’m all for criticism of Lucene.  I dish it out a fair amount, just read the java-dev mailing list (I try to do it in a constructive way), but I would rather it be about something that matters to Lucene, i.e how well it indexes and finds results or how we can improve _______, not about the fact that we don’t have a system administrator volunteering on the project.

This guy’s comment system wasn’t working at the moment, so I will leave my comment here. This won’t make much sense without reading the post first:

It’s funny you mention Wikipedia as an example, since they are running Lucene. As is Technorati and the Internet Archive. As is IBM Omnifind Yahoo! Edition. Are those big enough for you? If not, then choose any of them at http://wiki.apache.org/lucene-java/PoweredBy
And that list is just the companies who are public about it.

Speaking for myself (and not the ASF), as a Lucene developer, I would love to see us using it at Apache. It is something we are well aware of and have discussed. However, Lucene, like all Apache projects is VOLUNTEER and our volunteer infrastructure team is already loaded providing support to the actual products in terms of Subversion, JIRA, Confluence/MoinMoin, countless mailing lists, guarding against security attacks, creating new projects, etc. Simply put, it requires resources and time. Perhaps I can find some time between my day job and the volunteer work I do actually making the code better, supporting the community and occasionally administrating the nightly builds on our virtual servers, etc. to find time to deploy and maintain Nutch (which, mind you would do just fine for the job, just ask, aw never mind, we’ve been down that road) in a 24/7 high volume website. Even Google or your ISP has people working in operations to make sure even the most stable things are running and not being attacked/spammed/you name it, so Apache would be no different.

And, just so we are clear, every developer of Lucene “eats the Lucene/Nutch/Solr dog food”, we just don’t necessarily do it at Apache.org. I use it my day job. I use it in pet projects, I recommend it to clients, etc. I even use it in things that 5 years ago I would never have thought I would use it for (object stores, etc.) If that isn’t eating my own dog food, than I don’t know what dog food tastes like.

Finally, I don’t think our priority is to be squeaky clean. My personal one is to make sure Lucene is as good as it can be within my personal limitations. Just go look at our JIRA installation or our mailing lists to see all of the dirt. We aren’t hiding it.