Comments on: MySQL, Solr and “Communications link failure”

By: Glen Newton

Glen Newton — Fri, 21 Nov 2008 00:15:45 +0000

I ran in to the same problem with LuSql.

Originally I solved it with paging, taking something like 5000 records at a time.
However, for large ResultSets, MySQL starts taking longer and longer with each subsequent query as it seems to run through the records on the server end. After ~500,000 this starts being 10s of seconds…

I settled on counting the records as they go by, catching the exception when it is thrown, then re-issuing the original query, with a limit offset to the last count.

By: grant_ingersoll

grant_ingersoll — Fri, 18 Jul 2008 19:55:39 +0000

Bob, I am looking into the queue approach, as that seems most useful. Shalin, you are definitely right on connection timeouts, but this was on the ResultSet, so you’d have to reexecute the query (or automatically page, as Archie suggests) I think.

By: Shalin

Shalin — Fri, 18 Jul 2008 14:56:45 +0000

This is one of the reasons DataImportHandler re-opens connections inactive for more than 10 seconds because there is no JDBC driver-independent way of managing time-outs.

By: Bob Stewart

Bob Stewart — Thu, 17 Jul 2008 13:07:13 +0000

We use a queue (MSMQ) between database and Lucene, as well as between web crawlers and Lucene, that way indexing is asynchronous and you never get any such time out errors. Load up the queue from the database, and let SOLR index from the queue. You only need a small service to watch for new items in queue and submit to SOLR.

By: Archie

Archie — Wed, 16 Jul 2008 22:25:00 +0000

I avoid this by running my queries with limits while I reindex. So, you might query 100 rows at a time from your db, index them, then query the next 100 rows. You could also split up the indexing work across many machines with this approach then merge your index together after each separate process is finished. Or maybe use the new federated index features from solr. Happy indexing!