Modifying Zookeeper ensemble distribution without disruption

This specific use case involves how to move a 5-node ensemble from one set of machines to an entirely different set of machines, with no disruption to the use of the SolrCloud that it's supporting.

First, don't do this unless you absolutely have to.  If you can afford just a little downtime, it's far easier and quicker to just stage the new zookeeper instances with their configurations, shut down the old ones, copy the version-2 directory over, start up the new ones, and then recover the Solr nodes pointing to the new ensemble.

But if you absolutely have to do the modifications with no downtime, this should work.  Also, I'd suggest doing no updates to the SolrCloud while going through this process (both in regards to configuration and data), and doing it during low query traffic time. 

The primary obstacle is ensuring that one has a valid zk quarum running at all times, and that the Solr nodes are pointing to members of that valid quarum.  The general strategy is to grow from a 5 node zk ensemble, to a 7 node, then a 10 node.  Then point the Solr nodes to the 5 nodes of the 10 node ensemble that you want to keep long term.  Then shrink the ensemble back to 7, then (finally) the 5 node ensemble that you want to keep long term.  I'm outlining how to do it in a minimum number of steps, which will leave you with short periods of (for example) having only 4 nodes running on a 7 node system, which is just one network hiccup from losing the quarum.  So if you wish to be more cautious, you can modify this to include more steps (i.e. expand from 5, to 6, then 7, then 8, etc., etc.).  But if you understand how this example works, then you can figure out the other interim steps.

Starting point for this example is a 5 node zk ensemble, with a 2 node SolrCloud cluster.  The cloud start and stop scripts and the cfg files for each zk node are in the  I'm doing this example on a single machine, so the nodes are differentiated by port, real world use cases would differentiate by machine name, so you're details will be different.

0)  Before changing anything, be sure to be properly prepared.  This means getting the zookeeper software distributed to all the systems, making the appropriate directories for running zookeeper, creating the correct "myid" files, and distributing the cfg files (both zk and Solr) that will be used at the various steps.  The goal is to go through the process with just stops and starts of the zk and solr nodes, with little or no delay in modifying files along the way.

1) contains the zk cfg files for step 1.   They describe a 7 node zk ensemble using the original 5, plus 2 of the new ones.  Do a rolling restart of the zk nodes shifting them over to these new cfg files, starting with node1 of the original 5, then finishing with the 2 new nodes.  Give the ensemble about a couple of minutes between each node restart to copy over any needed files and elect a new leader for the quarum (if needed).

2) contains the zk cfg files for step 2.  They describe a 10 node zk ensemble utilizing the 5 old and all 5 new zk nodes.  Again, do a rolling restart starting with node 1 through 7, then finishing with the 3 new ones.  We now have a 10 node ensemble.

2.5)  Now modify your Solr node start scripts to point the solr nodes (which currently point to nodes1-5) to the last 5 zk nodes (nodes6-7).  And do a rolling restart of the nodes on the SolrCloud.

./uth/solr-4.10.1/bin/solr start -cloud -d node1 -p 7983 -z localhost:2181,localhost:2182,localhost:2183,localhost:2184,localhost:2195


./uth/solr-4.10.1/bin/solr start -cloud -d node1 -p 7983 -z localhost:2186,localhost:2187,localhost:2188,localhost:2189,localhost:2190

3) contains the zk cfg files for step 3.  They describe a 7 node zk ensemble utilizing 2 of the original 5, and all 5 new zk nodes.  First, shutdown 3 of the originals (nodes1-3), then do a rolling restart starting of the remaining 7.

4) contains the zk cfg files for step 4.  They describe the final 5 node ensemble that we wish to use moving forward, consisting of only the 5 new zk nodes.  Again, shutdown the last 2 of the originals (nodes4-5) first, then do a rolling restart starting of the remaining 5.

And we're finished.  A 5-node zookeeper ensemble, using only our 5 new nodes, and all SolrCloud nodes set-up to communicate with them.

Also attached is a raw notes text file, where I kept track of what I was trying to do, including 2 early (failed) attempts to go straight from 5 nodes to 10.  It led to getting a split zk ensemble, 1 quarum with 4 nodes, one with 6.  Not good. 



Have more questions? Submit a request


Please sign in to leave a comment.
Powered by Zendesk