Scaling out Fusion to Multi-Node Cluster w/ Zookeeper Ensemble

This article details the process of taking a one-node Fusion installation, which runs the embedded Solr and Zookeeper processes, and scaling out to multiple Fusion installations and a three-node Zookeeper ensemble.

The example below assumes Fusion versions 2.x and higher as well as Zookeeper versions 3.4.6 and higher.


1.Bring down your Fusion instance.

bin/fusion stop

2. create your external zk ensemble. 

The following documentation details the process of setting up the Zookeeper Ensemble. 

From the Zookeeper Reference Guide:

From the Solr Reference Guide:

Note: The Solr Reference guide example runs all 3 nodes on the same machine. In real world production environments, you'll be running 3 nodes scattered over 3 machines.

3. copy the $FUSION/data/zookeeper/version2 directory to each of the data directories of the new zookeeper instances

cp -r $FUSION/data/zookeeper/version2 $ZOOKEEPER_HOME/var/lib/zookeeperdata/

4. once you have configured the zookeeper ensemble appropriately, start each zookeeper instance and ensure quorum

bin/ start conf/zoo.cfg


5. update the connectString for the default search-cluster which lives in zookeeper

get the file from zk (it will appear on your desktop)

$FUSION_HOME/apps/solr-dist/server/scripts/cloud-scripts/ -z <ZOOKEEPER_HOST> -cmd getfile /lucid/search-clusters/default ~/Desktop/default.json

update the file

{"id" : "default",
  "connectString" : "<enter new zookeeper connect string here>",
  "zkClientTimeout" : 30000,
  "zkConnectTimeout" : 60000,
  "cloud" : true,
  "bufferFlushInterval" : 1000,
  "bufferSize" : 100,
  "concurrency" : 10,
  "validateCluster" : true}

push the file back to zk

$FUSION_HOME/apps/solr-dist/server/scripts/cloud-scripts/ -z <ZOOKEEPER_HOST> -cmd putfile /lucid/search-clusters/default ~/Desktop/default.json

6. change the conf/ files for both Fusion nodes to point to the zookeeper ensemble. 


7. in bin/fusion for both Fusion nodes, comment out any services which will not run on that node. (make sure to comment out all occurrences of zookeeper on both nodes)


function do_start_all() {
#"$FUSION_HOME/bin/zookeeper" start
#sleep 10
"$FUSION_HOME/bin/solr" start
sleep 25
#"$FUSION_HOME/bin/spark-master" start
#"$FUSION_HOME/bin/spark-worker" start
"$FUSION_HOME/bin/api" start
sleep 5
#"$FUSION_HOME/bin/connectors" start
sleep 5
"$FUSION_HOME/bin/ui" start
sleep 5

8. start the original Fusion node. check solr admin UI and Fusion admin UI to make sure everything appears as it should.

bin/fusion start

9. start Solr on the 2nd node, make sure everything appears as it should

bin/solr start

10. start the rest of the Fusion services on the 2nd node

bin/fusion start

11. Use the collections API to ADDREPLICA for each of your collections


12. Start searching!


Have more questions? Submit a request


Please sign in to leave a comment.
Powered by Zendesk