Scaling out Fusion to Multi-Node Cluster w/ Zookeeper Ensemble

This article details the process of taking a one-node Fusion installation, which runs the embedded Solr and Zookeeper processes, and scaling out to multiple Fusion installations and a three-node Zookeeper ensemble.

The example below assumes Fusion versions 2.x and higher as well as Zookeeper versions 3.4.6 and higher.

 

1.Bring down your Fusion instance.

bin/fusion stop

2. create your external zk ensemble. 

The following documentation details the process of setting up the Zookeeper Ensemble. 

From the Zookeeper Reference Guide:

https://zookeeper.apache.org/doc/r3.1.2/zookeeperAdmin.html#sc_zkMulitServerSetup

From the Solr Reference Guide:

https://cwiki.apache.org/confluence/display/solr/Setting+Up+an+External+ZooKeeper+Ensemble

Note: The Solr Reference guide example runs all 3 nodes on the same machine. In real world production environments, you'll be running 3 nodes scattered over 3 machines.

3. copy the $FUSION/data/zookeeper/version2 directory to each of the data directories of the new zookeeper instances

cp -r $FUSION/data/zookeeper/version2 $ZOOKEEPER_HOME/var/lib/zookeeperdata/


4. once you have configured the zookeeper ensemble appropriately, start each zookeeper instance and ensure quorum

bin/zkServer.sh start conf/zoo.cfg

 

5. update the connectString for the default search-cluster which lives in zookeeper

get the file from zk (it will appear on your desktop)

$FUSION_HOME/apps/solr-dist/server/scripts/cloud-scripts/zkcli.sh -z <ZOOKEEPER_HOST> -cmd getfile /lucid/search-clusters/default ~/Desktop/default.json

update the file

{"id" : "default",
  "connectString" : "<enter new zookeeper connect string here>",
  "zkClientTimeout" : 30000,
  "zkConnectTimeout" : 60000,
  "cloud" : true,
  "bufferFlushInterval" : 1000,
  "bufferSize" : 100,
  "concurrency" : 10,
  "validateCluster" : true}

push the file back to zk

$FUSION_HOME/apps/solr-dist/server/scripts/cloud-scripts/zkcli.sh -z <ZOOKEEPER_HOST> -cmd putfile /lucid/search-clusters/default ~/Desktop/default.json

6. change the conf/config.sh files for both Fusion nodes to point to the zookeeper ensemble. 

FUSION_ZK=zkhost1:2181,zkhost2:2181,zkhost3:2181
FUSION_SOLR_ZK=zkhost1:2181,zkhost2:2181,zkhost3:2181

7. in bin/fusion for both Fusion nodes, comment out any services which will not run on that node. (make sure to comment out all occurrences of zookeeper on both nodes)

ex: 

function do_start_all() {
#"$FUSION_HOME/bin/zookeeper" start
#sleep 10
"$FUSION_HOME/bin/solr" start
sleep 25
#"$FUSION_HOME/bin/spark-master" start
#"$FUSION_HOME/bin/spark-worker" start
"$FUSION_HOME/bin/api" start
sleep 5
#"$FUSION_HOME/bin/connectors" start
sleep 5
"$FUSION_HOME/bin/ui" start
sleep 5
}

8. start the original Fusion node. check solr admin UI and Fusion admin UI to make sure everything appears as it should.

bin/fusion start

9. start Solr on the 2nd node, make sure everything appears as it should

bin/solr start

10. start the rest of the Fusion services on the 2nd node

bin/fusion start

11. Use the collections API to ADDREPLICA for each of your collections


http://<host:port>/solr/admin/collections?action=ADDREPLICA&collection=collection&shard=shard&node=solr_node_name

12. Start searching!


 

 
 
 
Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk