We are interested in learning if there is a best practice for forcing Solr shards to shutdown after all in flight searches are completed. We have a mechanism to allow queries to be redirected to a different copy of the shard. What we would like to know if there is a way to have solr wait and finish all in flight queries before it shuts down. Currently, we are using a tail the log and watch approach to handle this and we would prefer to automate a solution if possible.
So, first of all, if you have it behind a loadbalancer I would take it out of the rotation to handle requests. Jetty actually handles this through the gracefulShutdown parameter in the jetty.xml file. The two relevant settings in the jetty.xml are:
So basically you can do a kill and it will wait for 1 sec (or whatever you set it to), for all requests to finish. New requests will not be serviced. Thus, if you have a loadbalancer which now is sending new requests to a different server, you will get a gracefulShutdown of up to 1 second for any remaining requests and any new requests in the meantime will get sent on the live server. By default, using solr (at least the version I have setup) it will come with the gracefulShutdown setting to be 1 second. LucidWorks does not contain this property but has it's own utility for starting/stopping the server using start.sh and stop.sh