Issue
Is there a solution or tool to recover damaged Solr/Lucene index in production ?
Environment
Solr
Resolution
Index corruption is generally very rare and might occur because of the following reasons:
1. Power outage - The system was running a "commit" and the server went down because of a power outage.
2. Process was killed while the index was being written to
3. Indexing concurrently into the same index by disabling index locking.
As mentioned the best approach would be to stop Solr and take a backup of the index.
Once backed up you can try the following command options on the index to:
1. Try to see if the index is really corrupt
2. Try to debug the cause of index corruption
3. Try to fix the damaged segments else, remove them.
4. Run CheckIndex to verify index
5. Restore the index (some documents will be gone) but hopefully a good part of the index should be recoverable.
Steps
1. Stop the Solr node
2. Backup the index directory if data is of paramount importance
3. Goto <Path_to_SolrInstall-X.Y.Z>/server/solr-webapp/webapp/WEB-INF/lib
You should see lucene-core-X.Y.Z.jar here:
Run the following command:
Syntax:
java -cp lucene-core-6.3.0.jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex <path_to_index_folder> -exorcise
Example:
java -cp lucene-core-6.3.0.jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex /Users/Rohit/Documents/SolrInstall/solr-6.3.0/example/cloud/node1/solr/gettingstarted_shard1_replica1/data/index -exorcise
This command will remove the corrupt segments from the index (so there might be some data loss as the corrupt segments are purged)
You will see an output similar to below:
-----------------------------------------------------
Opening index @ /Users/Rohit/Documents/SolrInstall/solr-6.3.0/example/cloud/node1/solr/gettingstarted_shard1_replica1/data/index
Segments file=segments_1 numSegments=0 id=1gbklfuh4fgecaa913367pgxy
No problems were detected with this index.
Took 0.077 sec total.