How to deal with Index Corruption

Symptoms

I feel like my index might be corrupted. I am getting stack traces in my logs and the Solr node is not starting up

Common stack traces:

 

1. org.apache.solr.core.SolrCore  - java.io.IOException: read past EOF
2. java.lang.RuntimeException: org.apache.lucene.index.CorruptIndexException 

 

Cause

A couple of common causes which lead to index corruption are -

1. Power outage - The system was running a "commit" and the server went down because of a power outage.

2. Indexing concurrently into the same index by disabling index locking

 

Resolution

Firstly take a backup of the index.

Now on the backup index, which is offline you can run the following command to see the index status: java -cp SOLR_HOME/example/webapps/WEB-INF/lib/lucene-core-[version].jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex /path/to/backup/data/index

 

If the output confirms that the index is corrupted you could keep the output of the following commands which would help us debug the cause for the index corruption - 

1. "ls -l" of the index directory

2. java -cp SOLR_HOME/example/webapps/WEB-INF/lib/lucene-core-[version].jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex /path/to/backup/data/index -verbose

 

To let CheckIndex 'fix' the index, which means it will remove the corrupted segments you need to run the following command on the original index while Solr is offline - 

For Solr / Lucene versions older than 5.X, try this,

java -cp SOLR_HOME/example/webapps/WEB-INF/lib/lucene-core-[version].jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex /path/to/data/index -fix

For Solr / Lucene versions newer than 5.X, try this,

java -cp SOLR_HOME/example/webapps/WEB-INF/lib/lucene-core-[version].jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex /path/to/data/index -exorcise

Once this has completed, you could restart Solr and be good to go. Although unfortunately there would be missing documents in the index because of the corruption.

 

Please keep the backup index in case anything goes wrong with the CheckIndex command or some extra information is required from the corrupted index.

 

 

 

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk