Issue
This is a known issue related to Solr SOLR-14431 that affects at least down to 8.1.1 and is fixed in 8.6 that makes it so replicas can get out of sync with each other.
Environment
Solr 8.1.1 through 8.5+
Fixed: 8.6
Cause
This happens as operations "stall" due to the threads attempting to execute them blocking forever waiting for a ReentrantReadWriteLock in DefaultSolrCoreState that will never be released.
Further reading is here: SOLR-14431
Resolution
There is a workaround without having to upgrade. You can add the following configuration into the solrconfig.xml. By explicitly declaring the /admin/segments handler in solrconfig.xml with an invariants param that requests additional info, forcing it down a code path where it uses the IndexWriter, and decrements the ref-count, releasing the lock.
<requestHandler name="/admin/segments" class="solr.SegmentsInfoRequestHandler">
<!-- work around for https://issues.apache.org/jira/browse/SOLR-14431 -->
<lst name="invariants">
<bool name="coreInfo">true</bool>
</lst>
</requestHandler>