Goal
Sometimes we need to quickly analyze a thread dump via command line. Here are some helpful ways to do that.
Environment
Java, Linux
Guide
How many threads total?
cmd> grep 'nid=' <threaddumpfile> | wc -l
145
What threads are in RUNNABLE and what are they doing?
cmd> grep -A 1 'RUNNABLE' <threaddumpfile> | grep 'at ' | sort | uniq -c | sort -rn
22 at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
3 at java.net.PlainSocketImpl.socketAccept(Native Method)
1 at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
1 at java.net.SocketInputStream.socketRead0(Native Method)
What thread states are they in?
cmd> grep 'Thread.State' <threaddumpfile> | sort | uniq -c | sort -rn
1555 java.lang.Thread.State: WAITING (parking)
42 java.lang.Thread.State: RUNNABLE
21 java.lang.Thread.State: TIMED_WAITING (parking)
5 java.lang.Thread.State: TIMED_WAITING (sleeping)
4 java.lang.Thread.State: TIMED_WAITING (on object monitor)
2 java.lang.Thread.State: WAITING (on object monitor)
How many threads are locked on a certain object:
cmd> grep 'waiting to lock <' <threaddumpfile> | sort | uniq -c | sort -rn
6 - waiting to lock <0x00000005215fab20> (a org.apache.solr.update.UpdateLog)
5 - waiting to lock <0x0000000533e4c370> (a org.apache.solr.update.UpdateLog)
5 - waiting to lock <0x000000053212c018> (a org.apache.solr.update.UpdateLog)
Getting the thread stack trace which is locking the top number of objects that are BLOCKED
cmd> grep 'waiting to lock <' <threaddumpfile> | sort | uniq -c | sort -rn | head -1 | awk -v q="'" '{print q "locked "$6q }' | xargs -p -J % grep -C 15 % <threaddumpfile>
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4412)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4061)
at org.apache.solr.update.SolrIndexWriter.merge(SolrIndexWriter.java:196)
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
"qtp428566321-543150" #543150 prio=5 os_prio=0 tid=0x00007faf3dd51800 nid=0x16592 waiting for monitor entry [0x00007faf12329000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.solr.update.UpdateLog.add(UpdateLog.java:558)
- locked <0x00000005215fab20> (a org.apache.solr.update.UpdateLog)
at org.apache.solr.update.UpdateLog.add(UpdateLog.java:550)
at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:366)
at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:284)
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:234)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:950)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1168)
- locked <0x0000000521cbdca0> (a org.apache.solr.update.VersionBucket)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:633)
There are many variations on doing basically the same type of thing.