Analyze a thread dump from the command line

Goal:

Sometimes we need to quickly analyze a thread dump via command line. Here are some helpful ways to do that.

Environment:

Java, Linux

Guide:

How many threads total?

cmd> grep 'nid=' <threaddumpfile> | wc -l

145

What threads are in RUNNABLE and what are they doing?

cmd> grep -A 1 'RUNNABLE' <threaddumpfile> | grep 'at ' | sort | uniq -c | sort -rn

  22 at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

   3 at java.net.PlainSocketImpl.socketAccept(Native Method)

   1 at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

   1 at java.net.SocketInputStream.socketRead0(Native Method)

What thread states are they in?

cmd> grep 'Thread.State' <threaddumpfile> | sort | uniq -c | sort -rn 

1555    java.lang.Thread.State: WAITING (parking)

  42    java.lang.Thread.State: RUNNABLE

  21    java.lang.Thread.State: TIMED_WAITING (parking)

   5    java.lang.Thread.State: TIMED_WAITING (sleeping)

   4    java.lang.Thread.State: TIMED_WAITING (on object monitor)

   2    java.lang.Thread.State: WAITING (on object monitor)

How many threads are locked on a certain object:

cmd> grep 'waiting to lock <' <threaddumpfile> | sort | uniq -c | sort -rn

   6 - waiting to lock <0x00000005215fab20> (a org.apache.solr.update.UpdateLog)

   5 - waiting to lock <0x0000000533e4c370> (a org.apache.solr.update.UpdateLog)

   5 - waiting to lock <0x000000053212c018> (a org.apache.solr.update.UpdateLog)

Getting the thread stack trace which is locking the top number of objects that are BLOCKED

cmd>  grep 'waiting to lock <' <threaddumpfile> | sort | uniq -c | sort -rn | head -1 | awk -v q="'" '{print q "locked "$6q }' | xargs -p -J % grep -C 15 % <threaddumpfile>

 at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)

 at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4412)

 at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4061)

 at org.apache.solr.update.SolrIndexWriter.merge(SolrIndexWriter.java:196)

 at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)

 at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)

"qtp428566321-543150" #543150 prio=5 os_prio=0 tid=0x00007faf3dd51800 nid=0x16592 waiting for monitor entry [0x00007faf12329000]

   java.lang.Thread.State: BLOCKED (on object monitor)

 at org.apache.solr.update.UpdateLog.add(UpdateLog.java:558)

 - locked <0x00000005215fab20> (a org.apache.solr.update.UpdateLog)

 at org.apache.solr.update.UpdateLog.add(UpdateLog.java:550)

 at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:366)

 at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:284)

 at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:234)

 at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)

 at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)

 at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:950)

 at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1168)

 - locked <0x0000000521cbdca0> (a org.apache.solr.update.VersionBucket)

 at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:633)

There are many variations on doing basically the same type of thing.