又一次Java线程卡死的调试经历

我的java代码，每天凌晨1点都会执行一个定时任务，定时任务中有循环执行的任务，每次循环都有开始和结束日志，某一天，我忽然发现，日志中前一天的循环只执行了几次就停止了，根据日志来看，执行到第4次循环的时候，只有开始的日志，没有结束的日志，那肯定是在里面卡死了

我首先执行

ps -ef | grep tomcat

获取到了线程的pid

然后执行jstack

jstack -l  10115 » dumpFile.o

获取到线程的dump日志

“scheduler-10” prio=10 tid=0x00007f6bf4004000 nid=0x7ece waiting on condition [0x00007f6be6ceb000]

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

• parking to wait for  <0x00000007b54809c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)

at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:744)

Locked ownable synchronizers:

• None

“DubboResponseTimeoutScanTimer” daemon prio=10 tid=0x00007f6c500b2800 nid=0x7ebd sleeping[0x00007f6ce953c000]

java.lang.Thread.State: TIMED_WAITING (sleeping)

at java.lang.Thread.sleep(Native Method)

at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture$RemotingInvocationTimeoutScan.run(DefaultFuture.java:300)

at java.lang.Thread.run(Thread.java:744)

Locked ownable synchronizers:

• None

“scheduler-9” prio=10 tid=0x00007f6bec135800 nid=0x7e1e waiting on condition [0x00007f6be5ddb000]

java.lang.Thread.State: TIMED_WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

• parking to wait for  <0x00000007b54809c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)

at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:744)

Locked ownable synchronizers:

• None

“scheduler-8” prio=10 tid=0x00007f6be8011000 nid=0x7e1d waiting on condition [0x00007f6be5edc000]

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

• parking to wait for  <0x00000007b54809c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)

at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:744)

Locked ownable synchronizers:

• None

“http-nio-10.137.12.23-9114-Acceptor-1” daemon prio=10 tid=0x00007f6dd4a02800 nid=0x7e1a runnable [0x00007f6be60df000]

java.lang.Thread.State: RUNNABLE

at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)

• locked <0x0000000779128e28> (a java.lang.Object)

at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:809)

at java.lang.Thread.run(Thread.java:744)

Locked ownable synchronizers:

• None

“http-nio-10.137.12.23-9114-Acceptor-0” daemon prio=10 tid=0x00007f6dd4a00800 nid=0x7e19 waiting for monitor entry [0x00007f6be61e0000]

java.lang.Thread.State: BLOCKED (on object monitor)

at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:225)

• waiting to lock <0x0000000779128e28> (a java.lang.Object)

at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:809)

at java.lang.Thread.run(Thread.java:744)

Locked ownable synchronizers:

• None

“http-nio-10.137.12.23-9114-ClientPoller-1” daemon prio=10 tid=0x00007f6dd49fe000 nid=0x7e18 runnable [0x00007f6be62e1000]

java.lang.Thread.State: RUNNABLE

at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

• locked <0x00000007b5404c68> (a sun.nio.ch.Util$2)

• locked <0x00000007b5404c78> (a java.util.Collections$UnmodifiableSet)

• locked <0x00000007b5404c20> (a sun.nio.ch.EPollSelectorImpl)

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1182)

at java.lang.Thread.run(Thread.java:744)

Locked ownable synchronizers:

• None

“http-nio-10.137.12.23-9114-ClientPoller-0” daemon prio=10 tid=0x00007f6dd49fc800 nid=0x7e17 runnable [0x00007f6be63e2000]

java.lang.Thread.State: RUNNABLE

at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)

at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)

at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)

at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)

• locked <0x00000007b5700198> (a sun.nio.ch.Util$2)

• locked <0x00000007b57001a8> (a java.util.Collections$UnmodifiableSet)

• locked <0x00000007b5700150> (a sun.nio.ch.EPollSelectorImpl)

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1182)

at java.lang.Thread.run(Thread.java:744)

Locked ownable synchronizers:

• None

“ContainerBackgroundProcessor[StandardEngine[Catalina]]” daemon prio=10 tid=0x00007f6dd49fb800 nid=0x7e16 waiting on condition [0x00007f6d50e4e000]

java.lang.Thread.State: TIMED_WAITING (sleeping)

at java.lang.Thread.sleep(Native Method)

at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1513)

at java.lang.Thread.run(Thread.java:744)

Locked ownable synchronizers:

• None

“scheduler-7” prio=10 tid=0x00007f6bf0001800 nid=0x7e15 waiting on condition [0x00007f6be64e3000]

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

• parking to wait for  <0x00000007b54809c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)

at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:744)

Locked ownable synchronizers:

• None

“scheduler-6” prio=10 tid=0x00007f6cf64de800 nid=0x7e14 waiting on condition [0x00007f6be65e4000]

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

• parking to wait for  <0x00000007b54809c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)

at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:744)

Locked ownable synchronizers:

• None

“scheduler-5” prio=10 tid=0x00007f6bf4001800 nid=0x7e13 waiting on condition [0x00007f6be66e5000]

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

• parking to wait for  <0x00000007b54809c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)

at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:744)

Locked ownable synchronizers:

• None

“scheduler-4” prio=10 tid=0x00007f6cf64dc800 nid=0x7e12 waiting on condition [0x00007f6be67e6000]

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

• parking to wait for  <0x00000007b54809c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)

at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:744)

Locked ownable synchronizers:

• None

“scheduler-3” prio=10 tid=0x00007f6cf64da800 nid=0x7e11 waiting on condition [0x00007f6be68e7000]

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

• parking to wait for  <0x00000007b54809c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)

at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:744)

Locked ownable synchronizers:

• None

“scheduler-2” prio=10 tid=0x00007f6cf64d8000 nid=0x7e10 runnable [0x00007f6be69e7000]

java.lang.Thread.State: RUNNABLE

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)

• locked <0x00000007c4ed9b40> (a java.net.SocksSocketImpl)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:579)

at org.apache.thrift.transport.TSocket.open(TSocket.java:178)

at com.sogou.baike.admin.service.MatchBaikeService.getMatchResult(MatchBaikeService.java:27)

at com.sogou.baike.admin.service.LemmaCompareService.markText(LemmaCompareService.java:617)

at com.sogou.baike.admin.service.LemmaCompareService.compareAbstract(LemmaCompareService.java:550)

at com.sogou.baike.admin.service.LemmaCompareService.compareLemma(LemmaCompareService.java:63)

at com.sogou.baike.admin.schedule.ParseBaiduHtmlSchedule.compareAndInsert(ParseBaiduHtmlSchedule.java:303)

at com.sogou.baike.admin.schedule.ParseBaiduHtmlSchedule.findAndCompare(ParseBaiduHtmlSchedule.java:228)

at com.sogou.baike.admin.schedule.ParseBaiduHtmlSchedule.startParseToLemma(ParseBaiduHtmlSchedule.java:162)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)

at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)

at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:744)

Locked ownable synchronizers:

• <0x00000007b5405620> (a java.util.concurrent.ThreadPoolExecutor$Worker)

“scheduler-1” prio=10 tid=0x00007f6cf64d6800 nid=0x7e0e waiting on condition [0x00007f6be6ae9000]

java.lang.Thread.State: WAITING (parking)

at sun.misc.Unsafe.park(Native Method)

• parking to wait for  <0x00000007b54809c8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)

at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)

at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:744)

Locked ownable synchronizers:

• None

scheduler-* 应该是我的定时任务的名称，可以看到他们都在等待一个条件的发生，当时我以为我发现了问题所在，我以为卡死的原因就是这个，于是各种查这个被等待的资源是什么，我想到了是不是线程池满了，而线程池中的任务一直占着资源不退出？导致新的任务卡死无法继续执行？是不是因为发生了deadlock死锁？可是分析了半天这个日志，也没有看出来哪里有死锁。

最后继续查了很多资料，说这个应该是定时任务线程池在等待新的定时任务来执行而已，那既然这样，那这个等待日志就是正常的了。

好吧，最后我采用了最原始的debug的方式，看究竟是卡死到哪里去了。

我一步步debug，最终定位到了一段调用thrift接口的代码中

原来那个接口提供方所在的机器已经被下掉了，而我这里又没有设置超时时间，所以一直连不上，就卡死在那了。

作者：田野上的希望

来源：CSDN

原文：https://blog.csdn.net/u011734144/article/details/62216288

版权声明：本文为博主原创文章，转载请附上博文链接！