multithreading - Java ExecuterService deadlock? -
recently wrote code read json data web , update mongodb entry according data recieved. has done 500,000 entries. accelerate process, use threadpoolexecutor 5 threads , blocking queue of size 50. furthermore use callerrunspolicy handling excess jobs. every worker job follows:
- read json data url
- parse json
- update corresponding mongodb entry
however programm gets stuck every time. early, after hours. far haven't found faults in code. exceptions handled not thrown , methods used supposed thread safe.
here thread dump after programm hangs:
2016-01-16 23:40:05 full thread dump java hotspot(tm) 64-bit server vm (25.66-b18 mixed mode): "jmx server connection timeout 711" #711 daemon prio=5 os_prio=0 tid=0x000000001813d000 nid=0x4e8 in object.wait() [0x000000004733f000] java.lang.thread.state: timed_waiting (on object monitor) @ java.lang.object.wait(native method) @ com.sun.jmx.remote.internal.servercommunicatoradmin$timeout.run(unknown source) - locked <0x00000000e26162c8> (a [i) @ java.lang.thread.run(unknown source) locked ownable synchronizers: - none "rmi scheduler(0)" #710 daemon prio=5 os_prio=0 tid=0x000000001813b800 nid=0x1ba0 waiting on condition [0x000000004723f000] java.lang.thread.state: timed_waiting (parking) @ sun.misc.unsafe.park(native method) - parking wait <0x00000000e1ce64e0> (a java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobject) @ java.util.concurrent.locks.locksupport.parknanos(unknown source) @ java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobject.awaitnanos(unknown source) @ java.util.concurrent.scheduledthreadpoolexecutor$delayedworkqueue.take(unknown source) @ java.util.concurrent.scheduledthreadpoolexecutor$delayedworkqueue.take(unknown source) @ java.util.concurrent.threadpoolexecutor.gettask(unknown source) @ java.util.concurrent.threadpoolexecutor.runworker(unknown source) @ java.util.concurrent.threadpoolexecutor$worker.run(unknown source) @ java.lang.thread.run(unknown source) locked ownable synchronizers: - none "rmi tcp connection(1)-192.168.56.1" #709 daemon prio=5 os_prio=0 tid=0x0000000017714000 nid=0x11f0 runnable [0x000000004713e000] java.lang.thread.state: runnable @ java.net.socketinputstream.socketread0(native method) @ java.net.socketinputstream.socketread(unknown source) @ java.net.socketinputstream.read(unknown source) @ java.net.socketinputstream.read(unknown source) @ java.io.bufferedinputstream.fill(unknown source) @ java.io.bufferedinputstream.read(unknown source) - locked <0x00000000e25e5fe8> (a java.io.bufferedinputstream) @ java.io.filterinputstream.read(unknown source) @ sun.rmi.transport.tcp.tcptransport.handlemessages(unknown source) @ sun.rmi.transport.tcp.tcptransport$connectionhandler.run0(unknown source) @ sun.rmi.transport.tcp.tcptransport$connectionhandler.lambda$run$256(unknown source) @ sun.rmi.transport.tcp.tcptransport$connectionhandler$$lambda$1/917377658.run(unknown source) @ java.security.accesscontroller.doprivileged(native method) @ sun.rmi.transport.tcp.tcptransport$connectionhandler.run(unknown source) @ java.util.concurrent.threadpoolexecutor.runworker(unknown source) @ java.util.concurrent.threadpoolexecutor$worker.run(unknown source) @ java.lang.thread.run(unknown source) locked ownable synchronizers: - <0x00000000e23ce048> (a java.util.concurrent.threadpoolexecutor$worker) "rmi tcp accept-0" #708 daemon prio=5 os_prio=0 tid=0x0000000017fa2000 nid=0xd18 runnable [0x0000000046f3e000] java.lang.thread.state: runnable @ java.net.dualstackplainsocketimpl.accept0(native method) @ java.net.dualstackplainsocketimpl.socketaccept(unknown source) @ java.net.abstractplainsocketimpl.accept(unknown source) @ java.net.plainsocketimpl.accept(unknown source) - locked <0x00000000e1ce8078> (a java.net.sockssocketimpl) @ java.net.serversocket.implaccept(unknown source) @ java.net.serversocket.accept(unknown source) @ sun.management.jmxremote.localrmiserversocketfactory$1.accept(unknown source) @ sun.rmi.transport.tcp.tcptransport$acceptloop.executeacceptloop(unknown source) @ sun.rmi.transport.tcp.tcptransport$acceptloop.run(unknown source) @ java.lang.thread.run(unknown source) locked ownable synchronizers: - none "pool-1-thread-5" #19 prio=5 os_prio=0 tid=0x000000001859e000 nid=0x628 waiting on condition [0x0000000018f9e000] java.lang.thread.state: waiting (parking) @ sun.misc.unsafe.park(native method) - parking wait <0x00000000a57af940> (a java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobject) @ java.util.concurrent.locks.locksupport.park(unknown source) @ java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobject.await(unknown source) @ java.util.concurrent.arrayblockingqueue.take(unknown source) @ java.util.concurrent.threadpoolexecutor.gettask(unknown source) @ java.util.concurrent.threadpoolexecutor.runworker(unknown source) @ java.util.concurrent.threadpoolexecutor$worker.run(unknown source) @ java.lang.thread.run(unknown source) locked ownable synchronizers: - none "pool-1-thread-4" #18 prio=5 os_prio=0 tid=0x000000001859d800 nid=0x2208 waiting on condition [0x0000000018e9f000] java.lang.thread.state: waiting (parking) @ sun.misc.unsafe.park(native method) - parking wait <0x00000000a57af940> (a java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobject) @ java.util.concurrent.locks.locksupport.park(unknown source) @ java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobject.await(unknown source) @ java.util.concurrent.arrayblockingqueue.take(unknown source) @ java.util.concurrent.threadpoolexecutor.gettask(unknown source) @ java.util.concurrent.threadpoolexecutor.runworker(unknown source) @ java.util.concurrent.threadpoolexecutor$worker.run(unknown source) @ java.lang.thread.run(unknown source) locked ownable synchronizers: - none "pool-1-thread-3" #17 prio=5 os_prio=0 tid=0x000000001859c800 nid=0x239c waiting on condition [0x0000000018d9e000] java.lang.thread.state: waiting (parking) @ sun.misc.unsafe.park(native method) - parking wait <0x00000000a57af940> (a java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobject) @ java.util.concurrent.locks.locksupport.park(unknown source) @ java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobject.await(unknown source) @ java.util.concurrent.arrayblockingqueue.take(unknown source) @ java.util.concurrent.threadpoolexecutor.gettask(unknown source) @ java.util.concurrent.threadpoolexecutor.runworker(unknown source) @ java.util.concurrent.threadpoolexecutor$worker.run(unknown source) @ java.lang.thread.run(unknown source) locked ownable synchronizers: - none "pool-1-thread-2" #16 prio=5 os_prio=0 tid=0x000000001859a000 nid=0x18a8 runnable [0x0000000018c9e000] java.lang.thread.state: runnable @ java.net.socketinputstream.socketread0(native method) @ java.net.socketinputstream.socketread(unknown source) @ java.net.socketinputstream.read(unknown source) @ java.net.socketinputstream.read(unknown source) @ java.io.bufferedinputstream.fill(unknown source) @ java.io.bufferedinputstream.read1(unknown source) @ java.io.bufferedinputstream.read(unknown source) - locked <0x00000000f921a080> (a java.io.bufferedinputstream) @ sun.net.www.http.httpclient.parsehttpheader(unknown source) @ sun.net.www.http.httpclient.parsehttp(unknown source) @ sun.net.www.protocol.http.httpurlconnection.getinputstream0(unknown source) - locked <0x00000000f921a0c0> (a sun.net.www.protocol.http.httpurlconnection) @ sun.net.www.protocol.http.httpurlconnection.getinputstream(unknown source) - locked <0x00000000f921a0c0> (a sun.net.www.protocol.http.httpurlconnection) @ java.net.url.openstream(unknown source) @ pr0grammdb.readurl(pr0grammdb.java:217) @ pr0grammdb.readjson(pr0grammdb.java:210) @ pr0grammdb$1job.run(pr0grammdb.java:98) @ java.util.concurrent.threadpoolexecutor.runworker(unknown source) @ java.util.concurrent.threadpoolexecutor$worker.run(unknown source) @ java.lang.thread.run(unknown source) locked ownable synchronizers: - <0x00000000a57afa50> (a java.util.concurrent.threadpoolexecutor$worker) "pool-1-thread-1" #15 prio=5 os_prio=0 tid=0x0000000018599000 nid=0x20fc waiting on condition [0x0000000018b9f000] java.lang.thread.state: waiting (parking) @ sun.misc.unsafe.park(native method) - parking wait <0x00000000a57af940> (a java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobject) @ java.util.concurrent.locks.locksupport.park(unknown source) @ java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobject.await(unknown source) @ java.util.concurrent.arrayblockingqueue.take(unknown source) @ java.util.concurrent.threadpoolexecutor.gettask(unknown source) @ java.util.concurrent.threadpoolexecutor.runworker(unknown source) @ java.util.concurrent.threadpoolexecutor$worker.run(unknown source) @ java.lang.thread.run(unknown source) locked ownable synchronizers: - none "pool-1-thread-1" #12 daemon prio=5 os_prio=0 tid=0x0000000017ecc800 nid=0x2bd8 waiting on condition [0x000000001879f000] java.lang.thread.state: timed_waiting (parking) @ sun.misc.unsafe.park(native method) - parking wait <0x00000000a57af298> (a java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobject) @ java.util.concurrent.locks.locksupport.parknanos(unknown source) @ java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobject.awaitnanos(unknown source) @ java.util.concurrent.scheduledthreadpoolexecutor$delayedworkqueue.take(unknown source) @ java.util.concurrent.scheduledthreadpoolexecutor$delayedworkqueue.take(unknown source) @ java.util.concurrent.threadpoolexecutor.gettask(unknown source) @ java.util.concurrent.threadpoolexecutor.runworker(unknown source) @ java.util.concurrent.threadpoolexecutor$worker.run(unknown source) @ java.lang.thread.run(unknown source) locked ownable synchronizers: - none "cluster-clusterid{value='569a6466a802c513644d5a51', description='null'}-127.0.0.1:27017" #11 daemon prio=5 os_prio=0 tid=0x000000001785e000 nid=0x1de8 waiting on condition [0x000000001869e000] java.lang.thread.state: timed_waiting (parking) @ sun.misc.unsafe.park(native method) - parking wait <0x00000000a57ad528> (a java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobject) @ java.util.concurrent.locks.locksupport.parknanos(unknown source) @ java.util.concurrent.locks.abstractqueuedsynchronizer$conditionobject.awaitnanos(unknown source) @ com.mongodb.connection.defaultservermonitor$servermonitorrunnable.waitforsignalortimeout(defaultservermonitor.java:237) @ com.mongodb.connection.defaultservermonitor$servermonitorrunnable.waitfornext(defaultservermonitor.java:218) @ com.mongodb.connection.defaultservermonitor$servermonitorrunnable.run(defaultservermonitor.java:167) - locked <0x00000000a57ad270> (a com.mongodb.connection.defaultservermonitor$servermonitorrunnable) @ java.lang.thread.run(unknown source) locked ownable synchronizers: - none "service thread" #9 daemon prio=9 os_prio=0 tid=0x0000000017493800 nid=0x23a8 runnable [0x0000000000000000] java.lang.thread.state: runnable locked ownable synchronizers: - none "c1 compilerthread2" #8 daemon prio=9 os_prio=2 tid=0x0000000015af8800 nid=0x2f5c waiting on condition [0x0000000000000000] java.lang.thread.state: runnable locked ownable synchronizers: - none "c2 compilerthread1" #7 daemon prio=9 os_prio=2 tid=0x0000000015af1800 nid=0x18c0 waiting on condition [0x0000000000000000] java.lang.thread.state: runnable locked ownable synchronizers: - none "c2 compilerthread0" #6 daemon prio=9 os_prio=2 tid=0x0000000015aef000 nid=0x15f8 waiting on condition [0x0000000000000000] java.lang.thread.state: runnable locked ownable synchronizers: - none "attach listener" #5 daemon prio=5 os_prio=2 tid=0x0000000015aec800 nid=0x2074 waiting on condition [0x0000000000000000] java.lang.thread.state: runnable locked ownable synchronizers: - none "signal dispatcher" #4 daemon prio=9 os_prio=2 tid=0x0000000015aec000 nid=0x2c94 runnable [0x0000000000000000] java.lang.thread.state: runnable locked ownable synchronizers: - none "finalizer" #3 daemon prio=8 os_prio=1 tid=0x00000000022b8000 nid=0x2f14 in object.wait() [0x0000000016e4f000] java.lang.thread.state: waiting (on object monitor) @ java.lang.object.wait(native method) - waiting on <0x00000000a57ebd08> (a java.lang.ref.referencequeue$lock) @ java.lang.ref.referencequeue.remove(unknown source) - locked <0x00000000a57ebd08> (a java.lang.ref.referencequeue$lock) @ java.lang.ref.referencequeue.remove(unknown source) @ java.lang.ref.finalizer$finalizerthread.run(unknown source) locked ownable synchronizers: - none "reference handler" #2 daemon prio=10 os_prio=2 tid=0x00000000022af000 nid=0x83c in object.wait() [0x0000000016d4f000] java.lang.thread.state: waiting (on object monitor) @ java.lang.object.wait(native method) - waiting on <0x00000000a57ebf20> (a java.lang.ref.reference$lock) @ java.lang.object.wait(unknown source) @ java.lang.ref.reference$referencehandler.run(unknown source) - locked <0x00000000a57ebf20> (a java.lang.ref.reference$lock) locked ownable synchronizers: - none "main" #1 prio=5 os_prio=0 tid=0x000000000207e000 nid=0x1964 runnable [0x000000000217e000] java.lang.thread.state: runnable @ java.net.socketinputstream.socketread0(native method) @ java.net.socketinputstream.socketread(unknown source) @ java.net.socketinputstream.read(unknown source) @ java.net.socketinputstream.read(unknown source) @ java.io.bufferedinputstream.fill(unknown source) @ java.io.bufferedinputstream.read1(unknown source) @ java.io.bufferedinputstream.read(unknown source) - locked <0x00000000f921c318> (a java.io.bufferedinputstream) @ sun.net.www.http.httpclient.parsehttpheader(unknown source) @ sun.net.www.http.httpclient.parsehttp(unknown source) @ sun.net.www.protocol.http.httpurlconnection.getinputstream0(unknown source) - locked <0x00000000f921c358> (a sun.net.www.protocol.http.httpurlconnection) @ sun.net.www.protocol.http.httpurlconnection.getinputstream(unknown source) - locked <0x00000000f921c358> (a sun.net.www.protocol.http.httpurlconnection) @ java.net.url.openstream(unknown source) @ pr0grammdb.readurl(pr0grammdb.java:217) @ pr0grammdb.readjson(pr0grammdb.java:210) @ pr0grammdb$1job.run(pr0grammdb.java:98) @ java.util.concurrent.threadpoolexecutor$callerrunspolicy.rejectedexecution(unknown source) @ java.util.concurrent.threadpoolexecutor.reject(unknown source) @ java.util.concurrent.threadpoolexecutor.execute(unknown source) @ pr0grammdb.addadditionaldata(pr0grammdb.java:114) @ pr0grammdb.main(pr0grammdb.java:48) locked ownable synchronizers: - none "vm thread" os_prio=2 tid=0x0000000015ab7000 nid=0x684 runnable "gc task thread#0 (parallelgc)" os_prio=0 tid=0x00000000021d7800 nid=0x28e0 runnable "gc task thread#1 (parallelgc)" os_prio=0 tid=0x00000000021d9000 nid=0x2658 runnable "gc task thread#2 (parallelgc)" os_prio=0 tid=0x00000000021da800 nid=0x273c runnable "gc task thread#3 (parallelgc)" os_prio=0 tid=0x00000000021dd000 nid=0x20f8 runnable "vm periodic task thread" os_prio=2 tid=0x00000000174a2000 nid=0x824 waiting on condition jni global references: 385 all threads except 1 seem waiting. don't know waiting for. other 1 seems stuck in socketread.
when code hangs heap usage weird, too. ascends 400mb before dropping 100mb , repeating. 1 cycle taking 1h.
maybe can me find problem?
the solution adding timeout urlconnection, there no standard timeout:
url url = new url(urlstring); urlconnection urlconn = url.openconnection(); urlconn.setconnecttimeout(10000); urlconn.setreadtimeout(10000); 
Comments
Post a Comment