Analysis of a deadlock

"[ACTIVE] ExecuteThread: '402' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.transaction.internal.ResourceCheckpoint@4e5c2e7c WAITING
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:474)
weblogic.transaction.internal.ResourceCheckpoint.blockingStore(ResourceCheckpoint.java:330)
weblogic.transaction.internal.XAResourceDescriptor.checkpointResourceDiscovery(XAResourceDescriptor.java:1102)
weblogic.transaction.internal.XAResourceDescriptor.refreshCheckpoint(XAResourceDescriptor.java:925)
weblogic.transaction.internal.ServerTransactionManagerImpl.checkpoint(ServerTransactionManagerImpl.java:1421)
weblogic.transaction.internal.ServerTransactionManagerImpl.wakeUp(ServerTransactionManagerImpl.java:1483)
weblogic.transaction.internal.WLSTimer.timerExpired(WLSTimer.java:35)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)


this thread is blocked waiting on a lock on a JDBCConnection

"[ACTIVE] ExecuteThread: '401' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.jdbcx.base.BaseXAConnection@2923410e BLOCKED
weblogic.jdbcx.base.BaseXAResource.end(Unknown Source)
weblogic.jdbc.jta.DataSource.end(DataSource.java:803)
weblogic.transaction.internal.XAServerResourceInfo.end(XAServerResourceInfo.java:1232)
weblogic.transaction.internal.XAServerResourceInfo.internalDelist(XAServerResourceInfo.java:404)
weblogic.transaction.internal.XAServerResourceInfo.delist(XAServerResourceInfo.java:326)
weblogic.transaction.internal.ServerTransactionImpl.delistAll(ServerTransactionImpl.java:1624)
weblogic.transaction.internal.ServerTransactionImpl.localRollback(ServerTransactionImpl.java:2012)
weblogic.transaction.internal.ServerTransactionImpl.globalRetryRollback(ServerTransactionImpl.java:3020)
weblogic.transaction.internal.ServerTransactionImpl.access$100(ServerTransactionImpl.java:66)
weblogic.transaction.internal.ServerTransactionImpl$1.run(ServerTransactionImpl.java:3252)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)



"[STANDBY] ExecuteThread: '400' for queue: 'weblogic.kernel.Default (self-tuning)'" waiting for lock weblogic.transaction.internal.ServerTransactionImpl@73937719 TIMED_WAITING
java.lang.Object.wait(Native Method)
weblogic.transaction.internal.ServerTransactionImpl.log(ServerTransactionImpl.java:1980)
weblogic.transaction.internal.ServerTransactionImpl.globalPrepare(ServerTransactionImpl.java:2291)
weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:266)
weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:227)
weblogic.jms.backend.BEForwardingConsumer.processMessages(BEForwardingConsumer.java:478)
weblogic.jms.backend.BEForwardingConsumer.pushMessages(BEForwardingConsumer.java:285)
weblogic.messaging.util.DeliveryList.run(DeliveryList.java:256)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)



enable JTA2PC, JTATLOG

either by passing:
-Dweblogic.debug.DebugJTA2PC=true
in the startWeblogic command,

or by using the Weblogic console (server/debug/weblogic/tlog and set DebugJTATLOG to true... No restarts are necessary)


or with WLST:

java weblogic.WLST
connect('weblogic', 'weblogic', 't3://myhost6360')
edit()
startEdit()
cd('/Servers/myserver/ServerDebug/myserver')
cmo.setDebugJTATLOG(true)
save()
activate()
disconnect()


read also this excellent article:
http://e-docs.bea.com/jrockit/geninfo/diagnos/thread_basics.html
http://edocs.bea.com/jrockit/docs142/userguide/apstkdmp.html
http://edocs.bea.com/jrockit/geninfo/diagnos/ctrlbreakhndlr.html on the jrcmd