How to invoke an operation with WLST and with Java

Jython:

from java.util import Hashtable
from javax.management import ObjectName
from javax.management.remote import JMXConnectorFactory
from javax.management.remote import JMXServiceURL

serviceURL= JMXServiceURL("t3", "yourserver", 7001, "/jndi/weblogic.management.mbeanservers.runtime")
h= Hashtable()
h.put("java.naming.security.principal", "weblogic")
h.put("java.naming.security.credentials", "weblogic")
h.put("jmx.remote.protocol.provider.pkgs", "weblogic.management.remote")
connector = JMXConnectorFactory.connect(serviceURL, h)
connection = connector.getMBeanServerConnection()

mbeanName= "com.bea:ServerRuntime=atlasadmin,Name=TimerService,ApplicationRuntime=Worklist_Console,Type=EJBTimerRuntime,EJBComponentRuntime=weblogic-timer-control.jar,StatelessEJBRuntime=TimerService"
mbeanObjName= ObjectName(mbeanName)
operationName= "activateDisabledTimers"
paramTypes = jarray.array([], Class.forName("java.lang.String"))
paramValues= jarray.array([], Class.forName("java.lang.Object"))
result=connection.invoke(mbeanObjName, operationName, paramValues, paramTypes);

print result



Java

import java.util.Hashtable;
import javax.management.MBeanServerConnection;
import javax.management.ObjectName;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;
import javax.naming.Context;

class Test {
    public static void main(String[] args) {
        Hashtable h = new Hashtable();
        h.put(Context.SECURITY_PRINCIPAL  , "weblogic");
        h.put(Context.SECURITY_CREDENTIALS, "weblogic");
        h.put(JMXConnectorFactory.PROTOCOL_PROVIDER_PACKAGES, "weblogic.management.remote");
        try {
            JMXServiceURL serviceURL= new JMXServiceURL("t3", "yourserver", 7001, "/jndi/weblogic.management.mbeanservers.runtime");
            JMXConnector connector = JMXConnectorFactory.connect(serviceURL, h);
            MBeanServerConnection connection = connector.getMBeanServerConnection();

            String mbeanName= "com.bea:ServerRuntime=atlasadmin,Name=TimerService,ApplicationRuntime=Worklist_Console,Type=EJBTimerRuntime,EJBComponentRuntime=weblogic-timer-control.jar,StatelessEJBRuntime=TimerService";
            ObjectName mbeanObjName= new ObjectName(mbeanName);
            String operationName= "activateDisabledTimers";
            String[] paramTypes= new String[] {};
            Object[] paramValues= new Object[] {};
            Object result=connection.invoke(mbeanObjName, operationName, paramValues, paramTypes);
            System.out.println("result="+result);
        } catch (Exception exc) {
            exc.printStackTrace();
        }
    }
}

Session replication - a performance killer

watch out for values in  Weblogic.xml and application.xml (the global EAR settings override the WAR default settings which is "memory")

Weblogic.xml
   PersistentStoreType   replicated_if_clustered
the values can be

• memory—Disables persistent session storage.
  

• file—Uses file-based persistence (See also PersistentStoreDir, above).
  

• jdbc—Uses a database to store persistent sessions. (see also PersistentStorePool, above).
  

• replicated—Same as memory, but session data is replicated across the clustered servers.
  

• cookie—All session data is stored in a cookie in the user's browser.
  

• replicated_if_clustered—If the Web application is deployed on a clustered server, the in-effect PersistentStoreType will be replicated. Otherwise, memory is the default.
  

Tuning the Stateless Beans pool

On WebLogic Admin console, click on "Deployments", "Monitoring", "EJB", "Stateless EJB", "Miss Total Count"

Provides a count of the total number of times a failed attempt was made to get an instance from the free pool.
An attempt to get a bean from the pool will fail if there are no available instances in the pool.


 This metric should be considered in relationship with "Access Total Count"; if the percentage is meaningful (say 1%) it's worth to increase the "max beans in pool" size in the Deployment Descriptor

The doctor is in: anamneses of a JRA analysis

The question is: once you get a JRA analysis (.JRA), what indicators do you look at to detect problems? What are the "rules of thumb" to detect if your application is in trouble, or it lies within reasonable limits?

Open the JRA with a JRMC (I am using 3.1)

"General" tab:
percentage of CPU and HEAP:
any heap on average above 80% is a clear indicator of low memory available
any CPU on average above 70% indicates a CPU constraint

"Memory" tab
GC statistics: OLD collections should be < 1/5th of the YOUNG Collections
Average time between YC should be > 10 s
All collections total pause time should be < 1/20th of the whole recording time

GC Call trees:
see if any single method triggers more than 5% of all GCs

GC Charts: look for patterns of dense brown (OC), they signal serious memory struggle

in Allocation, look for any single threads allocating large amount of RAM.

Code
Hot Methods: see that no single method takes more than 5% time. If it does, seek that no single stacktrace is responsible for more than 5% of that time.

Threads/Locks
in the Total Threads Count chart, look for times when the number of total threads jumps high, it's a sure signal of high contention
in JVM locks, look for a Lock Name which is much more utilised than others. Typically if the Class Library Cache if very high this means lots of classes dynamically loaded (probably)

Latency
I would recommend to keep always Latency Analysis ON, with an interval of 100ms in order not to overload the system.
Look at the Java Blocked Lock Class list, add the "total latency" column to the table and sort by the most relevant item. Right click, "set operative set", and check "show only Operative Set".
At this point, go to the Log tab, check "show only Operative Set", this will show you all the events blocking on the given lock, inclusive of stacktrace.
The red stripe on top will show you when the latency reaches a top: you can slide the lateral handles to zoom on the area.

The Graph tab is excellent to identify vertical patterns of color: if you see a vertical stripe of blue (Socket Read) or red (Java Blocked) it means that something is blocking a lot of threads on a single operation.

See also http://download.oracle.com/docs/cd/E13188_01/jrockit/docs142/usingJRA/looking.html

A simple (and boring) list of SOA standard

for those who like to show their stardards culture during interviews:

Security
SAML
WS-Federation
WS-Security

Reliable Messaging
WS-Reliability
WS-ReliableMessaging

Transaction
WS-Coordination
WS-AtomicTransaction
WS-BusinessActivity

Messaging
WS-Addressing
SOAP

Metadata
WS-Policy
WSDL

Service Registry and Discovery
WS-Discovery
UDDI

Transport
HTTP
HTTPS
TCP-IP
SMTP
JMS


XML
XML
XML Schema
XML Encryption
XML Digital Signature

BPEL 
BPEL4WS
WSCI
BPML
XPDL
CDL4WS


See also http://www.innoq.com/soa/ws-standards/poster/innoQ%20WS-Standards%20Poster%202007-02.pdf

weblogic.management.utils.InvalidCursorException: [Security:090250]Cursor not found MemberGroups15957487191

weblogic.management.utils.InvalidCursorException: [Security:090250] Cursor not found MemberGroups15957487191

make sure that the user weblogic is member of the same groups in the ldap-authenticator as it is in the default authenticator

Troubleshooting classloaders with JRockit

you have 3 options:

-Xverbose:class to detect when a class is being loaded

-Xnoclassgc to disable (use with care!) GC of unused classes

-Xverbose:classgc to trace when a CLASS is garbagecollected...

XMing: in the log I find client 4 rejected by IP x.y.z.w

add -ac to your XMing shortcut:

"C:\Program Files\Xming\Xming.exe" :0 -clipboard -multiwindow -ac

Above all, don't buy a Sony VAIO

It costed me a fortune, it's the worst laptop I ever had... the power supply inlet and power button fall apart, the 3 lateral USB ports get in your way, not to mention the pre-installed VISTA which is utterly UNBEARABLE.

Classloaders implementations in WebLogic

doing a latency analysis, we discover 3 classloaders in theJRA recording

weblogic.utils.classloaders.ChangeAwareClassLoader
sun.misc.Launcher$AppClassLoader
weblogic.utils.classloaders.GenericClassLoader 


what would be the difference between these 3 classloaders? When one is invoked rather than the other? Which one is most performant? No clue.... still investigating....

PoolDisabledSQLException, ResourceDisabledException

weblogic.jdbc.extensions.PoolDisabledSQLException: weblogic.common.resourcepool.ResourceDisabledException
 : Pool bla is disabled, cannot allocate resources to applications..Code d'erreur : 0

http://forums.oracle.com/forums/thread.jspa?threadID=962573&tstart=30

 

 A pool may disable itself after a configurable number
of consectutive failures to replace dead connections. Therafter it will periodically try
to reconnect to the DBMS and when it succeeds, the pool will re-enable itself.

The inner ugliness of beehive

when using WLI, lots of threads are in wait on a classloading:


[ACTIVE] ExecuteThread: '186' for queue: 'weblogic.kernel.Default (self-tuning)'
   weblogic.utils.classloaders.ChangeAwareClassLoader.loadClass(String)
   org.apache.beehive.netui.pageflow.internal.DefaultReloadableClassHandler.loadClass(String)
   org.apache.beehive.netui.pageflow.FlowControllerFactory.getFlowControllerClass(String)
   org.apache.beehive.netui.pageflow.PageFlowRequestProcessor.getFlowController(RequestContext, String)
   org.apache.beehive.netui.pageflow.PageFlowRequestProcessor.processInternal(HttpServletRequest, HttpServletResponse)
   org.apache.beehive.netui.pageflow.PageFlowRequestProcessor.process(HttpServletRequest, HttpServletResponse)
   org.apache.beehive.netui.pageflow.AutoRegisterActionServlet.process(HttpServletRequest, HttpServletResponse)
   org.apache.beehive.netui.pageflow.PageFlowActionServlet.process(HttpServletRequest, HttpServletResponse)
   org.apache.struts.action.ActionServlet.doPost(HttpServletRequest, HttpServletResponse)
   javax.servlet.http.HttpServlet.service(HttpServletRequest, HttpServletResponse)

this is the source:

http://svn.apache.org/repos/asf/beehive/trunk/netui/src/pageflow/org/apache/beehive/netui/pageflow/FlowControllerFactory.java

private transient ReloadableClassHandler _rch;

/**
     * Get a FlowController class.  By default, this loads the class using the thread context class loader.
     * 
     * @param className the name of the {@link FlowController} class to load.
     * @return the loaded {@link FlowController} class.
     * @throws ClassNotFoundException if the requested class could not be found.
     */
    public Class getFlowControllerClass( String className )
        throws ClassNotFoundException
    {
        return _rch.loadClass( className );
    }

ReloadableClassHandler :

http://svn.apache.org/repos/asf/beehive/trunk/netui/src/pageflow/org/apache/beehive/netui/pageflow/internal/DefaultReloadableClassHandler.java

public Class loadClass( String className )
        throws ClassNotFoundException
    {
        if ( _pageFlowClassLoader != null )
        {
            synchronized ( this )
            {
                return Class.forName(className, false, _pageFlowClassLoader);
            }
        }
        
        return Class.forName(className, false, DiscoveryUtils.getClassLoader());
}

ReloadableClassHandler has also a 

public Class loadCachedClass( String className )

which would save us from Class.forName (highly deprecated for performance reasons....)

I think a patch is needed, or some sort of hack...

Beehive is a loser, it has not been touched since 2006....

log4j: always use additivity=false

the logging code synchronizes on each category; you waste a lot of time - and risk to create serious contention - if you let every category to bubble up to "root".

Category.callAppenders :

  public void callAppenders(LoggingEvent event)   {
    int writes = 0;

    for (Category c = this; c != null; c = c.parent)    {
      synchronized (c) {
        if (c.aai != null)
          writes += c.aai.appendLoopOnAppenders(event);

        if (!(c.additive)) {
          monitorexit; break label67:
        }
      }
    }

    if (writes == 0)
      label67: this.repository.emitNoAppenderWarning(this);
  }

 (thanks to Fabien for this finding)

WLI Worklists, quick introduction

Worklist: interaction with humans, as opposed with JPDs which are fully automated.

Worklist User Portal is the GUI

Tasks are the main entity: they involve steps and actions.

WebLogic WebServices, a quick tutorial

JAX-WS = Java API for XML-based Web Services
is replacing JAX-RPC 

it supports SOAP 1.2,
JAXB 2.1(Java Architecture for XML Binding)
attachments with MTOM 
EJB 3.0 
WS-Security WSS 1.1


A Java Web Service (JWS) contains a @WebService annotation
@WebMethod for each method

@WebParam and @WebResult
@SOAPBinding

these annotations belong to the javax.jws package.

You use the jwsc Ant task to compile it




The WS is available at:
http://host:port/HelloWorldImpl/HelloWorldService?WSDL
HelloWorldImpl=contextPath 
HelloWorldService=serviceUri



You can create a Web Service from an existing WSDL file, "the golden WSDL". Use wsdlc to generate the JWS artifacts.

A WSDL is made of:
service has a port, qualified by an address and a 
binding (soap, http) which contains a portType, specifying the input and output parameters

message

all these parameters are expressed in the @WebService tag

ClientGenTask is a Ant task which "generates, from an existing WSDL file, the client component files that client applications use to invoke both WebLogic and non-WebLogic Web Services"
http://download.oracle.com/docs/cd/E13222_01/wls/docs92/webserv/anttasks.html#wp1039270


Different styles of binding are possible:
http://www.ibm.com/developerworks/webservices/library/ws-whichwsdl/

RPC literal, RPC encoded
Document literal, Document encoded


More on how to use annotations:

http://www.oracle.com/technology/tech/java/oc4j/1013/how_to/how-to-ws-annotations/doc/how-to-ws-annotations.html

Some useful tutorials:

http://netbeans.org/kb/trails/web.html

On the performance of JMS Messaging Bridges

from WebLogic manuals:

http://download.oracle.com/docs/cd/E13222_01/wls/docs92/bridge/design.html#wp1173277

When to Avoid using a Messaging Bridge

Environment with low tolerance for message latency. Messaging Bridges increase latency and may lower throughput. Messaging bridges increase latency for messages as they introduce an extra destination in the message path and may lower throughput because they forward messages using a single thread.
Forward messages between WebLogic 9.0 and higher domains—Use WebLogic Store-and-Forward.

and more:


http://download.oracle.com/docs/cd/E12840_01/wls/docs103/perform/bridgetuning.html

Avoid using a Messaging Bridge if remote destinations are already highly available. JMS clients can send directly to remote destinations. Use messaging bridge in situations where remote destinations are not highly available, such as an unreliable network or different maintenance schedules.
Use the better performing JMS SAF feature instead of using a Messaging Bridge when forwarding messages to remote destinations. In general, a JMS SAF agent is significantly faster than a Messaging Bridge. One exception is a configuration when sending messages in a non-persistent exactly-once mode.

and more:

When the Exactly-once quality of service is used, the bridge must undergo a two-phase commit with both JMS servers in order to ensure the transaction semantics and this operation can be very expensive. However, unlike the other qualities of service, the bridge can batch multiple operations together using Exactly-once service.

You may need to experiment with this parameter to get the best possible performance. For example, if the queue is not very busy or if non-persistent messages are used, Exactly-once batching may be of little benefit.

A quality of service of Exactly-once has a significant effect on bridge performance. The bridge starts a new transaction for each message and performs a two-phase commit across both JMS servers involved in the transaction. Since the two-phase commit is usually the most expensive part of the bridge transaction, as the number of messages being processed increases, the bridge performance tends to decrease.

 

Quick WLI tutorial

the nodes ina project can be:

Start
Finish
Client Request nodes: receive an event
Client Response
Decision nodes: branch execution
    Condition
Control Send nodes: communicate with services
Control Send with return
Control Receive nodes: wait for an asynchronous response from external services

Perform

Other types of node:
Decision
Switch
While do
Do While

For Each
Branch
Parallel
Event Choice
Group
Transaction


Projects:
Web: contains WebServices, XQuery, Transformations, Business Processes, Service Controls


A Process is defined by an annotation @Process with a long XML defining the structure of the process (nodes, branches etc)


Data Palette:
lists all  variables (XML or Non-XML, Java types); and all Controls (ServiceControl, FileControl...)



@com.bea.wli.common.XQuery  is a tag to define XQuery code

A Process implements a com.bea.jpd.ProcessDefinition Interface (marker Interface).

It generally contains member variables of type XmlObject or XmlObjectList; they store portion of the XML documents.

@com.bea.wli.jpd.Callback identifies callback points for asynchronous services.

Controls extend com.bea.control.ServiceControl, which manages credentials, endpoints, timeouts to make calls to Services.

Transformations implement com.bea.transform.TransformSource (marker Interface).

All processes (.jpd) contain a JpdContext (com.bea.wli.jpd.Context)

Transaction "JDBC Internal" , "JMS Internal", "Messaging Bridge"

In the WebLogic console, servers/monitoring/JTA/Transactions By Name, I see plenty of


Transaction "JDBC Internal", "JMS Internal" and "Messaging Bridge"

What does this "JDBC Internal" mean? An implicit transaction generated in the JDBC Driven when a SQL Statement is executed outside a transactional context?

What about "JMS Internal"?

What about "Messaging Bridge"?

If you know the answer, please let me know....

How to read a transaction error message

Don't quote me on this, this is only my interpretation.

We start with a general message, with times (important in case of timeout, to understand where the timeout originates)

<[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <> <1260325532180>
<BEA-010026>
<Exception occurred during commit of transaction Xid=BEA1-53B00171B6B92DDEE6DD(585181039),
Status=Rolledback.
[Reason=weblogic.transaction.internal.AppSetRollbackOnlyException],
numRepliesOwedMe=0,numRepliesOwedOthers=0,seconds since begin=60,seconds left=0,

then a section with all the XA resources enrolled in the XA TX:

XAServerResourceInfo[WLStore_PEPPODomain_PEPPOJMSStore_1]=
   (ServerResourceInfo[WLStore_PEPPODomain_PEPPOJMSStore_1]=
   (state=rolledback,assigned=PEPPOappli1),
   xar=WLStore_PEPPODomain_PEPPOJMSStore_1786968327,
   re-Registered = false),


XAServerResourceInfo[cgDataSource]=
    (ServerResourceInfo[cgDataSource]=(state=rolledback,assigned=PEPPOappli1),
       xar=cgDataSource,
       re-Registered = false),


XAServerResourceInfo[jdbc/INFOCOMejbJTSToplink]=(ServerResourceInfo[jdbc/INFOCOMejbJTSToplink]=(state=rolledback,assigned=LAPPAappli1),
  xar=null,
re-Registered = false),

then a list of the domains involved in the TX, in this case 2 Domains are involved:

SCInfo[PEPPODomain+PEPPOappli1]=(state=rolledback),
SCInfo[LAPPADomain+LAPPAappli1]=(state=rolledback),

properties=({}),

the CoordinatorURL tells us on which domain the TX was originated:

local properties=({class com.bea.wli.bpm.runtime.JpdContainer$TxnListener[10.1.5.123-52d2aa43.1256f924548.-7855]postNotificationCtrl:10.1.5.122-1bdb3347.1256f922aaa.-6f7c=com.bea.wli.bpm.runtime.JpdContainer$TxnListener@37e9159d, weblogic.jdbc.jta.cgDataSource=[ No XAConnection is attached to this TxInfo ], modifiedListeners=[weblogic.ejb.container.internal.TxManager$TxListener@22e14e03]}),OwnerTransactionManager=ServerTM[ServerCoordinatorDescriptor
=(CoordinatorURL=PEPPOappli1+frparmat01.acme.dns:8122+PEPPODomain+t3+,

XAResources is a list of all the transactional resources registered in the TX Coordinator:

XAResources={WLStore_PEPPODomain_PEPPOJMSStore_1, WLStore_PEPPODomain_pfProliferationJMSStore_auto_1, cgDataSource, OatmialResource, portalDataSourceAlwaysXA, eis/jms/WLSConnectionFactoryJNDIXA, PEPPO_LAPPAejbJTSToplink, PEPPO_UTILejbJTSToplink
, PEPPO_PEPPOejbJTSToplink, WLStore_PEPPODomain_cgJMSStore_auto_1, PEPPO_CCejbJTSToplink, WLStore_PEPPODomain__WLS_PEPPOappli1, WLStore_PEPPODomain_cgJMSStore_auto_3, bpmArchDataSource, portalDataSource, WLStore_PEPPODomain_WseeFileStore_auto_1, PEPPO_ACTejbJTSToplink, PEPPO_CISejbJTSToplink},NonXAResources={})],
CoordinatorURL=PEPPOappli1+frparmat01.acme.dns:8122+PEPPODomain+t3+):

finally, the stacktrace:

weblogic.transaction.RollbackException: Unknown reason
at weblogic.transaction.internal.TransactionImpl.throwRollbackException(TransactionImpl.java:1809)
at weblogic.transaction.internal.ServerTransactionImpl.internalCommit(ServerTransactionImpl.java:331)
at weblogic.transaction.internal.ServerTransactionImpl.commit(ServerTransactionImpl.java:227)
at weblogic.ejb.container.internal.MDListener.execute(MDListener.java:463)
at weblogic.ejb.container.internal.MDListener.transactionalOnMessage(MDListener.java:335)
at weblogic.ejb.container.internal.MDListener.onMessage(MDListener.java:291)
at weblogic.jms.client.JMSSession.onMessage(JMSSession.java:4072)
at weblogic.jms.client.JMSSession.execute(JMSSession.java:3962)
at weblogic.jms.client.JMSSession$UseForRunnable.run(JMSSession.java:4490)
at weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

Short tutorial on using Weblogic Diagnostics and watches

see
http://download.oracle.com/docs/cd/E12839_01/apirefs.1111/e13952/taskhelp/diagnostics/CreateWatchesForADiagnosticModule.html

the trick is: DO NOT FORGET TO TARGET YOUR WATCH to a server!
I have chosen the dummy criteria JRockitRuntimeMBean.Uptime > 1, which triggers immediately.
You should see in the log this (set the Severity and Log Watch Severity to error, just to be sure it does log)

<8 déc. 2009 18 h 35 CET> <Notice> <Diagnostics> <BEA-320068> <Watch 'sockets' with severity 'Error' on server 'myserver' has triggered at 8 déc. 2009 18 h 35 CET. Notification details:
WatchRuleType: Harvester
WatchRule: (${[weblogic.management.runtime.JRockitRuntimeMBean]//Uptime} >= 1)
WatchData: com.bea:Name=myserver,ServerRuntime=myserver,Type=JRockitRuntime//Uptime = 17420813
WatchAlarmType: AutomaticReset
WatchAlarmResetPeriod: 60000
>


You can use the JMS notification: use weblogic.jms.ConnectionFactory as Connection Factory for your alarm queue.

It will generate a MapMessage.

Performance Checklist

What would you check for, in a WebLogic server, to make sure the customer is taking full advantage of the performance tricks?
I will gather here all the new stuff I discover around. Work in progress.


- are you using Local JNDI Names for JMS destinations?

Thin locks, Fat Locks, contended thin lock etc

voir http://download.oracle.com/docs/cd/E13188_01/jrockit/docs142/usingJRA/applocks.html

A Thin Lock is one on which there is little contention, and only by spinning threads (the thread trying to acquire the lock is not suspended, but it keep polling the lock until it's available).
Such a Thin Lock with only Spinning Threads is called contended thin lock.

When there is more contention, a Thin Lock is promoted to Fat Lock, and a list is made with all threads waiting to acquire the lock.
Threads trying to acquire a fat lock go to sleep, and are called contended fat lock.

A Fat Lock might deflate to a Thin Lock if the JVM believes there is not much contention on the lock.

Typical options for a JRA recording

jrcmd $pid jrarecording nativesamples=false threaddumpinterval=60s latency=false filename=/users/dick/jra.xml.zip time=300


lock analysis is enabled by passing -Djrockit.lockprofiling=true as a parameter to the JVM

the entire help can be obtained like this:

 jrcmd 3863 help startjrarecording
3863:
Starts a JRA recording.
        filename           - name of the file to store JRA recording to
                             (string, recording.jra)
        recordingtime      - length of the recording in seconds (int, 60)
        delay              - delay before starting recording in seconds (int,
                             0)
        methodsampling     - enable method sampling (bool, true)
        gcsampling         - enable gc information (bool, true)
        heapstats          - include heap statistics (bool, true)
        nativesamples      - include native code in sampling (bool, false)
        methodtraces       - include stack traces (bool, true)
        tracedepth         - depth of stack traces (int, 16)
        sampletime         - time between samples in milliseconds (int, 30)
        zip                - zip the recording (bool, true)
        hwsampling         - use hardware sampling if possible (bool, false)
        threaddump         - do full threaddumps at start and end of recording
                             (bool, true)
        threaddumpinterval - also do threaddumps every 'n' interval (can be
                             specified as x[ns|ms|s]) (time, 0s)
        latency            - include latency analysis (bool, false)
        latencythreshold   - do not record events shorter than this number (can
                             be specified as x[ns|ms|s]) (time, 20ms)
        cpusamples         - sample cpu usage during the recording (bool,
                             true)
        cpusampleinterval  - cpu sample interval (can be specified as
                             x[ns|ms|s]) (time, 1s)


unfortunately, latency=true generates a LOT of data, so you can't do a recording longer than 300 s. Try using  latencythreshold=200ms   

JRockit and Lock Profiling

http://download.oracle.com/docs/cd/E13188_01/jrockit/docs142/usingJRA/looking.html#1056684


Lock Profiling

You can enable the JRockit Runtime Analyzer to collect and analyze information about the locks
and the contention that has occurred while the runtime analyzer was recording. To do this, add
the following option when you start your application:


-Djrockit.lockprofiling=true


When lock profiling has been enabled, you can view information about Java locks on the Lock
Profiling tab in Mission Control.


You can dump at any time the lock profiling:

jrcmd 3863 lockprofile_print

Class, Lazy Banned, Thin Uncontended, Thin Contended, Lazy Reservation, Lazy lock, Lazy Reverted, Lazy Coop-Reverted, Thin Recursive, Fat Uncontended, Fat Contended, Fat Recursive, Fat Contended Sleep, Reserve Bit Uncontended, Reserve Bit Contended
[I, false, 546, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
java/util/WeakHashMap$Entry, false, 266, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
weblogic/timers/internal/TimerManagerImpl, false, 4398, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
sun/text/resources/DateFormatZoneData, false, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0


The alternative is:
you examine the thread dumps, looking for a pattern like:

-- Blocked trying to get lock: oracle/toplink/sessions/DatabaseLogin@0x2b9dd61a9968[fat lock]

The target of your search should be the [fat lock] with a number of locks on the same monitor (oracle/toplink/sessions/DatabaseLogin@0x2b9dd61a9968)

Plotting the number of threads waiting on the monitor, against the time of the thread dump, can prove whether a lock is really problematic.


About JRA Overhead when Recording
The overhead while recording is very low—typically less than two percent. If you enable Java lock profiling with the system property – Djrockit.lockprofiling=true the overhead could be considerable greater, typically between 3% and 25% depending upon the application.

More on the topic:

http://download.oracle.com/docs/cd/E13188_01/jrockit/docs142/usingJRA/applocks.html

Quick WLST tutorial

java weblogic.WLST

connect('weblogic', 'weblogic', "t3://localhost:7001")

ls()

dr-- atlasadmin
dr-- atlasappli1
dr-- atlasappli2

cd('atlasappli2')

ls()

dr-- COM
dr-- CandidateMachines
dr-- Cluster
dr-- DefaultFileStore
dr-- ExecuteQueues
dr-- FederationServices
dr-- IIOP
dr-- JTAMigratableTarget
dr-- Log
dr-- Machine
dr-- NetworkAccessPoints
dr-- OverloadProtection
dr-- ReliableDeliveryPolicy
dr-- SSL
dr-- ServerDiagnosticConfig
dr-- ServerStart
dr-- WebServer
dr-- WebService
dr-- XMLEntityCache
dr-- XMLRegistry

-r-- AcceptBacklog 50
-r-- AdminReconnectIntervalSeconds 10
-r-- AdministrationPort 9002
-r-- AdministrationProtocol t3s
-r-- AutoKillIfFailed false
-r-- AutoMigrationEnabled false
-r-- AutoRestart true
-r-- COMEnabled false
-r-- ClasspathServletDisabled false
-r-- ClientCertProxyEnabled false
-r-- ClusterRuntime null
-r-- ClusterWeight 100
-r-- CompleteCOMMessageTimeout -1
-r-- CompleteHTTPMessageTimeout -1
-r-- CompleteIIOPMessageTimeout -1
-r-- CompleteMessageTimeout 60
-r-- CompleteT3MessageTimeout -1
-r-- CustomIdentityKeyStoreFileName null
-r-- CustomIdentityKeyStorePassPhrase ******
-r-- CustomIdentityKeyStorePassPhraseEncrypted ******
-r-- CustomIdentityKeyStoreType null
-r-- CustomTrustKeyStoreFileName null
-r-- CustomTrustKeyStorePassPhrase ******
-r-- CustomTrustKeyStorePassPhraseEncrypted ******
-r-- CustomTrustKeyStoreType null
-r-- DGCIdlePeriodsUntilTimeout 5
-r-- DefaultIIOPPassword ******
-r-- DefaultIIOPPasswordEncrypted ******
-r-- DefaultIIOPUser weblogic
-r-- DefaultInternalServletsDisabled false
-r-- DefaultProtocol t3
-r-- DefaultSecureProtocol t3s
-r-- DefaultTGIOPPassword ******
-r-- DefaultTGIOPPasswordEncrypted ******
-r-- DefaultTGIOPUser guest
-r-- ExternalDNSName null
-r-- ExtraEjbcOptions null
-r-- ExtraRmicOptions null
-r-- GracefulShutdownTimeout 0
-r-- HealthCheckIntervalSeconds 180
-r-- HealthCheckStartDelaySeconds 120
-r-- HealthCheckTimeoutSeconds 60
-r-- HostsMigratableServices true
-r-- HttpTraceSupportEnabled false
-r-- HttpdEnabled true
-r-- IIOPEnabled true
-r-- IIOPTxMechanism ots
-r-- IdleConnectionTimeout 65
-r-- IdleIIOPConnectionTimeout -1
-r-- IdlePeriodsUntilTimeout 4
-r-- IgnoreSessionsDuringShutdown false
-r-- InstrumentStackTraceEnabled true
-r-- InterfaceAddress null
-r-- JDBCLLRTableName null
-r-- JDBCLoggingEnabled false
-r-- JDBCLoginTimeoutSeconds 0
-r-- JMSDefaultConnectionFactoriesEnabled true
-r-- JNDITransportableObjectFactoryList null
-r-- JavaCompiler javac
-r-- JavaCompilerPostClassPath null
-r-- JavaCompilerPreClassPath null
-r-- JavaStandardTrustKeyStorePassPhrase ******
-r-- JavaStandardTrustKeyStorePassPhraseEncrypted ******
-r-- KeyStores DemoIdentityAndDemoTrust
-r-- ListenAddress server02.acme.dns
-r-- ListenDelaySecs 0
-r-- ListenPort 8121
-r-- ListenPortEnabled true
-r-- ListenThreadStartDelaySecs 60
-r-- ListenersBindEarly false
-r-- LogRemoteExceptionsEnabled false
-r-- LoginTimeoutMillis 5000
-r-- LowMemoryGCThreshold 5
-r-- LowMemoryGranularityLevel 5
-r-- LowMemorySampleSize 10
-r-- LowMemoryTimeInterval 3600
-r-- MSIFileReplicationEnabled false
-r-- Machine null
-r-- ManagedServerIndependenceEnabled true
-r-- MaxCOMMessageSize -1
-r-- MaxHTTPMessageSize -1
-r-- MaxIIOPMessageSize -1
-r-- MaxMessageSize 10000000
-r-- MaxOpenSockCount -1
-r-- MaxT3MessageSize -1
-r-- MessageIdPrefixEnabled false
-r-- MessagingBridgeThreadPoolSize 5
-r-- MuxerClass null
-r-- Name atlasappli2
-r-- NativeIOEnabled true
-r-- Notes null
-r-- OutboundEnabled false
-r-- OutboundPrivateKeyEnabled false
-r-- PeriodLength 60000
-r-- PreferredSecondaryGroup null
-r-- ReplicationGroup null
-r-- RestartDelaySeconds 0
-r-- RestartIntervalSeconds 3600
-r-- RestartMax 2
-r-- ReverseDNSAllowed false
-r-- ServerLifeCycleTimeoutVal 120
-r-- ServerVersion unknown
-r-- SocketBufferSizeAsChunkSize false
-r-- SocketReaders -1
-r-- StagingDirectoryName /acme/serveur/wlsapp/ATLASDomain/servers/atlasappli2/stage
-r-- StagingMode stage
-r-- StartupMode RUNNING
-r-- StartupTimeout 0
-r-- StuckThreadMaxTime 600
-r-- StuckThreadTimerInterval 60
-r-- SystemPasswordEncrypted ******
-r-- TGIOPEnabled true
-r-- ThreadPoolPercentSocketReaders 33
-r-- TransactionLogFilePrefix ./
-r-- TransactionLogFileWritePolicy Direct-Write
-r-- TunnelingClientPingSecs 45
-r-- TunnelingClientTimeoutSecs 40
-r-- TunnelingEnabled false
-r-- Type Server
-r-- UploadDirectoryName ./servers/atlasappli2/upload
-r-- Use81StyleExecuteQueues false
-r-- VerboseEJBDeploymentEnabled false
-r-- WeblogicPluginEnabled false
-r-- XMLEntityCache null
-r-- XMLRegistry null

-r-x freezeCurrentValue Void : String(attributeName)
-r-x isSet Boolean : String(propertyName)
-r-x synchronousKill String :
-r-x synchronousStart String :
-r-x unSet Void : String(propertyName)


cd('ServerRuntimes')
ls()
dr--   atlasadmin
dr--   atlasappli1
 
cd('atlasappli1')
ls()




dr--   ApplicationRuntimes
dr--   ClusterRuntime
dr--   ConnectorServiceRuntime
dr--   DefaultExecuteQueueRuntime
dr--   EntityCacheCumulativeRuntime
dr--   EntityCacheCurrentStateRuntime
dr--   EntityCacheHistoricalRuntime
dr--   ExecuteQueueRuntimes
dr--   JDBCServiceRuntime
dr--   JMSRuntime
dr--   JTARuntime
dr--   JVMRuntime
dr--   JoltRuntime
dr--   LibraryRuntimes
dr--   LogBroadcasterRuntime
dr--   LogRuntime
dr--   MANReplicationRuntime
dr--   MailSessionRuntimes
dr--   MaxThreadsConstraintRuntimes
dr--   MinThreadsConstraintRuntimes
dr--   PathServiceRuntime
dr--   PersistentStoreRuntimes
dr--   RequestClassRuntimes
dr--   SAFRuntime
dr--   ServerChannelRuntimes
dr--   ServerSecurityRuntime
dr--   ThreadPoolRuntime
dr--   TimerRuntime
dr--   WANReplicationRuntime
dr--   WLDFRuntime
dr--   WLECConnectionServiceRuntime
dr--   WTCRuntime
dr--   WebServerRuntimes
dr--   WorkManagerRuntimes

-r--   ActivationTime                               1259864408758
-r--   AdminServer                                  false
-r--   AdminServerHost                              10.1.5.96
-r--   AdminServerListenPort                        8501
-r--   AdminServerListenPortSecure                  false
-r--   AdministrationPort                           9002
-r--   AdministrationPortEnabled                    false
-r--   AdministrationURL                            t3://frparmat01.coface.dns:8121
-r--   CurrentDirectory                             /coface/serveur/wlsapp/ATLASDomain/.
-rw-   CurrentMachine
-r--   DefaultExecuteQueueRuntime                   null
-r--   DefaultURL                                   t3://frparmat01.coface.dns:8121
-r--   EntityCacheCumulativeRuntime                 null
-r--   EntityCacheCurrentStateRuntime               null
-r--   EntityCacheHistoricalRuntime                 null
-r--   HealthState                                  State:HEALTH_OK,ReasonCode:[]
-r--   JoltRuntime                                  null
-r--   ListenAddress                                frparmat01.coface.dns/10.1.5.232
-r--   ListenPort                                   8121

-r--   ListenPortEnabled                            true
-r--   MANReplicationRuntime                        null
-r--   Name                                         atlasappli1
-r--   OpenSocketsCurrentCount                      10783
-rw-   Parent                                       null
-r--   PathServiceRuntime                           null
-r--   RestartRequired                              false
-r--   RestartsTotalCount                           0
-r--   SSLListenAddress                             null
-r--   SSLListenPort                                7002
-r--   SSLListenPortEnabled                         false
-r--   ServerClasspath                              /opt/jrmc-3.1.2-1.5.0/jre/lib/amd64/jrockit/jrockit1.5.0.jar:blablabla
-r--   SocketsOpenedTotalCount                      10783
-r--   State                                        RUNNING
-r--   StateVal                                     2
-r--   Type                                         ServerRuntime
-r--   WANReplicationRuntime                        null
-r--   WLECConnectionServiceRuntime                 null
-r--   WeblogicVersion                              WebLogic Server Temporary Patch for CR327368, CR351430 Fri Nov 09 16:31:48 EST 2007
WebLogic Server Temporary Patch for CR312139, CR358848 Thu Jan 24 10:59:46 PST 2008
WebLogic Server Temporary Patch for CR292910, CR326720, CR367941 Thu Apr 17 09:46:54 PDT 2008
WebLogic Server Temporary Patch for CR334765 Fri Sep 07 16:12:12 PDT 2007
blablabla
WebLogic Server 9.2 MP2  Mon Jun 25 01:32:01 EDT 2007 952826

-r-x   addRequestClassRuntime                       Boolean : WebLogicMBean(weblogic.management.runtime.RequestClassRuntimeMBean)
-r-x   forceShutdown                                Void :
-r-x   forceSuspend                                 Void :
-r-x   getServerChannel                             java.net.InetSocketAddress : String(protocol)
-r-x   getURL                                       String : String(protocol)
-r-x   restartSSLChannels                           Void :
-r-x   resume                                       Void :
-r-x   shutdown                                     Void :
-r-x   shutdown                                     Void : Integer(timeout),Boolean(ignoreSessions)
-r-x   start                                        Void :
-r-x   suspend                                      Void :
-r-x   suspend                                      Void : Integer(timeout),Boolean(ignoreSessions)








see also http://download.oracle.com/docs/cd/E11035_01/wls100/config_scripting/nav_edit.html






 

WLDF doesn't work in my Firefox, java.lang.NumberFormatException: For input string: "0,18"

Java Plug-in 1.6.0_16
Utilisation de la version JRE 1.6.0_16-b01 Java HotSpot(TM) Client VM
Répertoire d'accueil de l'utilisateur = C:\Documents and Settings\ezdine_ould-mohamed
----------------------------------------------------
c: effacer la fenêtre de la console
f: finaliser les objets de la file d'attente de finalisation
g: libérer la mémoire
h: afficher ce message d'aide
l: vider la liste des chargeurs de classes
m: imprimer le relevé d'utilisation de la mémoire
o: déclencher la consignation
q: masquer la console
r: recharger la configuration des politiques
s: vider les propriétés système et déploiement
t: vider la liste des threads
v: vider la pile des threads
x: effacer le cache de chargeurs de classes
0-5: fixer le niveau de traçage à
----------------------------------------------------


java.lang.NumberFormatException: For input string: "0,18"
at sun.misc.FloatingDecimal.readJavaFormatString(Unknown Source)
at java.lang.Double.parseDouble(Unknown Source)
at com.bea.diagnostics.dashboard.WLDFSlider$LocalSlider.calcTransMin(WLDFSlider.java:99)
at com.bea.diagnostics.dashboard.WLDFSlider$LocalSlider.setMinimum(WLDFSlider.java:94)
at com.bea.diagnostics.dashboard.WLDFSlider.setMinimum(WLDFSlider.java:373)
at com.bea.diagnostics.dashboard.GlobalPropPanel.(GlobalPropPanel.java:75)
at com.bea.diagnostics.dashboard.PropertyPanel$GlobalPropPanelWrapper.(PropertyPanel.java:167)
at com.bea.diagnostics.dashboard.PropertyPanel.(PropertyPanel.java:49)
at com.bea.diagnostics.dashboard.ChartPanel.addBrowsers(ChartPanel.java:678)
at com.bea.diagnostics.dashboard.ChartPanel.execute(ChartPanel.java:532)
at com.bea.diagnostics.dashboard.ChartPanel.(ChartPanel.java:181)
at com.bea.diagnostics.dashboard.ChartPanelApplet.init(ChartPanelApplet.java:25)
at sun.plugin2.applet.Plugin2Manager$AppletExecutionRunnable.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
java.lang.NullPointerException
at com.bea.diagnostics.dashboard.ChartPanelApplet.start(ChartPanelApplet.java:40)
at sun.plugin2.applet.Plugin2Manager$AppletExecutionRunnable.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Exception : java.lang.NullPointerException


http://forums.oracle.com/forums/thread.jspa?messageID=2965757


My problem was related to my pc language settings: I've solved it setting english language
This issue was related to a locale problem. It has been fixed in WLS 10.0 and will be back-ported to 9.2.
You can use : -Duser.language=en -Duser.region=US as JRE local param to force US setting.

Windows Config Panel/Java => Java Configuration
Tab Java, Applet Java runtime, display.
Set "-Duser.language=en -Duser.region=US" in exécution param for your JRE

Back to WSDF applet : It's work.



The operation is a bit tricky, you must validate twice before the parameters are written permanently.

WLI quick tutorial: ADMIN

login to
http://localhost:7001/wliconsole

Go to Process Instance Statistics.

A Process has a Service URI (blabla/processname.JPD)
Can be Stateful or Stateless


Each process can have a different Tracking Level, by default it's "default", that is the Sysyem tracking level. Do set the tracking level to minimum for least impact on performance

With SLA you can set off alarms when a process is too slow.

Versioning allows you to have many versions of a Process available, one only is active at any time.


With System Configuration : Purge you can remove all info about Completed/terminated processes. You can schedule it every few days, or run it on command.

Quick example on how to grep for Oracle errors using Java

package com.acme.logs;

import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class MatcherTest {
public static void main(String[] args) {

Pattern pattern = Pattern.compile("ORA-[0-9]*");
Matcher matcher = pattern.matcher("ciao ORA-11234 pippo");
boolean found = false;
while (matcher.find()) {
System.out.println("I found the text " + matcher.group() + " starting at " +
"index " + matcher.start() + " and ending at index " + 
matcher.end());
found = true;
}
if(!found){
System.out.println("No match found.%n");
}

}
}

I found the text ORA-11234 starting at index 5 and ending at index 14

Excellent practical tutorial on Parsing XML in Java

several techniques explained, with examples.

http://www.vogella.de/articles/JavaXML/article.html

I keep thinking that XMLBeans is the best way (at least the code is readable), but it requires regenerating all the Java code each time you change the XSD associated to the XMLs...

Careful with "pinned to thread" on the JDBC Connection Pool

it could allow the creation of JDBC connection above the "max capacity" assigned to the pool

The infamous BEA-000627

http://download.oracle.com/docs/cd/E13222_01/wls/docs100/messages/Common.html


Info: Reached maximum capacity of pool "pool", making "newVal" new resource instances instead of "oldVal".

Description
Hit configured maximum capacity of pool while making additional resources for the pool. Therefore, the number of resources being made has been increased.

Cause
Informational message.

Action
No action required.



I would expect to get a

####2 déc. 2009 04 h 10 CET Error WLI-Worklist peppe.acme.dns atlasappli2 [ACTIVE] ExecuteThread: '247' for queue: 'weblogic.kernel.Default
(self-tuning)' X5229701 1259723458590 BEA-493028 Internal error: unexpected ejb exception: java.sql.SQLException: Internal error: Cannot obtain
XAConnection weblogic.common.resourcepool.ResourceLimitException: No resources currently available in pool cgDataSource to allocate to applications, please in
crease the size of the pool and retry..
java.sql.SQLException: Internal error: Cannot obtain XAConnection weblogic.common.resourcepool.ResourceLimitException: No resources currently available in pool cgDataSource to allocate to applications, please increase the size of the pool and retry..
at weblogic.jdbc.jta.DataSource.refreshXAConnAndEnlist(DataSource.java:1325)
at weblogic.jdbc.jta.DataSource.getConnection(DataSource.java:441)
at weblogic.jdbc.jta.DataSource.connect(DataSource.java:397)
at weblogic.jdbc.common.internal.RmiDataSource.getConnection(RmiDataSource.java:359)
at com.bea.wli.util.JDBCHelper.getConnection(JDBCHelper.java:77)



Still investigating on this weird message

SiteScope and configuring a MDB for monitoring

Are you lost? That's normal.... SiteScope is designed with that purpose in mind.

When adding a counter, search for:

com.bea.YOURSERVERNAME.APPLICATION_NAME.MODULE_NAME.
EJBPoolRuntime.JAR_NAME.MODULE_NAME.BeansInUseCount

(or whatever other attribute)

practical example:

com.bea/ACMEappli2/InfoHostingProcessesMdbBean_jms/infoHostingProcessesQueue/InfoHostingProcessesMdbBean_jms/infoHostingProcessesQueue/infoHostingProcesses/EJBPoolRuntime/infoHostingProcesses.jar/BeansInUseCurrentCount


Easy, isn't it?

Anatomy of a MDB

EJBRuntimeMBean is the base Interface for all EJB-related MBeans.

subinterfaces are:

- MessageDrivenEJBRuntimeMBean, EntityEJBRuntimeMBean and StatelessEJBRuntimeMBean, which all contain a EJBPoolRuntimeMBean:

http://download.oracle.com/docs/cd/E12839_01/apirefs.1111/e13945/weblogic/management/runtime/EJBPoolRuntimeMBean.html


with attributes:

AccessTotalCount, BeansInUseCurrentCount, DestroyedTotalCount, MissTotalCount, PooledBeansCurrentCount, TimeoutTotalCount, WaiterCurrentCount

- StatefulEJBRuntimeMBean contains instead a EJBCacheRuntimeMBean (Stateful are cached, not pooled, since they contain stateful info which should not be discarded or shared) and EJBLockingRuntimeMBean

All types of Bean contain a EJBTransactionRuntimeMBean.


How do I retrieve a MDB RuntimeMBean?

From a DomainRuntimeMBean, retrieve a ServerRuntimeMBean, then its ApplicationRuntimeMBean[], then its ComponentRuntimeMBean[], and test if its type is EJBComponentRuntime, in this case you can cast it to EJBComponentRuntimeMBean

Painful, no? Here is the code:


public List getEBJs(String serverName) {
List result = new ArrayList();
ServerRuntimeMBean serverRuntimeMBean = getServerRuntimeMBean(serverName);
if (serverRuntimeMBean != null) {
ApplicationRuntimeMBean[] apps = serverRuntimeMBean.getApplicationRuntimes();
for (ApplicationRuntimeMBean app : apps) {
ComponentRuntimeMBean[] components = app.getComponentRuntimes();
for (ComponentRuntimeMBean bean : components) {
if (bean.getType().equals("EJBComponentRuntime")) {
result.add((EJBComponentRuntimeMBean)bean);
}
}
}
}
return result;
}

What to monitor in an Application Server

EJBCache (ActivationCount CacheAccessCount CachedBeansCurrentCount CacheHitCount PassivationCount)

EJBLocking (LockEntriesCurrentCount LockManagerAccessCount TimeoutTotalCount WaiterCurrentCount WaiterTotalCount)

EJBPool (BeansInUseCount IdleBeansCount TimeoutTotalCount WaiterTotalCount)

EJBTransactions (TransactionsCommittedTotalCount TransactionsRolledBackTotalCount TransactionsTimedOutTotalCount)

JDBC (ActiveConnectionsCurrentCount ActiveConnectionsHighCount ConnectionsTotalCount LeakedConnectionCount MaxCapacity WaitingForConnectionCurrentCount WaitingForConnectionHighCount WaitSecondsHighCount)

JMS (MessagesCurrentCount MessagesHighCount MessagesPendingCount MessagesReceivedCount BytesCurrentCount BytesHighCount BytesPendingCount BytesReceivedCount)

JRockit (JvmProcessorLoad AllProcessorsAverageLoad TotalGarbageCollectionTime)

ThreadPool (busyThrdCurrCnt idleThrdCurrCnt totalThrdCurrCnt standbyThrdCurrCnt stuckThrdCurrCnt queueLength)

kill -3 and JRockit

in your domain home, create a file ctrlhandler.act

set_filename filename=/tmp/output.txt
print_class_summary
heap_diagnostics
print_threads
timestamp
jrarecording nativesamples=false latency filename=myjrarecording time=120
stop


every time you do a kill -3 pid, the JRockit VM will execute the control handler.

http://download.oracle.com/docs/cd/E13188_01/jrockit/geninfo/diagnos/ctrlbreakhndlr.html