Friday, June 24, 2016

really cool way to determine which Threads are consuming more CPU

http://code.nomad-labs.com/2010/11/18/identifying-which-java-thread-is-consuming-most-cpu/

this decimal to hex converter helps http://www.binaryhexconverter.com/decimal-to-hex-converter



Posted by vernetto at 8:57 AM 0 comments
Labels: , ,

Monday, June 20, 2016

java.lang.OutOfMemoryError: GC overhead limit exceeded at java.lang.Class.getDeclaredMethods0 

 Root cause of ServletException.
java.lang.OutOfMemoryError: GC overhead limit exceeded
 at java.lang.Class.getDeclaredMethods0(Native Method)
 at java.lang.Class.privateGetDeclaredMethods(Class.java:2532)
 at java.lang.Class.getDeclaredMethod(Class.java:1975)
 at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1437)
 at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:51)
 at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:450)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.io.ObjectStreamClass.(ObjectStreamClass.java:425)
 at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:322)
 at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1115)
 at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1535)
 at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
 at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1413)
 at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1159)
 at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:329)
 at weblogic.jms.common.ObjectMessageImpl.setObject(ObjectMessageImpl.java:158)
 at weblogic.jms.common.ObjectMessageImpl.setObject(ObjectMessageImpl.java:133)

"From your write-up, it appears you've set up a use case where the application attempts to operate on multiple large objects in parallel. The JVM is simply running out of heap. It looks the same thing would regardless of whether the large objects were JMS messages, or some other type of object... Some ideas: (A) free up JVM memory - possibly by tuning paging (see the JMS chapter of the Performance and Tuning Guide), (B) reduce the size of the async message pipe-line (search for "MessagesMaximum") in the same guide, (C) increase JVM memory, (D) see if its possible to break each large object up into multiple smaller objects/messages, but perhaps mostly (E) don't expect to be able to have multiple threads each operate on a seperate very large object concurrently - each object necessarily consumes memory while it is being accessed - strongly consider reducing the number of concurrent threads as per the MDB chapter of the Performance and Tuning guide. Tom "
(great Tom, Tom for president)
https://community.oracle.com/thread/1774008?tstart=0
http://docs.oracle.com/cd/E15523_01/web.1111/e13814/jmstuning.htm#CHDHIHGH
How to override serialization mechanisms http://www.oracle.com/technetwork/articles/java/javaserial-1536170.html

BUT: there could be an issue in the JDK ( see OpenJDK story here https://bugs.openjdk.java.net/browse/JDK-8065563 ) so you might just add more heap and restart and ignore the issue.



Posted by vernetto at 1:56 PM 0 comments
Labels: ,

Thursday, June 16, 2016

XAER_NOTA : The XID is not valid

If you get this error, most likely you need to tune your transaction timeoutS (there are many!)

Look for Oracle Support documents "Recommended Timeout Configuration When Using an External Transaction Processing Monitor and XA with Oracle (Doc ID 338880.1)" and for "WebLogic Server: Getting "XAER_NOTA : The XID is not valid" Exception with XA Datasource (Doc ID 1352715.1)"

The golden rule is "global transaction timeout < session timeout < distributed_lock_timeout", so check the value of the DB "distributed_lock_timeout" value.

See also http://www.javamonamour.org/2011/09/pending-xa-transactions-on-oracle-db.html and http://www.javamonamour.org/2015/10/ora-24756-transaction-does-not-exist.html



Posted by vernetto at 1:57 PM 0 comments
Labels:

Wednesday, June 8, 2016

ORA-01031: insufficient privileges creating a JDBCStore 


<Jun 8, 2016 9:12:41 AM CEST> <Error> <Store> <BEA-280072> <JDBC store "PIPPOJDBCStore" failed to open table "PIPPO_myhostname_WLStore".
weblogic.store.io.jdbc.JDBCStoreException: [Store:280065][Store:280068]JDBC store failed to initialize. It did not find its database table and failed trying to create a new empty table using the commands in file "/weblogic/store/io/jdbc/ddl/oracle.ddl". (server="PIPPO_myhostname" store="PIPPOJDBCStore" table="PIPPO_myhostname_WLStore"):(Linked Cause, "java.sql.SQLSyntaxErrorException: ORA-01031: insufficient privileges
")
        at weblogic.store.io.jdbc.JDBCStoreIO.createTable(JDBCStoreIO.java:881)
        at weblogic.store.io.jdbc.JDBCStoreIO.open(JDBCStoreIO.java:538)
        at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:476)
        at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:453)
        at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:129)
        at weblogic.store.admin.JDBCAdminHandler.activate(JDBCAdminHandler.java:73)
        at weblogic.management.utils.GenericManagedDeployment.activate(GenericManagedDeployment.java:158)
        at weblogic.management.utils.GenericDeploymentManager.activate(GenericDeploymentManager.java:184)
        at weblogic.management.utils.GenericManagedService.activateDeployment(GenericManagedService.java:211)
        at weblogic.management.utils.GenericServiceManager.activateDeployment(GenericServiceManager.java:131)
        at weblogic.management.internal.DeploymentHandlerHome.invokeHandlers(DeploymentHandlerHome.java:823)
        at weblogic.management.internal.DeploymentHandlerHome.activateInitialDeployments(DeploymentHandlerHome.java:666)
        at weblogic.management.internal.DeploymentHandlerHome.activateInitialDeployments(DeploymentHandlerHome.java:627)
        at weblogic.management.deploy.internal.DeploymentAdapter$2.doPrepare(DeploymentAdapter.java:101)
        at weblogic.management.deploy.internal.DeploymentAdapter.prepare(DeploymentAdapter.java:191)
        at weblogic.management.deploy.internal.AppTransition$1.transitionApp(AppTransition.java:21)
        at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:240)
        at weblogic.management.deploy.internal.ConfiguredDeployments.prepare(ConfiguredDeployments.java:165)
        at weblogic.management.deploy.internal.ConfiguredDeployments.deploy(ConfiguredDeployments.java:122)
        at weblogic.management.deploy.internal.DeploymentServerService.resume(DeploymentServerService.java:211)
        at weblogic.management.deploy.internal.DeploymentServerService.start(DeploymentServerService.java:119)
        at weblogic.server.AbstractServerService.postConstruct(AbstractServerService.java:78)
        at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1017)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:388)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:430)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:225)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:82)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:606)
        at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:77)
        at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:231)
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:254)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:413)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:225)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:82)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:87)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1162)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1147)
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:553)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:263)

Caused By: java.sql.SQLSyntaxErrorException: ORA-01031: insufficient privileges

        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:450)
        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:399)
        at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1059)
        at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:522)
        at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:257)
        at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:587)
        at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:210)
        at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:30)
        at oracle.jdbc.driver.T4CStatement.executeForRows(T4CStatement.java:931)
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1150)
        at oracle.jdbc.driver.OracleStatement.executeInternal(OracleStatement.java:1792)
        at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:1745)
        at oracle.jdbc.driver.OracleStatementWrapper.execute(OracleStatementWrapper.java:334)
        at weblogic.jdbc.wrapper.Statement.execute(Statement.java:490)
        at weblogic.store.io.jdbc.ReservedConnection.executeDDLString(ReservedConnection.java:1169)
        at weblogic.store.io.jdbc.ReservedConnection.executeDDLStream(ReservedConnection.java:1141)
        at weblogic.store.io.jdbc.JDBCStoreIO.createTable(JDBCStoreIO.java:870)
        at weblogic.store.io.jdbc.JDBCStoreIO.open(JDBCStoreIO.java:538)
        at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:476)
        at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:453)
        at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:129)
        at weblogic.store.admin.JDBCAdminHandler.activate(JDBCAdminHandler.java:73)
        at weblogic.management.utils.GenericManagedDeployment.activate(GenericManagedDeployment.java:158)
        at weblogic.management.utils.GenericDeploymentManager.activate(GenericDeploymentManager.java:184)
        at weblogic.management.utils.GenericManagedService.activateDeployment(GenericManagedService.java:211)
        at weblogic.management.utils.GenericServiceManager.activateDeployment(GenericServiceManager.java:131)
        at weblogic.management.internal.DeploymentHandlerHome.invokeHandlers(DeploymentHandlerHome.java:823)
        at weblogic.management.internal.DeploymentHandlerHome.activateInitialDeployments(DeploymentHandlerHome.java:666)
        at weblogic.management.internal.DeploymentHandlerHome.activateInitialDeployments(DeploymentHandlerHome.java:627)
        at weblogic.management.deploy.internal.DeploymentAdapter$2.doPrepare(DeploymentAdapter.java:101)
        at weblogic.management.deploy.internal.DeploymentAdapter.prepare(DeploymentAdapter.java:191)
        at weblogic.management.deploy.internal.AppTransition$1.transitionApp(AppTransition.java:21)
        at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:240)
        at weblogic.management.deploy.internal.ConfiguredDeployments.prepare(ConfiguredDeployments.java:165)
        at weblogic.management.deploy.internal.ConfiguredDeployments.deploy(ConfiguredDeployments.java:122)
        at weblogic.management.deploy.internal.DeploymentServerService.resume(DeploymentServerService.java:211)
        at weblogic.management.deploy.internal.DeploymentServerService.start(DeploymentServerService.java:119)
        at weblogic.server.AbstractServerService.postConstruct(AbstractServerService.java:78)
        at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1017)
        at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:388)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:430)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:225)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:82)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:606)
        at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:77)
        at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:231)
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:254)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:413)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:225)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:82)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
        at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:87)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1162)
        at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1147)
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:553)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:263)
>
in com.bea.core.store.jdbc_4.0.0.0.jar you find the \weblogic\store\io\jdbc\ddl\oracle.ddl file 

# WebLogic JDBC Store DDL for Oracle
# Copyright (c) 2003 by BEA, Inc., All Rights Reserved

CREATE TABLE $TABLE (
  id     int  not null primary key,
  type   int  not null,
  handle int  not null,
  record long raw not null
);
The configuration in config.xml is: 


<jms-server>
<name>puppo_PlatformJMSserver</name>
<target>pippo_myhostname</target>
<persistent-store>PUPPOJDBCStore</persistent-store>
<temporary-template-resource xsi:nil="true"></temporary-template-resource>
<temporary-template-name xsi:nil="true"></temporary-template-name>
</jms-server>


<jms-system-resource>
<name>PUPPOqueue</name>
<target>pippo_myhostname</target>
<sub-deployment>
<name>PUPPOqueuesubdeployment</name>
<target>puppo_PlatformJMSserver</target>
</sub-deployment>
<descriptor-file-name>jms/PUPPOqueue-jms.xml</descriptor-file-name>
</jms-system-resource>

  <jdbc-store>
    <name>PUPPOJDBCStore</name>
    <prefix-name>PIPPO_myhostname_</prefix-name>
    <data-source>JDBCStoreDS</data-source>
    <target>pippo_myhostname</target>
  </jdbc-store>


The whole JDBCStore story is documented here http://docs.oracle.com/cd/E12840_01/wls/docs103/config_wls/store.html#wp1143086

So the solution is that either the DBA grant the user associated to the JDBCStoreDS the right to create tables, or they create the table themselves using the DDL above...

Posted by vernetto at 11:02 AM 0 comments
Labels:

Sunday, June 5, 2016

BEA-090171 and BEA-090169

in WebLogic there are 2 messages helping us identify the identity and trust store used :
BEA-090171 Loading the identity certificate and private key stored under the alias client ....

BEA-090169 Loading trusted certificates from the JKS file store

This flag is useful to trace: -Dweblogic.security.SSL.verbose=true -Djavax.net.debug=all
Check also the "ssl" element in config.xml: 
    <ssl>
      <use-java>true</use-java>
      <enabled>true</enabled>
      <ciphersuite>TLS_RSA_WITH_AES_256_CBC_SHA</ciphersuite>
      <ciphersuite>TLS_RSA_WITH_AES_128_CBC_SHA</ciphersuite>
      <ciphersuite>TLS_RSA_WITH_RC4_128_SHA</ciphersuite>
      <ciphersuite>TLS_RSA_WITH_3DES_EDE_CBC_SHA</ciphersuite>
      <hostname-verifier>com.pierre.HostNameVerifier</hostname-verifier>
      <hostname-verification-ignored>false</hostname-verification-ignored>
      <export-key-lifespan>500</export-key-lifespan>
      <client-certificate-enforced>true</client-certificate-enforced>
      <listen-port>33641</listen-port>
      <two-way-ssl-enabled>true</two-way-ssl-enabled>
      <server-private-key-alias>some.certificate.alias</server-private-key-alias>
      <server-private-key-pass-phrase-encrypted>{AES}blablabla=</server-private-key-pass-phrase-encrypted>
      <ssl-rejection-logging-enabled>true</ssl-rejection-logging-enabled>
      <inbound-certificate-validation>BuiltinSSLValidationAndCertPathValidators</inbound-certificate-validation>
      <outbound-certificate-validation>BuiltinSSLValidationOnly</outbound-certificate-validation>
      <jsse-enabled>true</jsse-enabled>
    </ssl>


SSLMBean.ServerPrivateKeyAlias https://docs.oracle.com/cd/E13222_01/wls/docs100/schemaref/security/http.www.bea.com.ns.weblogic.920.domain/types/ssltype.server-private-key-alias.html



Posted by vernetto at 10:38 AM 0 comments
Labels: ,
Subscribe to: Posts (Atom)