Puppet: getting started with hiera (hiera example)

Official doc here

I don't like hiera, but it's better than hardcoding configuration in .pp files.
First attempt:
vi hieratest.pp

$pippo = hiera('pippo')
notify { "${pippo}" : }

I run "puppet apply hieratest.pp" and I get:
Hiera config file /etc/puppetlabs/puppet/hiera.yaml not readable at /root/puppettests/hieratest.pp:1
then I do vi /etc/puppetlabs/puppet/hiera.yaml and I enter:

:hierarchy:
    - hosts/%{::fqdn}
    - environments/%{environment}
    - common
    - users

:backends:
    - yaml

:yaml:
    :datadir: '/etc/puppetlabs/puppet/hieradata/'

and rerun "puppet apply hieratest.pp" and I get:

Could not find data item pippo in any Hiera data file and no default supplied at /root/puppettests/hieratest.pp:1

Better than before :o)
so I do
mkdir /etc/puppetlabs/puppet/hieradata/
vi /etc/puppetlabs/puppet/hieradata/common.yaml
(remark: common is a member of the hierarchy in the hiera.yaml' and I enter

pippo : 'hello world!'

Again "puppet apply hieratest.pp":
notice: hello world!
Hurrah! My first hiera-based module!

Gridlink Datasource registering with ONS daemons

When you have Gridlink DS, you will see in the logs this debug message (of course if your log level is debug :o) )

<BEA-001556> <Data Source ACME_ConfigDS for service o01osb_app.acme.com registering with ONS daemons using configuration string nodes=acme535:6200,acme536:6200,acme531:6200,acme532:6200,acme533:6200,acme534:6200

In our case, this would take 30 seconds for each DS, and make the restart of the cluster painfully slow.

Check that each individual node in the list, running ONS agent, is actually reachable (no firewall!). Check also that it's up and running.

Vertical Tabs in Internet Explorer

I am forced to use Internet Exploder lately, and I go literally mad with its non-customizeability of tabs: they are stuck on the top, stealing valuable vertical space.

As very well highlighted in this post:

http://pim.famnit.upr.si/blog/index.php?/archives/301-Web-browsers-on-Windows-OS-comparison-of-vertical-vs.-horizontal-tabs.html

the most ergonomic browser IMHO remains Firefox.

The good news is that you can install Firefox Portable Edition without admin rights.... FAREWELL IE!!!

Iterations in Puppet: create_resources

So, the BAD news is that in Puppet you cannot do a for loop and create resources specified in a Collection.
Why not? Because. Don't ask. The Founding Fathers of Puppet decided so, and thou shall not dare question why.
The good news is that you can STILL iterate, sticking all your (homogeneous) resources in a hash (Puppet supports this very advanced concept of Hash, disregarding 30 years of Object Oriented programming technology... call it Time Travel).

#Where the .crt .key and .cer files are
$certsFolder = '/home/soa/jkstest/source/'

#where to create the JKS files
$targetJKSFolder = '/home/soa/jkstest/target/'

$trustPassword = '111111'


/* This is how a traditional java_ks invokation looks like
java_ks { "ca_nestle:trustDEV.jks" :
    ensure       => latest,
    certificate  => "${certsFolder}ACMECA.cer",
    target       => "${targetJKSFolder}/trustDEV.jks",
    password     => "${trustPassword}",
    trustcacerts => true,
}

 */
 

$jksHash = {
  trustDEV1 => {
    ensure       => latest,
    certificate  => "${certsFolder}ACMECA.cer",
    target       => "${targetJKSFolder}/trustDEV.jks",
    password     => "${trustPassword}",
    trustcacerts => true,
  },

  trustDEV2 => {
    ensure       => latest,
    certificate  => "${certsFolder}ItalianSignCA.cer",
    target       => "${targetJKSFolder}/trustDEV.jks",
    password     => "${trustPassword}",
    trustcacerts => true,
  }
}

create_resources(java_ks, $jksHash)

This is how, in one go, I can add 2 certificates in a JKS store.

What can I say. It could be worse. It could be Maven.

Syria

Import an existing Private Key (.key file) into a JKS store

Sounds easy but it's not. I have an existing .key (private key) and .crt file (certificate for public key) and I want to import them into a JKS.

http://stackoverflow.com/questions/906402/importing-an-existing-x509-certificate-and-private-key-in-java-keystore-to-use-i

The only way seems to go through an intermediate pkcs12 store, to be imported later in the JKS. So much fuss for a simple operation which should be natively supported.

http://cunning.sharp.fm/2008/06/importing_private_keys_into_a.html

In fact, puppet JAVA_JKS module does this:

openssl pkcs12 -export -passout stdin -in /home/soa/jkstest/source/acme.com.crt -inkey /home/soa/jkstest/source/acme.com.key -name test4acme.com

when you ask him to import .crt and .key into a keystore:

    java_ks { "${nesoa2env}acme.com" :
        ensure       => latest,
        certificate  => "${certsFolder}acme.com.crt",
        private_key  => "${certsFolder}acme.com.key", 
        target       => "${targetJKS}",
        password     => "${identityPassword}",
        trustcacerts => false,
    }

JKS management made easy with Portecle

http://sourceforge.net/projects/portecle/?source=dlp

One of the irritating things about security is all those different options in the keytool command line. Portecle makes life a lot easier by managing ONE JKS.

Unfortunately I still need to find an application which can ease the pain of maintaining all the security-related artifacts in a complex organization - based on a DATABASE of certificates, JKS stores, private keys etc.

Book: The Healthy Programmer

http://pragprog.com/book/jkthp/the-healthy-programmer

The book provides a host of evidence about the negative effect (=early death and painful life) of the lifestyle associated with most office jobs: long hours sitting, not enough aerobic exercise, strain on specific parts of the body (neck, wrists, spine)....

The solution is quite simple: take more breaks, walk, have a healthy nutrition (NO SUGAR, NO SODAS). Nothing new honestly.

keytool error :java.io.IoException:Incorrect AVA format

I run this command:

keytool -genkeypair -alias alias -keyalg RSA -keysize 1024 -dname dn -keystore keystore

and this fails with "keytool error :java.io.IoException:Incorrect AVA format"

Just remove -dname:

keytool -genkeypair -alias alias -keyalg RSA -keysize 1024 -keystore keystore

and answer all questions directly, and you will be fine. Otherwise, provide a proper dname, like "CN=Mark Smith, OU=JavaSoft, O=Sun, L=Cupertino, S=California, C=US". For details, see the EXCELLENT http://docs.oracle.com/javase/6/docs/technotes/tools/windows/keytool.html (when everything else fails, read the manual). Especially, avoid funny characters in the dname, including . or $ or #

Stderr: VBoxManage.exe: error: Could not rename the directory

I am not a big fan of Vagrant, its behavior is way too erratic and opaque for me. When it works we are happy, when it fails - and it fails way too often - we are left googling sparse and stern documentation.

This new error "Stderr: VBoxManage.exe: error: Could not rename the directory" could be fixed only after LOTS of googling and trial and error:

Vagrant.configure("2") do |config2|
  # ... (other config)
  config2.vm.provider :virtualbox do |vb|
      vb.name = "jkstest"
  end
end

This should be added just after the config.vm.box_url clause, and BEFORE any customize.

How frustrating.

Despite this settings, my VB is created in C:\Users\myuser\VirtualBox VMs\workspaceJKS_1377339021, rather than in the jkstest folder.

The original error message was:

There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["modifyvm", "3a81f6e5-ba5a-438b-85a8-501a63a3a053", "--name", "trunk_1377853840"]

Stderr: VBoxManage.exe: error: Could not rename the directory 'C:\Users\pippo\VirtualBox VMs\acme-basebox' to
'C:\Users\pippo\VirtualBox VMs\trunk_1377853840' to save the settings file (VERR_ACCESS_DENIED)
VBoxManage.exe: error: Details: code E_FAIL (0x80004005), component SessionMachine, interface IMachine, callee IUnknown
VBoxManage.exe: error: Context: "SaveSettings()" at line 2527 of file VBoxManageModifyVM.cpp

In fact, there was another VBOX running, which had been created with the name "acme-basebox" (this is the setting config.vm.box = "acme-basebox" . Shutting down this machine allowed me to start the other without the above mentioned hack.

Display JKS content in Python (WLST)

from java.security import KeyStore
from java.security import MessageDigest
from java.io import FileInputStream


def hexify(bytes):
    hexDigits = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'A', 'B', 'C', 'D', 'E', 'F']
    buf = StringBuffer()
    for i in range(len(bytes)):
        buf.append(hexDigits[(bytes[i] & 0xf0) >> 4])
        buf.append(hexDigits[bytes[i] & 0x0f])
        if i < len(bytes) - 1:
            buf.append(':')
    return buf.toString()


filestore='myacme.jks'
password='111111'
ks = KeyStore.getInstance("JKS")
ks.load(FileInputStream(filestore), password)
md = MessageDigest.getInstance("MD5")

for item in ks.aliases():
 print "alias", item, "isCertificate", ks.isCertificateEntry(item)
 if ks.isCertificateEntry(item):
  print "isCertificate"
  cert = ks.getCertificate(item)
  der = cert.getEncoded()
  md.update(der)
  print hexify(md.digest())

Inspired by http://stackoverflow.com/questions/1270703/how-to-retrieve-compute-an-x509-certificates-thumbprint-in-java

Javadoc: MessageDigest , KeyStore, Certificate, Key

Display JKS content in Windows (keytool)

There is an excellent plugin for Eclipse:

http://keytool.sourceforge.net/

I was bored of running always the command line

keytool -keystore BLA -list

OSB transaction required

The advantage of setting "transaction required" for your Proxies is that the incoming request will be starting a transaction and set the default <con:qualityOfService>exactly-once</con:qualityOfService> , instead of <con:qualityOfService>best-effort</con:qualityOfService>. You can examine the $inbound you will find this setting.

Subsequent Publish or Route will use this same Quality Of Service, instead of the default which is "best-effort".

FileListDAO::Unable to pickup

We use HAFileAdapter to read files from a shared folder. Occasionally we find in the logs

FileListDAO::Unable to pickup [BLA], this error will ignored

2 Managed Servers at the same time have detected the file with their FilePoller, so the 2 of them at the same time run to get the exclusive lock on the file by updating the table FILEADAPTER_IN to set PROCESSED = 1, but only one of them gets it, the other gets no lock and logs a warning saying "I was unable to get the lock". This is not an issue, actually it should not even be a warning at all, since it's meant to work that way.

This is done in FileListDAO: public boolean claimLock(FileInfo fileInfo)

OSB : weblogic datasource suspended

This is the overall story: several JMS Proxy Services (MDB) use a certain DS (datasource). Each of these Proxy has 16 MDB created by OSB.
To avoid overloading the DB with too many concurrent requests we sized the Connection Pool (CP) of this DS to 25. Evidently if there is a burst of activity (JMS) on 2 different Proxy Services, the DS will be overloaded and eventually we shall get a

weblogic.jdbc.extensions.PoolDisabledSQLException: weblogic.common.resourcepool.ResourceDisabledException: Pool BLA is Suspended, cannot allocate resources to applications...

http://egherardini.wordpress.com/2011/04/27/oracle-soa-suite-11g-datasource-has-been-suspended/
select count(*) from v$process;
select count(*) from v$session;
select * from v$parameter where name in ('sessions','processes','transactions');


The only effective workaround I see is associating the same WorkManager with a Max Threads Constraint of 25 to ALL JMS Proxy Services.

OSB FAQ

Just a collection of useful links on less-understood topics

what is a SB protocol in OSB?

(this was copied from Oracle Forum:) SB transport is primarily used for invocations between different OSB instances where transaction propagation is required. eg In an organization, two department use OSB for hosting their services. Finance services are hosted on OSB_Fin domain and Marketing services are hosted on OSB_Mar domain. If there is an requirement for transactional communication between the proxies of these two service then it would be ideal to use SB transport. Also can also be used for routing to local proxies (SB business service without a JNDI provider) and in that case SB transport is optimized for RMI.

_______________

how do I use client certificates to secure a HTTP Proxy?

see this excellent tutorial http://www.middlewareguru.com/mw/?p=425 and http://www.middlewareguru.com/mw/?p=459#more-459

and a must-read on TLS http://www.infoq.com/articles/HTTPS-Connection-Jeff-Moser

_______________

should I enable "Transaction Required" in a Proxy?

see explanation here: http://docs.oracle.com/cd/E14571_01/doc.1111/e15866/ui_ref.htm

Most likely yes, unless you want to set explicitly "Quality Of Service=exactly once" for every Service Callout or Route operation

for an excellent coverage of Transactions in OSB, read http://atheek.wordpress.com/2011/04/21/transaction-handling-within-osb/

_______________

in a JMS Business Service, what is the "Response Queue" for?

a BS can handle synchronous calls to external services using 2 JMS queues (one to send the request, the other to receive the response) as channels. correlation of the 2 messages is done commonly through JMSCorrelationID

____________

in a JMS Proxy Service, what is the "Same Transaction For Response" option for?

look at this excellent video http://www.youtube.com/watch?v=KtUXaXeoUVU

and the "Is Response Required " ?

JSP in a WAR showing maven version

I have googled the planet for a simple solution to displaying the build version of the war project in a JSP.



If you use a war plugin, then you might try filtering.



Personally I HATE maven so, after a few failed attempts, I have simply done this and it works magically:




in my jsp I simply read a file in META_INF folder:

Properties mavenproperties = new Properties();
mavenproperties.load(getServletConfig().getServletContext().getResourceAsStream("META-INF/maven/com.acme.tools/wlobelix/pom.properties"));
out.write("
obelixversion=" + mavenproperties.getProperty("version") + "
");

being wlobelix my artifactIdand and com.acme.common my groupId




pom.properties contains:

#Generated by Maven
#Fri Aug 16 15:38:28 CEST 2013
version=1.8-SNAPSHOT
groupId=com.acme.common
artifactId=wlobelix

Pushpaka Vimanamu : one of the most enjoyable movies ever

best experimental indian movie:


http://en.wikipedia.org/wiki/Pushpaka_Vimana_%28film%29

Quick guide to OPatch

http://docs.oracle.com/cd/E15586_01/doc.1111/e16793/opatch.htm

Create a folder /opt/oracle/patches
Copy there your patch zip file (p16266172_111150_Generic.zip)
unzip p16266172_111150_Generic.zip
cd /opt/oracle/patches/16266172
export MW_HOME=/opt/oracle/fmw11_1_1_5/
export ORACLE_HOME=/opt/oracle/fmw11_1_1_5/osb
shut down all wl servers
/opt/oracle/fmw11_1_1_5/oracle_common/OPatch/opatch apply -invPtrLoc /opt/oracle/fmw11_1_1_5/osb/oraInst.loc
restart the domain

I got an error "OPatch failed with error code 73", this because opatch could not find the /etc/config/actions.xml file.

Multiple definitions of server-group BI-ADF-ADMIN-SVR are not allowed

I had got occasionally this error in the past, while creating a OSB domain. No clue what caused it. Normally I delete all and restart. Once I find the root cause I will update this post.

I think something in a template is defined twice.

wlst NameError: cd (using WLST as a Jython module)

I am defining a function in a python module acmelibrary.py:

def createMailSession(theMailSessionName, adminName, clusterName):
    cd('/')

and I invoke it like this:

from acmelibrary import createMailSession

createMailSession(theMailSessionName, adminName, clusterName)

Much to my dismay, the cd('/') statement works perfectly when invoked in the main module, but not in the library:
NameError: cd

Workaround:

in all WLST scripts there is a variable WLS of type com.oracle.cie.domain.script.jython.WLScriptContext, and you have to pass it as a parameter to the createMailSession:

def createMailSession(wls, theMailSessionName, adminName, clusterName):
    wls.cd('/')

A more radical approach is to create a wl.py module:

from weblogic.management.scripting.utils import WLSTUtil
import sys
origPrompt = sys.ps1
theInterpreter = WLSTUtil.ensureInterpreter();
WLSTUtil.ensureWLCtx(theInterpreter)
execfile(WLSTUtil.getWLSTScriptPath())
execfile(WLSTUtil.getOfflineWLSTScriptPath())
exec(WLSTUtil.getOfflineWLSTScriptForModule())
execfile(WLSTUtil.getWLSTCommonModulePath())
theInterpreter = None
sys.ps1 = origPrompt
modules = WLSTUtil.getWLSTModules()
for mods in modules:
  execfile(mods.getAbsolutePath())
wlstPrompt = "false"  

and in your acmelibrary.py you start with a

import wl

at this point your code can do

def createMailSession(theMailSessionName, adminName, clusterName):
    wl.cd('/')

See also http://docs.oracle.com/cd/E15051_01/wls/docs103/config_scripting/using_WLST.html#wp1094333

The OSB framework, intercepting an logging all requests

The base issue is that whenever we have a stuck thread, we don't have a means to tell which Proxy Service it was processing. This is a sample StackTrace:

"[STUCK] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)'" RUNNABLE native
 java.net.SocketInputStream.socketRead0(Native Method)
 java.net.SocketInputStream.read(SocketInputStream.java:129)
 oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:718)
 oracle.net.ns.Packet.receive(Packet.java:295)
 oracle.net.ns.DataPacket.receive(DataPacket.java:106)
 oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:317)
 oracle.net.ns.NetInputStream.read(NetInputStream.java:262)
 oracle.net.ns.NetInputStream.read(NetInputStream.java:187)
 oracle.net.ns.NetInputStream.read(NetInputStream.java:104)
 oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:126)
 oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:82)
 oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1177)
 oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1153)
 oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:312)
 oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:204)
 oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:540)
 oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:213)
 oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:1075)
 oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1466)
 oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3752)
 oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3937)
 oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:9259)
 oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1535)
 weblogic.jdbc.wrapper.PreparedStatement.execute(PreparedStatement.java:99)
 oracle.tip.adapter.db.sp.AbstractStoredProcedure.execute(AbstractStoredProcedure.java:123)
 oracle.tip.adapter.db.sp.SPInteraction.executeStoredProcedure(SPInteraction.java:141)
 oracle.tip.adapter.db.DBInteraction.executeStoredProcedure(DBInteraction.java:1102)
 oracle.tip.adapter.db.DBInteraction.execute(DBInteraction.java:247)
 oracle.tip.adapter.sa.impl.fw.wsif.jca.WSIFOperation_JCA.performOperation(WSIFOperation_JCA.java:529)
 oracle.tip.adapter.sa.impl.fw.wsif.jca.WSIFOperation_JCA.executeOperation(WSIFOperation_JCA.java:353)
 oracle.tip.adapter.sa.impl.fw.wsif.jca.WSIFOperation_JCA.executeRequestResponseOperation(WSIFOperation_JCA.java:312)
 oracle.tip.adapter.sa.impl.JCABindingReferenceImpl.invokeWsifProvider(JCABindingReferenceImpl.java:350)
 oracle.tip.adapter.sa.impl.JCABindingReferenceImpl.request(JCABindingReferenceImpl.java:253)
 com.bea.wli.sb.transports.jca.binding.JCATransportOutboundOperationBindingServiceImpl.invoke(JCATransportOutboundOperationBindingServiceImpl.java:150)
 com.bea.wli.sb.transports.jca.JCATransportEndpoint.sendRequestResponse(JCATransportEndpoint.java:209)
 com.bea.wli.sb.transports.jca.JCATransportEndpoint.send(JCATransportEndpoint.java:170)
 com.bea.wli.sb.transports.jca.JCATransportProvider.sendMessageAsync(JCATransportProvider.java:598)
 sun.reflect.GeneratedMethodAccessor645.invoke(Unknown Source)
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 java.lang.reflect.Method.invoke(Method.java:597)
 com.bea.wli.sb.transports.Util$1.invoke(Util.java:83)
 $Proxy140.sendMessageAsync(Unknown Source)
 com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:148)
 com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToServiceAsync(LoadBalanceFailoverListener.java:603)
 com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToService(LoadBalanceFailoverListener.java:538)
 com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:558)
 com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:426)
 com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:670)
 com.bea.wli.sb.pipeline.PipelineContextImpl.dispatchSync(PipelineContextImpl.java:551)
 stages.transform.runtime.WsCalloutRuntimeStep$WsCalloutDispatcher.dispatch(WsCalloutRuntimeStep.java:1391)
 stages.transform.runtime.WsCalloutRuntimeStep.processMessage(WsCalloutRuntimeStep.java:236)
 com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:346)
 com.bea.wli.sb.pipeline.PipelineStage.processMessage(PipelineStage.java:84)
 com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1055)
 com.bea.wli.sb.pipeline.Pipeline.processMessage(Pipeline.java:141)
 com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1055)
 com.bea.wli.sb.pipeline.PipelineNode.doRequest(PipelineNode.java:55)
 com.bea.wli.sb.pipeline.Node.processMessage(Node.java:67)
 com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1055)
 com.bea.wli.sb.pipeline.Router.processMessage(Router.java:214)
 com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:96)
 com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:593)
 com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:591)
 weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
 weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
 com.bea.wli.sb.security.WLSSecurityContextService.runAs(WLSSecurityContextService.java:55)
 com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:590)
 com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:375)
 com.bea.wli.sb.transports.local.LocalMessageContext$1.run(LocalMessageContext.java:179)
 weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
 weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
 weblogic.security.Security.runAs(Security.java:61)
 com.bea.wli.sb.transports.local.LocalMessageContext.send(LocalMessageContext.java:174)
 com.bea.wli.sb.transports.local.LocalTransportProvider.sendMessageAsync(LocalTransportProvider.java:322)
 sun.reflect.GeneratedMethodAccessor645.invoke(Unknown Source)
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 java.lang.reflect.Method.invoke(Method.java:597)
 com.bea.wli.sb.transports.Util$1.invoke(Util.java:83)
 $Proxy125.sendMessageAsync(Unknown Source)
 com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:148)
 com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToServiceAsync(LoadBalanceFailoverListener.java:603)
 com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToService(LoadBalanceFailoverListener.java:538)
 com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:558)
 com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:426)
 com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:670)
 com.bea.wli.sb.pipeline.PipelineContextImpl.dispatchSync(PipelineContextImpl.java:551)
 stages.transform.runtime.WsCalloutRuntimeStep$WsCalloutDispatcher.dispatch(WsCalloutRuntimeStep.java:1391)
 stages.transform.runtime.WsCalloutRuntimeStep.processMessage(WsCalloutRuntimeStep.java:236)
 com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:346)
 com.bea.wli.sb.stages.impl.SequenceRuntimeStep.processMessage(SequenceRuntimeStep.java:33)
 com.bea.wli.sb.pipeline.PipelineStage.processMessage(PipelineStage.java:84)
 com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1055)
 com.bea.wli.sb.pipeline.Pipeline.processMessage(Pipeline.java:141)
 com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1055)
 com.bea.wli.sb.pipeline.PipelineNode.doRequest(PipelineNode.java:55)
 com.bea.wli.sb.pipeline.Node.processMessage(Node.java:67)
 com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1055)
 com.bea.wli.sb.pipeline.Router.processMessage(Router.java:214)
 com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:96)
 com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:593)
 com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:591)
 weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
 weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
 com.bea.wli.sb.security.WLSSecurityContextService.runAs(WLSSecurityContextService.java:55)
 com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:590)
 com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:375)
 com.bea.wli.sb.transports.local.LocalMessageContext$1.run(LocalMessageContext.java:179)
 weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
 weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
 weblogic.security.Security.runAs(Security.java:61)
 com.bea.wli.sb.transports.local.LocalMessageContext.send(LocalMessageContext.java:174)
 com.bea.wli.sb.transports.local.LocalTransportProvider.sendMessageAsync(LocalTransportProvider.java:322)
 sun.reflect.GeneratedMethodAccessor645.invoke(Unknown Source)
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 java.lang.reflect.Method.invoke(Method.java:597)
 com.bea.wli.sb.transports.Util$1.invoke(Util.java:83)
 $Proxy125.sendMessageAsync(Unknown Source)
 com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageAsync(LoadBalanceFailoverListener.java:148)
 com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToServiceAsync(LoadBalanceFailoverListener.java:603)
 com.bea.wli.sb.transports.LoadBalanceFailoverListener.sendMessageToService(LoadBalanceFailoverListener.java:538)
 com.bea.wli.sb.transports.TransportManagerImpl.sendMessageToService(TransportManagerImpl.java:558)
 com.bea.wli.sb.transports.TransportManagerImpl.sendMessageAsync(TransportManagerImpl.java:426)
 com.bea.wli.sb.pipeline.PipelineContextImpl.doDispatch(PipelineContextImpl.java:670)
 com.bea.wli.sb.pipeline.PipelineContextImpl.dispatch(PipelineContextImpl.java:585)
 stages.publish.runtime.PublishRuntimeStep.processMessage(PublishRuntimeStep.java:181)
 com.bea.wli.sb.pipeline.StatisticUpdaterRuntimeStep.processMessage(StatisticUpdaterRuntimeStep.java:41)
 com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:346)
 com.bea.wli.sb.stages.impl.SequenceRuntimeStep.processMessage(SequenceRuntimeStep.java:33)
 stages.transform.runtime.ForEachRuntimeStep.processMessage(ForEachRuntimeStep.java:101)
 com.bea.wli.sb.pipeline.StatisticUpdaterRuntimeStep.processMessage(StatisticUpdaterRuntimeStep.java:41)
 com.bea.wli.sb.stages.StageMetadataImpl$WrapperRuntimeStep.processMessage(StageMetadataImpl.java:346)
 com.bea.wli.sb.pipeline.PipelineStage.processMessage(PipelineStage.java:84)
 com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1055)
 com.bea.wli.sb.pipeline.Pipeline.processMessage(Pipeline.java:141)
 com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1055)
 com.bea.wli.sb.pipeline.PipelineNode.doRequest(PipelineNode.java:55)
 com.bea.wli.sb.pipeline.Node.processMessage(Node.java:67)
 com.bea.wli.sb.pipeline.PipelineContextImpl.execute(PipelineContextImpl.java:1055)
 com.bea.wli.sb.pipeline.Router.processMessage(Router.java:214)
 com.bea.wli.sb.pipeline.MessageProcessor.processRequest(MessageProcessor.java:96)
 com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:593)
 com.bea.wli.sb.pipeline.RouterManager$1.run(RouterManager.java:591)
 weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
 weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
 com.bea.wli.sb.security.WLSSecurityContextService.runAs(WLSSecurityContextService.java:55)
 com.bea.wli.sb.pipeline.RouterManager.processMessage(RouterManager.java:590)
 com.bea.wli.sb.transports.TransportManagerImpl.receiveMessage(TransportManagerImpl.java:375)
 com.bea.wli.sb.transports.http.generic.RequestHelperBase.invokePipeline(RequestHelperBase.java:179)
 com.bea.wli.sb.transports.http.wls.HttpTransportServlet$RequestHelperWLS.invokePipeline(HttpTransportServlet.java:227)
 com.bea.wli.sb.transports.http.generic.RequestHelperBase$1.run(RequestHelperBase.java:154)
 com.bea.wli.sb.transports.http.generic.RequestHelperBase$1.run(RequestHelperBase.java:152)
 weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
 weblogic.security.service.SecurityManager.runAs(SecurityManager.java:146)
 com.bea.wli.sb.transports.http.generic.RequestHelperBase.securedInvoke(RequestHelperBase.java:151)
 com.bea.wli.sb.transports.http.generic.RequestHelperBase.service(RequestHelperBase.java:107)
 com.bea.wli.sb.transports.http.wls.HttpTransportServlet.service(HttpTransportServlet.java:129)
 weblogic.servlet.FutureResponseServlet.service(FutureResponseServlet.java:24)
 javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
 weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
 weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
 weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
 weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:183)
 weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3717)
 weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
 weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
 weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
 weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
 weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183)
 weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454)
 weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
 weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
 

We vaguely understand that (bottom to top):

this was a HTTP Proxy, since at transport level it's initiated by a ServletRequest

after a LOT of layers of transport and security related processing, we reach the core RouterManager.processMessage() method. We understand that the RouterManager is the core on the Proxy Message Flow processing.

the RouterManager creates a RouterContext, which contains a reference both to the MessageContext
public MessageContext getMessageContext();
and to the ProxyService:
public ProxyService getProxyService();
these 2 classes identify both the message (messageId) and the proxy being invoked.

a static call to MessageProcessor.processRequest(RouterContext context) will then deal with getting the right Pipeline and pushing the message through the Message Flow (Router, PipelineContextImpl, PipelineStage....).


The problem is that there is no means to grab a reference to a collection of the running messages with associated proxy. We can't even override the implementation of these classes, because OSB doesn't use Dependency Injection, and implementations are instantiated with a new().

The only way to log these messages being processed is either using some instrumentation to intercept the static MessageProcessor.processRequest(RouterContext context) call (like BTrace), or by explicitely call a Java Callout / Custom XPath at the beginning of each Proxy, logging to a file the Thread Name, the Proxy Service and the messageId.

Caveat: if you use a Route node, the thread used to process the response is different from the one used for the request. You should use intercept also MessageProcessor.processResponse()

I have also been thinking of storing all that info (messageID, proxy) in a ThreadLocal variable, and use log4j MDC to log that info in the WebLogic logs. But again this requires a Java Callout.

One could also change the thread name by appending the Proxy Name http://stackoverflow.com/questions/5532864/thread-setnamename-caveats

See also https://forums.oracle.com/thread/2449899

PL/SQL: using INSTR and SUBSTR is a lot faster than REGEXP_SUBSTR

Given this "labels":
"InterfaceID=Common_NCRS;TechnicalMessageID=Common_NCRS^ACMEPreOrder.quote_order^5505352925343722746--4337ac89.1404f3a3d07.-8bc;EventType=InvokedACME;PathOfService=Commons_NCRS/ProxyServices/Common_NCRS_PS;EventOccuredOn=2013-08-05T18:17:59.470+02:00;BusinessID=DE11PC089^d3a59751-38e4-eed5-7bf7-4f45592a9d18;ServerName=osbpr1ms2" and this labelname "EventOccuredOn", the following function extracts the value "2013-08-05T18:17:59.470+02:00" :

create or replace 
FUNCTION ACME_findLabelValue 
  (labels IN VARCHAR2, labelname IN VARCHAR2 )  return VARCHAR2
is 
begin
  return  REPLACE(REGEXP_SUBSTR(labels, labelname || '=[^;]+'), labelname || '=', '');
end;

Performance was abysmal. I have replaced with this much faster function:

create or replace 
FUNCTION ACME_findLabelValue 
  (labels IN VARCHAR2, labelname IN VARCHAR2 )  return VARCHAR2
is 
v_delimpos1 PLS_INTEGER;
v_delimpos2 PLS_INTEGER;
labels2 VARCHAR2(4000);
begin
 
  v_delimpos1 := INSTR(labels, labelname || '=' );
  if v_delimpos1 > 0 then
    labels2 := SUBSTR(labels, v_delimpos1 + 1 + LENGTH(labelname));
    v_delimpos2 := INSTR(labels2, ';' );
    return SUBSTR(labels2, 1, v_delimpos2 - 1);  
  else
    return '';
  end if;
end;

Associative Arrays in PL/SQL

When splitting/parsing strings in PL/SQL, using REGEXP can bee too computationally expensive.
Parsing a CSV string can be done more effectively with SUBSTR, INSTR and associative arrays.
Most of my inspiration is coming from this post.

SET SERVEROUTPUT ON;

DECLARE
  TYPE MSG_LABELS_TYPE IS TABLE OF VARCHAR2(400) INDEX BY VARCHAR2(50);
  v_delimpos1 PLS_INTEGER;
  v_delimpos2 PLS_INTEGER;
  p_delim1 VARCHAR2(1);
  p_delim2 VARCHAR2(1);
  INPUT_STRING VARCHAR2(500);
  v_label varchar(50);
  v_value varchar(400);
  v_result MSG_LABELS_TYPE;
  
BEGIN
  INPUT_STRING := 'InterfaceID=ACMEPIPPOConnector;TechnicalMessageID=ACMEPIPPOConnector^INVOICE^f6de3e52000001404d914d04ffff847a^7-382734;EventType=ACMEMessage For PIPPO Posted;PathOfService=ACME_CommonServices/ProxyServices/ACMECommonServices_NESOA_to_PIPPO_PS;EventOccuredOn=2013-08-05T10:22:11.765+02:00;BusinessID=7-382734;ServerName=osbpr1ms3';
  p_delim1  := ';';
  p_delim2  := '=';
  INPUT_STRING := INPUT_STRING || ';';
  v_delimpos1 := INSTR(INPUT_STRING, p_delim1);
  while v_delimpos1 > 0 and LENGTH(INPUT_STRING) > 1
  loop
    v_delimpos2 := INSTR(INPUT_STRING, p_delim2);
    v_label := SUBSTR(INPUT_STRING, 1, v_delimpos2 - 1);
    v_value := SUBSTR(INPUT_STRING, v_delimpos2 + 1, v_delimpos1 - v_delimpos2 - 1);
    v_result(v_label) := v_value;
    INPUT_STRING := SUBSTR(INPUT_STRING, v_delimpos1 + 1);
    v_delimpos1 := INSTR(INPUT_STRING, p_delim1);
  END LOOP;
  
  dbms_output.put_line('InterfaceID ' || v_result('InterfaceID'));
  dbms_output.put_line('TechnicalMessageID ' || v_result('TechnicalMessageID'));
  dbms_output.put_line('EventType ' || v_result('EventType'));
  dbms_output.put_line('PathOfService ' || v_result('PathOfService'));
  dbms_output.put_line('EventOccuredOn ' || v_result('EventOccuredOn'));
  dbms_output.put_line('BusinessID ' || v_result('BusinessID'));
  dbms_output.put_line('ServerName ' || v_result('ServerName'));
  
END;
/

More info on Collections here.

Escaping strings in Java or Groovy

Working with Strings in Java is a pain, because Java does its best to prevent you from using characters like " or \ in a string.

Escaping is a pain but there is no way around it - apart from ditching Java in favor of Groovy which is a lot more flexible.

Either you use http://commons.apache.org/proper/commons-lang/javadocs/api-2.6/org/apache/commons/lang/StringEscapeUtils.html apache commons library escapeJava(), or Groovy escapeUtils http://groovy.codehaus.org/gapi/groovy/json/StringEscapeUtils.html , or just write a little function:

def escapeString(theString) {
    return theString.replace('\\', '\\\\').replaceAll('"',"'").replaceAll('\n', '" +\n"')
    //this is the list of all replace you need to implement: {"'", "\\'"}, {"\"", "\\\""}, {"\\", "\\\\"},{"/", "\\/"}
}

and good luck.

The picture of Dorian Palm

Today I have received by mail another Palm Treo 650, to be ready to replace my current Treo which is falling apart.... the only "smart" phone I like. I hate Iphones and the likes. I like a cartesian interface, bare bone features and, most of all, a mechanical keyboard.
I bought the first (a Treo 600) some 10 years ago, when I was a wealthy, healthy man with a lot of dreams for the future.
Ever since I have suffered several MAJOR blows: I have been betrayed by the woman I loved and whose child I had supported through a costly education... I have almost lost the use of my legs for orthopedic problems, and gained permanent arthritis... herniated disk is torturing me... I have been robbed by a guy who took my confidence and then fled to Peru with my money (he was later sentenced to 6 months in jail, but he will never have to pay)... most of all, I have totally lost confidence in humanity, and I am deeply disillusioned about humans, and willing to never have confidence in one of them again.

So, comparing the 2 phones, it's like I was seeing myself in a mirror, on the left the man I used to be, on the right the one full of scars and missing pieces, still miraculously working .... I will never throw it away, even if it should fail altogether.

copy and paste not working (or only intermittently) in Firefox 22

It literally run me crazy for several weeks, until I have installed Firefox 23 beta. I am a happy man again now.