bash: redirect stdout and stderr for a block of "code"

(I was hesitant to call "code" a bash script, as it's mostly molasses of un-refactorable hieroglyphs)

vi redirtest.sh

{
plutti
plitti
} > plutti.log 2>&1

{
pippi
} > pippi.log 2>&1


{
echo ciao
} > ciao.txt

{
echo miao
} > miao.txt

The command "plutti" and "plitti" don't exist, so I will have an error "./redirtest.sh: line 2: plutti: command not found ./redirtest.sh: line 2: plitti: command not found".
But only that block will be redirected to plutti.log.
Same story for pippi.log: you have 2 separate error logs for the 2 blocks of code.
So, not necessarily redirection has to be at the whole script level, or at the single statement level.... one can group several statements in a "try/catch" block, which is cool...IMHO at least, it gives more flexibility ...

WebLogic password change

quick miscellaneous notes on the topic... it's VERY easy to get the procedure wrong... http://middlewaremagic.com/weblogic/?p=323

BEA-090078 User weblogic in security realm myrealm has had 5 invalid login attempts, locking account for 30 minutes.
See also: Problem Resetting the Weblogic User Password (Doc ID 1589360.1)

Tried to reset the weblogic users password with these steps:
  a. Stopped Weblogic Server instance.
  b. Make a backup of the LDAP folder of the admin server as well as managed
  c. Set your environment variables by running setDomainEnv.sh (UNIX):. ./setDomainEnv.sh
  e. cd to security directory in your instance.(eg: $WL_HOME/user_projects/domains/base_domain/security)
  d. Run: java weblogic.security.utils.AdminAccount weblogic password .
  e. After running the command, the file “DefaultAuthenticatorInit.ldift” will get updated.
  f. Delete the following file from “ldap” folder: DefaultAuthenticatormyrealmInit.initialized
  g. Edit the boot.properties file and change the password to the value already used on step d.

Started NodeManager and WebLogic Server. Log files show ‘Running’ but can not 
display the WLS Admin Console log in Screen.
Changes
 Updated the password for the weblogic user.

Run:
java weblogic.security.utils.AdminAccount weblogic hello .

Don’t forget the period “ .” at the end of the above command, it is required

Remember also

Reset the node manager password in WLS Admin Console. 
Reference the note: WebLogic - Getting exception in WLST "weblogic.nodemanager.NMException: 
Access to domain for user denied" (Doc ID 889842.1)
  A. Log in to the WLS Admin Console
  B. In the Domain Structure navigation window of the left click on the Domain name.
  C. Click on the 'Security' link on the right hand side
  D. Click the Advanced link near the bottom
  E. Update the 'NodeManager Password' to be the new password you created for weblogic > Save.

Book: Jenkins, the definitive guide

This is a great reading, the book covers not only Jenkins, but also the vast ecosystem around Continuous Integration (maven, nexus..); all in a very readable way, without indulging too much in the gory details.

It's even downloadable for free from the above Wakaleo URL. Have fun!

WebLogic, auditing invalid login attempts

I was getting such messages in the log, after changing weblogic password:

####<Oct 15, 2014 9:15:47 PM CEST> <Notice> <Security> <acme105> <osbpp1ms1> <[ACTIVE] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <553e43a3c186ec6d:-ae5bdb3:149153b2e29:-8000-0000000000000068> <1413400547980> <BEA-090078> <User weblogic in security realm myrealm has had 5 invalid login attempts, locking account for 30 minutes.>

I was unable to trace the origin of this invalid login, until I setup a DefaultAuditRecorder:

http://docs.oracle.com/cd/E13222_01/wls/docs90/secmanage/providers.html

myrealm Providers Auditing New

Add these :

com.bea.contextelement.channel.Address
com.bea.contextelement.channel.ChannelName
com.bea.contextelement.channel.Port
com.bea.contextelement.channel.Protocol
com.bea.contextelement.channel.PublicAddress
com.bea.contextelement.channel.PublicPort
com.bea.contextelement.channel.RemoteAddress
com.bea.contextelement.channel.RemotePort
com.bea.contextelement.channel.Secure


and restart the server. Then you do

less /opt/oracle/domains/osbpp1do/servers/osbpp1ms1/logs/DefaultAuditRecorder.log

#### Audit Record Begin <Oct 15, 2014 9:16:27 PM>  <Severity =FAILURE>  
<<<Event Type = Authentication Audit Event><weblogic><AUTHENTICATE>>>  
<FailureException =javax.security.auth.login.FailedLoginException: 
[Security:090304]Authentication Failed: User weblogic javax.security.auth.login.FailedLoginException: 
[Security:090302]Authentication Failed: User weblogic denied>
 <<<CONTEXTELEMENT: com.bea.contextelement.channel.Port:
  8001
 CONTEXTELEMENT>>>
 <<<CONTEXTELEMENT: com.bea.contextelement.channel.PublicPort:
  8001
 CONTEXTELEMENT>>>
 <<<CONTEXTELEMENT: com.bea.contextelement.channel.RemotePort:
  53443
 CONTEXTELEMENT>>>
 <<<CONTEXTELEMENT: com.bea.contextelement.channel.Protocol:
  t3
 CONTEXTELEMENT>>>
 <<<CONTEXTELEMENT: com.bea.contextelement.channel.Address:
  pippo2-osbpp1ms1.acme.com
 CONTEXTELEMENT>>>
 <<<CONTEXTELEMENT: com.bea.contextelement.channel.PublicAddress:
  pippo2-osbpp1ms1.acme.com
 CONTEXTELEMENT>>>
 <<<CONTEXTELEMENT: com.bea.contextelement.channel.RemoteAddress:
  /10.56.10.188
 CONTEXTELEMENT>>>
 <<<CONTEXTELEMENT: com.bea.contextelement.channel.ChannelName:
  Default[t3]
 CONTEXTELEMENT>>>
Audit Record End ####

So the client's address is 10.56.10.188 and the remote port is 53443. I go on that box and I do

netstat -an | grep 53443
tcp        0      0 10.56.10.188:53443           10.56.10.183:8001            ESTABLISHED


netstat --all --program | grep 53443
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 acme106.acme53443 pippo2-osbpp1ms:vcom-tunnel ESTABLISHED -

so if you are root you can find the PID of the offending process.

Mount nfs and rpcbind

I was trying to mount a NFS4 share:

mount -t nfs 10.0.2.15:/drbd/main/shared/ /opt/oracle/domains/osbpl1do/shared
mount.nfs: Connection timed out

Of course, I forgot to start the nfs service:

/etc/init.d/nfs start

Starting NFS services:                                     [  OK  ]
Starting NFS quotas: Cannot register service: RPC: Unable to receive; errno = Connection refused
rpc.rquotad: unable to register (RQUOTAPROG, RQUOTAVERS, udp).
                                                           [FAILED]
Starting NFS mountd:                                       [FAILED]
Starting NFS daemon: rpc.nfsd: writing fd to kernel failed: errno 111 (Connection refused)
rpc.nfsd: unable to set any sockets for nfsd
                                                           [FAILED]

Mmmmm, what is going on here..... I try to get some info on the rpc service:
rpcinfo -p

rpcinfo: can't contact portmapper: RPC: Remote system error - No such file or directory

hell on Earth... let me restart the rpcbind service:

service rpcbind restart

Stopping rpcbind:                                          [FAILED]
Starting rpcbind:                                          [  OK  ]

rpcinfo -p

   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper

at last I can start the nfs service: /etc/init.d/nfs start

Starting NFS services:                                     [  OK  ]
Starting NFS quotas:                                       [  OK  ]
Starting NFS mountd:                                       [  OK  ]
Starting NFS daemon:                                       [  OK  ]
Starting RPC idmapd:                                       [  OK  ]

Fun run of JClarity Censum - Garbage Collection analysis tool

I have downloaded the 14 days trial of JClarity Censum. It's EXTREMELY simple to use and gives you a quick feedback on your GC issues.

First, you HAVE to enable a few flags: -XX:+PrintTenuringDistribution and -XX:+PrintGCDetails (safe in PROD). Then you run your tests for some 24 hours and feed your GC log into Censum.

It gives you statistical info like:

Longest GC pause (I had 5 seconds, quite bad)

Percentage of time spent in GC pauses (mine is 9%, pretty bad, should be max 5%)

Average memory allocation rate (I had 90 MB / second, quite bad)

Max memory allocation rate (I had 900 MB / second, horrible)

Total Full GC pause time, and Total Pause time

Full GC to GC ratio (mine was 45%, pretty stiff)

Application throughput (percentage of time spent working, versus time spent in GC.... should be 95% or more)

The graphs are breathtaking...

All in all, they charge you a quite hefty (ANNUAL!) fee for this software... to perform analysis that doesn't seem all that complicated - I am sure I can figure out most of this stuff in Excel in a matter of hours!

However, if your time is precious, it seems a quite valuable tool...

WebLogic: check which patches are applied

Just throwing all available technology at this issue....

Method 1: grep BEA-141107 in the logs (I don't get anything)

Method 2: bsu

cd /opt3/oracle/fmw11_1_1_5/utils/bsu
./bsu.sh -report -bea_home=/opt3/oracle/fmw11_1_1_5/

Patch Report
============
  Report Info
    Report Options
      bea_home.................. /opt3/oracle/fmw11_1_1_5/
      product_mask.............. ### OPTION NOT SET
      release_mask.............. ### OPTION NOT SET
      profile_mask.............. ### OPTION NOT SET
      patch_id_mask............. ### OPTION NOT SET
    Report Messages
  BEA Home.................. /opt3/oracle/fmw11_1_1_5

  Product Description
  Product Name.............. Oracle Coherence
  Product Version........... 3.6.0.4
  Installed Components...... Coherence Product Files
  Product Install Directory. /opt3/oracle/fmw11_1_1_5/coherence_3.6
  Java Home................. null
  Jave Vendor............... null
  Java Version.............. null
  Patch Directory........... /opt3/oracle/fmw11_1_1_5/patch_ocp360

  Product Description
  Product Name.............. WebLogic Server
  Product Version........... 10.3.5.0
  Installed Components...... Core Application Server, Administration Console, Configuration Wizard and Upgrade
 Framework, Web 2.0 HTTP Pub-Sub Server, WebLogic SCA, WebLogic JDBC Drivers, Third Party JDBC Drivers, WebLogic Server
 Clients, WebLogic Web Server Plugins, UDDI and Xquery Support, Evaluation Database, Workshop Code Completion Support

  Product Install Directory. /opt3/oracle/fmw11_1_1_5/wlserver_10.3
  Java Home................. null
  Jave Vendor............... Sun
  Java Version.............. 1.6.0_24
  Patch Directory........... /opt3/oracle/fmw11_1_1_5/patch_wls1035

Method 3: opatch

export MW_HOME=/opt3/oracle/fmw11_1_1_5/
export ORACLE_HOME=/opt3/oracle/fmw11_1_1_5/osb
export JDK_HOME=/opt3/oracle/java
/opt3/oracle/fmw11_1_1_5/oracle_common/OPatch/opatch lsinv -all -jdk /opt3/oracle/java/ -invPtrLoc /opt/oracle/bin/software/oraInst_soa3.loc

Invoking OPatch 11.1.0.8.2

Oracle Interim Patch Installer version 11.1.0.8.2
Copyright (c) 2010, Oracle Corporation.  All rights reserved.


Oracle Home       : /opt3/oracle/fmw11_1_1_5/osb
Central Inventory : /opt3/oracle/orainventory
   from           : /opt/oracle/bin/software/oraInst_soa3.loc
OPatch version    : 11.1.0.8.2
OUI version       : 11.1.0.9.0
OUI location      : /opt3/oracle/fmw11_1_1_5/osb/oui
Log file location : /opt3/oracle/fmw11_1_1_5/osb/cfgtoollogs/opatch/opatch2014-10-02_12-06-03PM.log

Patch history file: /opt3/oracle/fmw11_1_1_5/osb/cfgtoollogs/opatch/opatch_history.txt


OPatch detects the Middleware Home as "/opt3/oracle/fmw11_1_1_5"

Lsinventory Output file location : /opt3/oracle/fmw11_1_1_5/osb/cfgtoollogs/opatch/lsinv/lsinventory2014-10-02_12-06-03PM.txt

--------------------------------------------------------------------------------
List of Oracle Homes:
  Name          Location
   OH827611496         /opt3/oracle/fmw11_1_1_5/oracle_common
   OH401177498         /opt3/oracle/fmw11_1_1_5/osb

Installed Top-level Products (1):

Oracle Service Bus                                                   11.1.1.5.0
There are 1 products installed in this Oracle Home.


Interim patches (1) :

Patch  12362492     : applied on Mon Sep 15 16:37:11 CEST 2014
Unique Patch ID:  14334178
   Created on 5 Dec 2011, 21:20:01 hrs PST8PDT
   Bugs fixed:
     12362492



--------------------------------------------------------------------------------

OPatch succeeded.

Method 4: look in WebLogic console, Monitoring tab, you should have a list of patches

Since the patch was applied with opatch, it can only be detected by opatch. Strange thought that no trace of this patch appears in the logs...

I assume that bsu is basically dead now, and all patches should be installed with opatch... I am not an expert though...

WebLogic: Registered more than one instance with the same objectName

We keps getting an error in the logs:

BEA-149500> (RuntimeMBeanDelegate.java:255)
        at weblogic.management.runtime.RuntimeMBeanDelegate.(RuntimeMBeanDelegate.java:215)
        at weblogic.management.runtime.RuntimeMBeanDelegate.(RuntimeMBeanDelegate.java:193)
        at weblogic.management.runtime.RuntimeMBeanDelegate.(RuntimeMBeanDelegate.java:182)

until we restarted and it went away. No change was done in the configuration recently.

Looking in Oracle Support I found:

BEA-149500 Error Registered more than one instance with the same objectName using JDBC TLOG Store (Doc ID 1544879.1)

WebLogic Server tries to create a new MBean every time a PersistentStoreRuntimeMBean is called. This issue has been fixed in unpublished defect 16063328

Our MBean type is "JMSPooledConnectionRuntime" and not "PersistentStoreRuntime", but maybe the origin is the same... anyway I am not going to apply the patch, as this has occurred only once....