What to monitor in Unix/Linux using Sitescope

a good start would be:

Load Average\1minAvg
Memory\MemFree
Memory\MemTotal
Memory\SwapFree
Memory\SwapTotal
Processor\Average\Idle
Processor\Average\System
Processor\Average\User
System Stats\Context Switches/sec

add the JMX monitor this way:

service:jmx:rmi:///jndi/iiop://yourserver:yourport/weblogic.management.mbeanservers.runtime

and wait forever. Use IE, we had trouble using Firefox.

If you receive this error message:
Error receiving browse data: org.omg.CORBA.NO_PERMISSION: vmcid: 0x0 minor code: 0 completed: No

make sure that Enable IIOP is checked (protocols/IIOP), and that the Default IIOP Username and password are the same as your weblogic user/pw.
Also, make sure you restart if you make any changes.


If you get

Error receiving browse data: java.rmi.MarshalException: CORBA COMM_FAILURE 1398079696 Maybe; nested exception is: org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 208 completed: Maybe

then add “-Dcom.sun.CORBA.transport.ORBTCPReadTimeouts=10:30000:500:10” to JVM start parameters

at command, how to specify a date

echo "echo ciao" | at 17:23 Oct 23

the trick is to put first the time and then the date

Thanks to the Linux man pages for NOT explaining in the examples something so simple yet so vital =€:o(

FileDataIndexer

weblogic.diagnostics.archive.filestore.FileDataIndexer seems to generate plenty of garbage in memory


try using
-D_Offline_FileDataArchive=true

-Dcom.bea.wlw.netui.disableInstrumentation=true


also, com.bea.wli.bpm.runtime.ProcessVariableValueImpl seems to generate a lot of garbage

java.lang.NoClassDefFoundError: com/bea/wli/ide/jpd/core/graph/node/ProcessGraphNode

This happens in BEA WebLogic Integration Administration Console
when trying to display the Graphical View of a Process....

still investigating on the issue...

Which MBeans should I monitor?

• JVMRuntimeMBean
o attribute heapFreeCurrent
o attribute heapSizeCurrent

• JRockitRuntimeMBean
o attribute jvmProcessorLoad
o attribute allProcessorsAverageLoad
o attribute totalGarbageCollectionTime

• JMSDestinationRuntimeMBean
o attribute messagesCurrentCount
o attribute messagesHighCount
o attribute messagesPendingCount
o attribute messagesReceivedCount

• JDBCConnectionPoolRuntimeMBean
o attribute activeConnectionsCurrentCount
o attribute activeConnectionsHighCount
o attribute connectionsTotalCount
o attribute leakedConnectionCount
o attribute maxCapacity
o attribute waitingForConnectionCurrentCount
o attribute waitingForConnectionHighCount
o attribute waitSecondsHighCount

• JTATransactionStatisticsRuntime
o attribute transactionTotalCount
o attribute transactionCommittedTotalCount
o attribute transactionRolledBackTotalCount
o attribute transactionHeuristicsTotalCount
o attribute secondsActiveTotalCount
o attribute transactionAbandonedTotalCount
o attribute transactionRolledBackAppTotalCount
o attribute transactionRolledBackResourceTotalCount
o attribute transactionRolledBackSystemTotalCount
o attribute transactionRolledBackTimeoutTotalCount

• EJBTransactionRuntimeMBean
o attribute transactionsCommittedTotalCount
o attribute transactionsRolledBackTotalCount
o attribute transactionsTimedOutTotalCount

• EJBPoolRuntimeMBean
o attribute beansInUseCount
o attribute idleBeansCount
o attribute timeoutTotalCount
o attribute waiterTotalCount

When to use a JMS bridge and when a SAF agent

SAF agents are much more performant, in terms of latency and throughput.
The only advantage of Bridges is the interoperability with 3rd party products and with previous releases of WebLogic servers, and it's supported also in pre-9 releases.

Difference between NoClassDefFoundError and ClassNotFoundException

http://java.sun.com/j2se/1.5.0/docs/api/java/lang/NoClassDefFoundError.html

"The searched-for class definition existed when the currently executing class was compiled, but the definition can no longer be found. "


http://java.sun.com/j2se/1.5.0/docs/api/java/lang/ClassNotFoundException.html

"but no definition for the class with the specified name could be found. "


The explanation given by the javadocs is very misleading.
In reality, NoClassDefFoundError is thrown when the class CAN be found, but there was an exception in the static initializers. The naming strategy chosen by Sun is VERY misleading... one tries to fix the classpath, while the problem is in the class definition itself.

Debugging Tuxedo (WTC) in WebLogic

no "monitoring" tag is available for WTC Services.

there are some debug flags for the server:

DebugWTCCorbaEx
DebugWTCConfig
DebugWTCTDomPdu
DebugWTCUData
DebugWTCGwtEx
DebugWTCJatmiEx
DebugWTCtBridgeEx

jrcmd stops working when jvm is in outofmemory

jrcmd 25629 help
25629:
25629: Unable to open socket file: target process not responding or HotSpot VM not loaded


this while on the Weblogic Server we had:

<BEA-000110> <Multicast socket receive error: java.lang.OutOfMemoryError: mmAllocObject - Object size: 144
java.lang.OutOfMemoryError: mmAllocObject - Object size: 144
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2395)
at java.lang.Class.getDeclaredMethod(Class.java:1907)
at java.io.ObjectStreamClass.getInheritableMethod(ObjectStreamClass.java:1321)
at java.io.ObjectStreamClass.access$2200(ObjectStreamClass.java:52)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:432)
at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:400)
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:531)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
at weblogic.cluster.MulticastReceiver.dispatch(MulticastReceiver.java:109)
at weblogic.cluster.MulticastManager.run(MulticastManager.java:469)
at java.lang.Thread.run(Thread.java:595)

Shell Script to build a classpath dynamically with all jars in a directory

copied from http://twit88.com/blog/2008/02/21/unix-shell-script-to-build-classpath-dynamically/ (thanks!)

#!/bin/sh

buildClassPath() {
jar_dir=$1
if [ $# -ne 1 ]; then
echo "Jar directory must be specified."
exit 1
fi
class_path=
c=1
for i in `ls $jar_dir/*.jar`
do
if [ "$c" -eq "1" ]; then
class_path=${i}
c=2
else
class_path=${class_path}:${i}
fi
done
echo $class_path
#return $class_path
}

CP=`buildClassPath /tmp/lib`
echo $CP

A simplified version:

COH_CLASSPATH=.:$SL_CACHE_CONF_DIR:$COHERENCE_HOME/lib/coherence.jar
for i in `ls $SL_CACHE_LIB_DIR/*.jar`
do
COH_CLASSPATH=${COH_CLASSPATH}:${i}
done

find class or classes in jar for Unix or Windows

http://www.inetfeedback.com/jarscan/ on Unix


Examples ...
# To display the help message type:
java -jar jarscan.jar -help OR java -jar jarscan.jar
# To find the class ArrayList starting from the root of your c: drive type:
java -jar jarscan.jar -dir c:\ -class ArrayList



http://www.idesksoft.com/classfinder.html on Windows


Also, this command works on Linux:

find / -name '*.jar' 2> /dev/null | while read jarfile; do if jar tf "$jarfile" 2> /dev/null | grep FabricMeshUtils; then echo "$jarfile"; fi; done

(maybe you want to add a "| grep jar", otherwise it returns also .class files.

To dump all classes preceded by their jar file name:
find /acme/appsrv/bin/wl12.1a/ -type f -name '*.jar' | while read jarfile; do echo "$jarfile"; /acme/java/jdk170_71-64b/bin/jar -tf "$jarfile" ; done > /tmp/alljars.txt

Last but not least the great jar-explorer https://code.google.com/p/jar-explorer/wiki/HowToInstall, or rather https://github.com/javalite/jar-explorer. just run:

java -jar jarexplorer-0.7-BETA.jar

and remember that to search a class with package names you have to use the / and not the . as separator.

This command works also wonderfully:

find /cs/cfw/ -name "*.jar" -exec grep -Hls MyClass {} \;

WebLogic Tuning tips (from theWebLogic performance tips)

NETWORK:

see also http://fasterdata.es.net/TCP-tuning/linux.html

- Adjust the wait_time and the TCP queue size

- do a cat /etc/sysctl.conf to view system parameters:

# Kernel sysctl configuration file for Red Hat Linux
#
# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
# sysctl.conf(5) for more details.

# Controls IP packet forwarding
net.ipv4.ip_forward = 0

# Controls source route verification
net.ipv4.conf.default.rp_filter = 1

# Do not accept source routing
net.ipv4.conf.default.accept_source_route = 0

# Controls the System Request debugging functionality of the kernel
kernel.sysrq = 0

# Controls whether core dumps will append the PID to the core filename
# Useful for debugging multi-threaded applications
kernel.core_uses_pid = 1

# Controls the use of TCP syncookies
net.ipv4.tcp_syncookies = 1

# Controls the maximum size of a message, in bytes
kernel.msgmnb = 65536

# Controls the default maxmimum size of a mesage queue
kernel.msgmax = 65536

# Controls the maximum shared segment size, in bytes
kernel.shmmax = 68719476736

# Controls the maximum number of shared memory segments, in pages
kernel.shmall = 4294967296

#
#
#

net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_keepalive_time = 1800
net.ipv4.tcp_window_scaling = 0
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_fin_timeout = 20
net.ipv4.ip_local_port_range = 1024 65000
kernel.msgmni = 512
fs.file-max = 2048000
kernel.sem = 250 32000 100 128
kernel.shmmax = 536870912
net.core.rmem_default = 262144
net.core.rmem_max = 262144
net.core.wmem_default = 262144
net.core.wmem_max = 262144





- netstat -s -p --tcp displays plenty of statistics on the state of networking:

IcmpMsg:
InType0: 520
InType3: 17210
InType8: 4282432
InType11: 2285
OutType0: 4282415
OutType3: 14683
OutType8: 520
Tcp:
13494422 active connections openings
16851165 passive connection openings
3174513 failed connection attempts
2989658 connection resets received
97 connections established
2032708482 segments received
1797592474 segments send out
2034074 segments retransmited
0 bad segments received.
19698882 resets sent
TcpExt:
1599568 invalid SYN cookies received
33244 resets received for embryonic SYN_RECV sockets
717 packets pruned from receive queue because of socket buffer overrun
34 ICMP packets dropped because they were out-of-window
8074204 TCP sockets finished time wait in fast timer
1470 time wait sockets recycled by time stamp
31993765 delayed acks sent
81156 delayed acks further delayed because of locked socket
Quick ack mode was activated 194244 times
1465773 times the listen queue of a socket overflowed
1465773 SYNs to LISTEN sockets ignored
617478244 packets directly queued to recvmsg prequeue.
1866789567 packets directly received from backlog
2618179228 packets directly received from prequeue
764515521 packets header predicted
53336874 packets header predicted and directly queued to user
214677769 acknowledgments not containing data received
1325675260 predicted acknowledgments
131990 times recovered from packet loss due to fast retransmit
Detected reordering 15 times using reno fast retransmit
3 congestion windows fully recovered
2 congestion windows partially recovered using Hoe heuristic
3452 congestion windows recovered after partial ack
0 TCP data loss events
9153 timeouts after reno fast retransmit
13533 timeouts in loss state
54353 fast retransmits
1086481 retransmits in slow start
665969 other TCP timeouts
TCPRenoRecoveryFail: 103601
40848 times receiver scheduled too late for direct processing
43658 packets collapsed in receive queue due to low socket buffer
3483274 connections reset due to unexpected data
719921 connections reset due to early user close
17585 connections aborted due to timeout
IpExt:
InMcastPkts: 22414134
OutMcastPkts: 13165007
InBcastPkts: 10954581



- monitor OS for swaps (vmstat)

vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 2638660 97288 36468 922356 7 4 36 23 0 1 5 0 93 1 0


- in the JRockit Mission Control JRA recordings, watch out for weblogic.utils.io.Chunk , they can be harbingers of need to adjust the weblogic.utils.io.chunkpoolsize parameter



place one file store per disk if you have non-RAID disks

if you notice latency in the Connection Pool, use the property Pinned-To-Thread to associate a connection to a thread

Performance Analysis for Java Websites

http://www.amazon.com/Performance-Analysis-Java-TM-Websites/dp/0201844540

this book is really great, it tells you EVERYTHING without wasting your mind into too intricated issues. It's a bit old but still very valuable.

Quick and dirty Java program to print statistics on execution times

the statistics files have this format:
2009-09-30 23:10:55,432 userParameters.facade - getString 1254345055430 1254345055432


and they are in 2 different dirs, acmeappli1 and acmeappli2, one per application server





package com.acme.stats;

import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.HashMap;

/**
* Prints stats on the statistics.log
* @author vernetto
*
*/

public class StatsReport {
//static String BASE_DIR = "/users/acme/stats";
static String BASE_DIR = "D:/pierre/stats";
static String[] SUBDIRS = new String[] {"acmeappli1", "acmeappli2"};
static String FILE_NAME_FILTER = "statistics.log";

/** the list of all entries */
EntryList entryList = new EntryList();
StatEntryList statEntryList = new StatEntryList();

SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");

/** main entry point
* @throws IOException
* @throws ParseException */
public static void main(String[] args) throws IOException, ParseException {
StatsReport report = new StatsReport();
report.run();
}

/**
* Load all data into memory
* @throws IOException
* @throws ParseException
*/
private void run() throws IOException, ParseException {
for (String subdir : SUBDIRS) {
File dir = new File(BASE_DIR + File.separatorChar + subdir);
File[] statFiles = dir.listFiles();
for (File statFile : statFiles) {
if (statFile.getName().startsWith(FILE_NAME_FILTER)) {
parseFile(subdir, statFile);
}
}
}
printStats();

}

/**
* Elaborate all entries and produce report
*/

private void printStats() {
Date minDate = null;
Date maxDate = null;
for (Entry entry : entryList) {
if (minDate == null || entry.date.before(minDate)) {
minDate = entry.date;
}
if (maxDate == null || entry.date.after(maxDate)) {
maxDate = entry.date;
}

StatEntry el = statEntryList.find(entry.clazz, entry.service);
if (el == null) {
el = new StatEntry(entry.clazz, entry.service);
statEntryList.add(el);
}
el.count++;
long elapsed = entry.getElapsed();
el.cumulated += elapsed;
if (el.max == (-1) || el.max < elapsed) {
el.max = elapsed;
}
if (el.min == (-1) || el.min > elapsed) {
el.min = elapsed;
}

}

log("mindate = " + minDate + " maxdate=" + maxDate);
System.out.println(statEntryList.toReport());

}

private void log(String string) {
System.out.println(string);

}

/**
* Load a single file into memory
* @param statFile
* @throws IOException
* @throws ParseException
*/
private void parseFile(String appli, File statFile) throws IOException, ParseException {
BufferedReader input = new BufferedReader(new FileReader(statFile));
String line = null;
while (( line = input.readLine()) != null) {
//2009-09-30 23:10:55,338 advertisement.facade - US_ADMINISTRATOR_UA searchMktAdvertisements 1254345055220 1254345055338
line = line.replace("", "");
String[] splits = line.split("\\s");
String dateTime = splits[0] + " " + splits[1];
// remove the milliseconds
int indexMillis = dateTime.indexOf(",");
if (indexMillis > 0) {
dateTime = dateTime.substring(0, indexMillis);
}
Date date = sdf.parse(dateTime);
String clazz = splits[2];
String user = splits[4];
String service = splits[5];
String start = splits[6];
String stop = splits[7];
long elapsed = Long.parseLong(stop) - Long.parseLong(start);

Entry entry = new Entry(appli, date, clazz, user, service, elapsed);
entryList.add(entry);
}
}


/**
* Represents an entry in the statistics.log file
* @author vernetto
*
*/

static class Entry {
String appli;
Date date;
String clazz;
String user;
String service;
long elapsed;
public Date getDate() {
return date;
}
public void setDate(Date date) {
this.date = date;
}
public String getClazz() {
return clazz;
}
public void setClazz(String clazz) {
this.clazz = clazz;
}
public String getUser() {
return user;
}
public void setUser(String user) {
this.user = user;
}
public String getService() {
return service;
}
public void setService(String service) {
this.service = service;
}
public long getElapsed() {
return elapsed;
}
public void setElapsed(long elapsed) {
this.elapsed = elapsed;
}

/** Constructor */
public Entry(String appli, Date date, String clazz, String user, String service,
long elapsed) {
super();
this.date = date;
this.clazz = clazz;
this.user = user;
this.service = service;
this.elapsed = elapsed;
}

public String toString() {
final String SEP = " ";
StringBuilder sb = new StringBuilder();
sb.append(date.toString()).append(SEP).append(clazz).append(SEP).
append(user).append(SEP).
append(service).append(SEP).
append(elapsed).append(SEP);

return sb.toString();
}
}

/**
* Represents a collection of entries in the statistics.log file
* @author vernetto
*
*/

static class EntryList extends ArrayList {

}


static class StatEntry {
String clazz;
String service;
long count = 0;
long cumulated = 0;
long max = -1;
long min = -1;

public StatEntry(String clazz, String service) {
this.clazz = clazz;
this.service = service;
}

public String getClazz() {
return clazz;
}


public void setClazz(String clazz) {
this.clazz = clazz;
}


public String getService() {
return service;
}


public void setService(String service) {
this.service = service;
}


public long getCount() {
return count;
}


public void setCount(long count) {
this.count = count;
}


public long getCumulated() {
return cumulated;
}


public void setCumulated(long cumulated) {
this.cumulated = cumulated;
}


/**
* Constructs a String with all attributes
* in name = value format.
*
* @return a String representation
* of this object.
*/
public String toString()
{
final String TAB = " ";

String retValue = "";

retValue = "class = " + this.clazz + TAB
+ "service = " + this.service + TAB
+ "count = " + this.count + TAB
+ "cumulated = " + this.cumulated + TAB
+ "average = " + (this.cumulated / this.count) + TAB
+ "min = " + this.min + TAB
+ "max = " + this.max
;

return retValue;
}

public static String getHeader() {
return "class service count cumulated average min max";
}

public Object toReport() {

return this.clazz + " " + this.service + " " + this.count + " " + this.cumulated + " " + (this.cumulated / this.count)
+ " " + this.min + " " + this.max;
}

}


static class StatEntryList extends HashMap {

public StatEntry find(String clazz, String service) {
StatEntry result = this.get(clazz + " " + service);
return result;
}

public String toReport() {
StringBuilder sb = new StringBuilder();
sb.append(StatEntry.getHeader()).append("\n");
for (StatEntry entry : this.values()) {
sb.append(entry.toReport()).append("\n");
}

return sb.toString();
}

public void add(StatEntry el) {
this.put(el.clazz + " " + el.service, el);
}

public String toString() {
StringBuilder sb = new StringBuilder();
for (StatEntry statEntry : this.values()) {
sb.append(statEntry.toString());
sb.append("\n");
}
return sb.toString();
}

}

}