javax.net.ssl.SSLHandshakeException: null cert chain

Some HtTP 500 was being generated in a webapp.
by enabling these flags

-Djavax.net.debug=ssl:handshake
-Dssl.debug=true
-Dweblogic.log.StdoutSeverity=Debug
-Dweblogic.StdoutDebugEnabled=true
-Dwls.debug.https=true

we discovered this error:

weblogic.socket.Muxer']]weblogic.security.SSL.jsseadapter: SSLENGINE: SSLEngine.unwrap(ByteBuffer,ByteBuffer[]) called: result=Status = OK HandshakeStatus = NEED_TASK
bytesConsumed = 12 bytesProduced = 0.> 
*** Certificate chain
***
ExecuteThread: '1' for queue: 'weblogic.socket.Muxer', fatal error: 42: null cert chain
javax.net.ssl.SSLHandshakeException: null cert chain
ExecuteThread: '1' for queue: 'weblogic.socket.Muxer', SEND TLSv1 ALERT:  fatal, description = bad_certificate
ExecuteThread: '1' for queue: 'weblogic.socket.Muxer', WRITE: TLSv1 Alert, length = 2
ExecuteThread: '1' for queue: 'weblogic.socket.Muxer', fatal: engine already closed.  Rethrowing javax.net.ssl.SSLHandshakeException: null cert chain
 
 
 
 
 
<Jan 18, 2017 4:39:09 PM CET> <Debug> <SecuritySSL> <BEA-000000> <[Thread[ExecuteThread: '1' for queue: 'weblogic.socket.Muxer',5,Thread Group for Queue: 'weblogic.socket.Muxer']]weblogic.security.SSL.jsseadapter: SSLENGINE: Exception occurred during SSLEngine.wrap(ByteBuffer,ByteBuffer).
javax.net.ssl.SSLHandshakeException: null cert chain
               at com.sun.net.ssl.internal.ssl.Handshaker.checkThrown(Handshaker.java:1227)
               at com.sun.net.ssl.internal.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:489)
               at com.sun.net.ssl.internal.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1165)
               at com.sun.net.ssl.internal.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1137)
               at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:450)
               at weblogic.security.SSL.jsseadapter.JaSSLEngine$1.run(JaSSLEngine.java:68)
               at weblogic.security.SSL.jsseadapter.JaSSLEngine.doAction(JaSSLEngine.java:732)
               at weblogic.security.SSL.jsseadapter.JaSSLEngine.wrap(JaSSLEngine.java:66)
               at weblogic.socket.JSSEFilterImpl.wrapAndWrite(JSSEFilterImpl.java:625)
               at weblogic.socket.JSSEFilterImpl.doHandshake(JSSEFilterImpl.java:93)
               at weblogic.socket.JSSEFilterImpl.doHandshake(JSSEFilterImpl.java:66)
               at weblogic.socket.JSSEFilterImpl.isMessageComplete(JSSEFilterImpl.java:288)
               at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:955)
               at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:897)
               at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:130)
               at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
               at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
               at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
               at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)
Caused By: javax.net.ssl.SSLHandshakeException: null cert chain
               at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:172)
               at com.sun.net.ssl.internal.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1599)
               at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:269)
               at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:257)
               at com.sun.net.ssl.internal.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1512)
               at com.sun.net.ssl.internal.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:212)
               at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:817)
               at com.sun.net.ssl.internal.ssl.Handshaker$1.run(Handshaker.java:757)
               at java.security.AccessController.doPrivileged(Native Method)
               at com.sun.net.ssl.internal.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1164)
               at weblogic.socket.JSSEFilterImpl.doTasks(JSSEFilterImpl.java:191)
               at weblogic.socket.JSSEFilterImpl.doHandshake(JSSEFilterImpl.java:97)
               at weblogic.socket.JSSEFilterImpl.doHandshake(JSSEFilterImpl.java:66)
               at weblogic.socket.JSSEFilterImpl.isMessageComplete(JSSEFilterImpl.java:288)
               at weblogic.socket.SocketMuxer.readReadySocketOnce(SocketMuxer.java:955)
               at weblogic.socket.SocketMuxer.readReadySocket(SocketMuxer.java:897)
               at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:130)
               at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)
               at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)
               at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:145)
               at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:117)

Examining WebLogic config.xml we notice that

<ssl>
      <client-certificate-enforced>false</client-certificate-enforced>
      <listen-port>32008</listen-port>
      <two-way-ssl-enabled>true</two-way-ssl-enabled>
</ssl>

and this also appears in the logs:

<Jan 20, 2017 3:07:23 PM CET> <Debug> <SecuritySSL> <BEA-000000> <[Thread[DynamicJSSEListenThread[DefaultSecure],9,WebLogicServer]]weblogic.security.SSL.jsseadapter: SSLENGINE: SSLEngine.setNeedClientAuth(boolean): value=false.>

(setNeedClientAuth should have value=true ! )
Setting the client-certificate-enforced to true fixed the issue.

Poor man's wget for Windows

Windows not only is an awfully stinking mastodon, but it even lacks the most basic tools commonly available on Linux, such as telnet and wget.

I have found a bare bones wget implementation for Java and duly simplified (I love to strip away all useless exception handling)

import java.io.InputStream;
import java.io.DataInputStream;
import java.io.BufferedInputStream;
import java.net.*;

public class JGet {
    public static void main(String[] args) throws Exception {
        if ((args.length != 1)) {
           System.err.println("\nUsage: java JGet [urlToGet]");
           System.exit(1);
        }
        String url = args[0];
        URL u;
        InputStream is = null;
        DataInputStream dis;
        String s;
        try {
           u = new URL(url);
           is = u.openStream();
           dis = new DataInputStream(new BufferedInputStream(is));
           while ((s = dis.readLine()) != null) {
               System.out.println(s);
           }
        } finally {
           is.close();
        }
    }
}

All the credits to Alexander.

WebLogic: all MS in a cluster hang while starting up.... weblogic.cluster.MemberManager.getJNDIStateDump issue

in the thread dump of both MS I see several blocked threads:

weblogic.cluster.MemberManager.getRemoteMembers
weblogic.iiop.ClusterServices.getMembers
weblogic.cluster.ClusterRuntime.clusterMembersChanged
weblogic.cluster.MemberManager.findOrCreate
plus some 150 DynamicJSSEListenThread threads....
In particular all BLOCKED threads are waiting for lock 0x000000060276f168 who is held by this getJNDIStateDump:

"[STANDBY] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f6a800024c0 nid=0x74bd runnable [0x00007f6b1f7e6000]
   java.lang.Thread.State: RUNNABLE
               at java.net.SocketInputStream.socketRead0(Native Method)
               at java.net.SocketInputStream.read(SocketInputStream.java:152)
               at java.net.SocketInputStream.read(SocketInputStream.java:122)
               at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
               at sun.security.ssl.InputRecord.read(InputRecord.java:480)
               at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:946)
               - locked <0x000000060bcf5160> (a java.lang.Object)
               at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:903)
               at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
               - locked <0x000000060bd2a9b0> (a sun.security.ssl.AppInputStream)
               at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
               at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
               at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
               - locked <0x000000060bd2a988> (a java.io.BufferedInputStream)
               at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:690)
               at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
               at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1325)
               - locked <0x000000060c29d3c8> (a sun.net.www.protocol.https.DelegateHttpsURLConnection)
               at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
               - locked <0x000000060c29d4a8> (a sun.net.www.protocol.https.HttpsURLConnectionImpl)
               at weblogic.cluster.MemberManager.getJNDIStateDump(MemberManager.java:244)
               at weblogic.cluster.MemberManager.waitForSync(MemberManager.java:222)
               at weblogic.cluster.MemberManager.waitToSyncWithCurrentMembers(MemberManager.java:182)
               - locked <0x000000060276f168> (a weblogic.cluster.MemberManager)
               at weblogic.cluster.InboundService.start(InboundService.java:52)
               at weblogic.server.AbstractServerService.postConstruct(AbstractServerService.java:78)
               at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
               at java.lang.reflect.Method.invoke(Method.java:606)
               at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1017)
               at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:388)
               at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:430)
               at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)
               at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:225)
               at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:82)
               at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)
               at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)
               - locked <0x000000060acd0028> (a java.lang.Object)
               at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:87)
               at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1162)
               at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1147)
               at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:553)
               at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)
               at weblogic.work.ExecuteThread.run(ExecuteThread.java:263)

It turned out that the SAME domain was running before on a different set of servers, and while migrating them the operator forgot to shut down the previous instances. How this could interfere with the current domain, it's still a mystery