Monday, June 30, 2014

Book: Extending Puppet





Alessandro "the Great" Franceschi, prominent Puppet guru, just authored this EXCELLENT book...

Not only the book is written in a fresh and speedy language, but it really covers ALL the big important topics related to managing a complex Puppet infrastructure, using the latest technology available... PuppetDB, Foreman, Hiera with Puppet 3...

However.... I am not a big fan of Puppet, I find it really outdated in its core design and choice to develop an independent DSL rather than relying on some existing language... but one thing that REALLY strikes me weird is how late the Puppet galaxy started worrying about separating configuration from code... something that in the Java world has been done since the origin of the language, with property files with all formats, databases, any pluggable datasource you can imagine... in Puppet they started worrying only in 2010, and still struggling to define standards...Infrastructure as code, wonderful, but at least give me the power of Java 8... AT LEAST ! I am not asking for Scala...



Packt gives away ANY book for 10 USD - until 5th of July

http://www.packtpub.com/10years

http://bit.ly/1mMwwSC



Why not...

Friday, June 27, 2014

SOA Suite 12c contains the "next generation" OSB

I am not going to blog over the new features of SOA Suite 12c, this is already taken care of by the excellent Jan

Installation http://jvzoggel.wordpress.com/2014/06/27/installing-oracle-soa-suite-12-1-3-for-developers/

Create Project http://jvzoggel.wordpress.com/2014/06/27/creating-a-new-oracle-service-bus-12c-project/

OSB Hello World http://jvzoggel.wordpress.com/2014/06/27/building-a-simple-oracle-service-bus-12c-helloworld-service/

Integrated WebLogic server http://jvzoggel.wordpress.com/2014/06/27/how-to-run-and-debug-oracle-service-bus-12c-services-on-the-integrated-weblogic-server/

Here starting from minute 4 you can actually see a living demo of OSB in Jdeveloper... the view is familiar to SOA Suite developer...




Anyway lot of information can be obtained on the official Oracle page


Also, quite useful this illustration of the new features and comparison with 11g

http://niallcblogs.blogspot.ch/2014/06/1-soa-12c-new-features-overview.html
http://niallcblogs.blogspot.ch/2014/06/2-soa12c-new-features-project-structure.html
http://niallcblogs.blogspot.ch/2014/06/4-soa-12c-new-features-osb-design-time.html

etc etc

and Maven support http://biemond.blogspot.ch/2014/06/maven-support-for-1213-service-bus-soa.html

and dependency visualization http://technology.amis.nl/2014/06/28/soa-suite-12c-exploring-dependencies-visualizing-dependencies-between-soa-artifacts/

and XQuery libraries http://technology.amis.nl/2014/06/28/soa-suite-12c-support-for-reusable-xquery-libraries-modules/



Vagrant snapshot

http://priyaaank.tumblr.com/post/50707609769/snapshotting-vagrant

I have just discovered there are PLUGINS in Vagrant (do "vagrant plugin list" to know what you have installed), and that you can take snapshots of a box with Vagrant 1.6.3 (it was not available with my previous Vagrant version) and doing "vagrant plugin install vagrant-vbox-snapshot"

I never really needed this feature, but it's cool to know it exists.

Securing OSB SOAP service with WS-Policy UsernameToken

Since we had some issue with OWSM and Attachments (solved in 11.1.1.7, but we are on 11.1.1.5) we decided to use some Predefined Web Service Security Policy.

The proxy has this WSDL :

<wsdl:definitions name="PVHelloWorld" 
targetNamespace="http://www.example.org/PVHelloWorld/" 
xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/" xmlns:tns="http://www.example.org/PVHelloWorld/" 
xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:wsp="http://schemas.xmlsoap.org/ws/2004/09/policy"
   xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">

  <wsdl:types>
 <xsd:schema>
  <xsd:import namespace="http://www.example.org/PVHelloWorld/" schemaLocation="PVHelloWorld.xsd"/>
 </xsd:schema>
  </wsdl:types>
  
  <wsdl:message name="HelloWorldRequest">
    <wsdl:part element="tns:HelloWorldRequest" name="parameters"/>
  </wsdl:message>
  
  <wsdl:message name="HelloWorldResponse">
    <wsdl:part element="tns:HelloWorldResponse" name="parameters"/>
  </wsdl:message>
  
  <wsdl:portType name="PVHelloWorldPortType">
    <wsdl:operation name="HelloWorld">

      <wsdl:input message="tns:HelloWorldRequest">

</wsdl:input>
      <wsdl:output message="tns:HelloWorldResponse"/>
    </wsdl:operation>
  </wsdl:portType>
  
  <wsdl:binding name="PVHelloWorldPortBinding" type="tns:PVHelloWorldPortType">

    <soap:binding style="document" transport="http://schemas.xmlsoap.org/soap/http"/>
    <wsdl:operation name="HelloWorld">
      <soap:operation soapAction="http://www.example.org/PVHelloWorld/HelloWorld"/>
      <wsdl:input>
        <soap:body use="literal"/>
      </wsdl:input>
      <wsdl:output>
        <soap:body use="literal"/>
      </wsdl:output>
    </wsdl:operation>
  </wsdl:binding>
  
  <wsdl:service name="PVHelloWorld">
    <wsdl:port binding="tns:PVHelloWorldPortBinding" name="PVHelloWorldPort">
      <soap:address location="http://www.example.org/"/>
    </wsdl:port>
  </wsdl:service>
  
</wsdl:definitions>



where the PVHelloWorld.xsd is:
<?xml version="1.0" encoding="UTF-8"?>
<schema xmlns="http://www.w3.org/2001/XMLSchema" 
 targetNamespace="http://www.example.org/PVHelloWorld/" 
 xmlns:tns="http://www.example.org/PVHelloWorld/" 
 elementFormDefault="qualified">

    <element name="HelloWorldRequest" type="string"></element>

    <element name="HelloWorldResponse" type="string"></element>
    
</schema>


After you have to apply security to Request only.


If you apply Security to Operation, then it tries to apply security also to Response, and it fails:
####<Jun 27, 2014 10:23:40 AM CEST> <Error> <OSB Security> <acme102> <osbdev1ms1> <[ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <7f9b72b69446518a:670088d3:146d8a74ad2:-8000-0000000000004f16> <1403857420200> <BEA-387023> <An error ocurred during web service security inbound response processing [error-code: Fault, message-id: 6720812917736772435-670088d3.146d8a74ad2.-7882, proxy: PVSecurityTest/PVHelloWorld, operation: HelloWorld]
--- Error message:
<env:Envelope xmlns:env="http://schemas.xmlsoap.org/soap/envelope/"><env:Header/><env:Body><env:Fault><faultcode>env:Server</faultcode><faultstring>Unable to add security token for identity</faultstring></env:Fault></env:Body></env:Envelope>
weblogic.xml.crypto.wss.WSSecurityException: Unable to add security token for identity
        at weblogic.wsee.security.wss.SecurityPolicyDriver.doIdentity(SecurityPolicyDriver.java:193)
        at weblogic.wsee.security.wss.SecurityPolicyDriver.processIdentity(SecurityPolicyDriver.java:162)
        at weblogic.wsee.security.wss.SecurityPolicyDriver.processOutbound(SecurityPolicyDriver.java:74)
        at weblogic.wsee.security.wss.SecurityPolicyDriver.processOutbound(SecurityPolicyDriver.java:65)
        at weblogic.wsee.security.WssServerHandler.processOutbound(WssServerHandler.java:88)
        at weblogic.wsee.security.WssServerHandler.processResponse(WssServerHandler.java:70)
        at weblogic.wsee.security.WssHandler.handleResponse(WssHandler.java:127)
        at com.bea.wli.sb.security.wss.wls.Wls92InboundHandler.processResponse(Wls92InboundHandler.java:261)
        at com.bea.wli.sb.security.wss.WssHandlerImpl.doInboundResponse(WssHandlerImpl.java:918)
        at com.bea.wli.sb.context.BindingLayerImpl.createTransportReplySender(BindingLayerImpl.java:416)
        at com.bea.wli.sb.context.BindingLayerImpl.addResponse(BindingLayerImpl.java:506)
        at com.bea.wli.sb.pipeline.MessageProcessor.finishProcessing(MessageProcessor.java:349)
        at com.bea.wli.sb.pipeline.RouterCallback.onReceiveResponse(RouterCallback.java:108)
        at com.bea.wli.sb.pipeline.RouterCallback.run(RouterCallback.java:183)
        at weblogic.work.ContextWrap.run(ContextWrap.java:41)
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
>




You must then customize the policy (by default, everyone can access):




and add the condition User = BLA (it's covered in red in the picture below)




The alternative is to embed policies in the WSDL:

This to be put only once:

<wsp:UsingPolicy wsdl:Required="true" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/"/>

and this to be applied for each part you want to protect with Authorization:

<wsp:Policy> <wsp:PolicyReference URI="policy:Auth.xml"/> </wsp:Policy>

The curious thing is that if Request authentication fails, the service error handler is invoked:


 <con:fault xmlns:con="http://www.bea.com/wli/sb/context">
  <con:errorCode>BEA-386201</con:errorCode>
  <con:reason>A web service security fault occurred[{http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd}FailedAuthentication][Failed to assert identity with UsernameToken.]</con:reason>
  <con:details>
    <err:WebServiceSecurityFault xmlns:err="http://www.bea.com/wli/sb/errors">
      <err:faultcode xmlns:oas="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">oas:FailedAuthentication</err:faultcode>
      <err:faultstring>Failed to assert identity with UsernameToken.</err:faultstring>
    </err:WebServiceSecurityFault>
  </con:details>
  <con:location>
    <con:path>request-pipeline</con:path>
  </con:location>
</con:fault>






Oracle File Adapter Scalable DOM

I have tried to activate streaming parsing of an xml file with OSB:


this consists in replacing oracle.tip.adapter.file.inbound.FileActivationSpec with oracle.tip.adapter.file.inbound.ScalableFileActivationSpec, and enabling streaming parsing on the File poller Proxy Service. I also set MaxFileAge to 5, to avoid picking up a file before it's completely copied to the Input folder.

I put a SMALL (3 kb file) document 608-12.xml in the input folder, and I get:

Generic Error caught while translating inbound file  in streaming mode  : 608-12.xml
java.lang.NullPointerException
        at oracle.tip.adapter.file.inbound.InboundTranslatorDelegate.xlate(InboundTranslatorDelegate.java:295)
        at oracle.tip.adapter.file.inbound.InboundTranslatorDelegate.doXlate(InboundTranslatorDelegate.java:121)
        at oracle.tip.adapter.file.inbound.ProcessorDelegate.doXlate(ProcessorDelegate.java:388)
        at oracle.tip.adapter.file.inbound.ProcessorDelegate.process(ProcessorDelegate.java:174)
        at oracle.tip.adapter.file.inbound.ProcessWork.run(ProcessWork.java:349)
        at weblogic.work.ContextWrap.run(ContextWrap.java:41)
        at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
>



According to Oracle Support KB, this should happen only for HUUUGE documents and in old versions of OSB (we use 11.1.1.5). Related thread on the forum, reporting the same issue, are unanswered.

I am afraid we shall not use this streaming option for the time being.

See also https://community.oracle.com/thread/2195895 (discussion archived without solution...)

Tuesday, June 24, 2014

ORA-00932: inconsistent datatypes: expected NUMBER got INTERVAL DAY TO SECOND

It looks like you are trying to do maths (+, -) with TIMESTAMP.

If you try to sum or subtract two timestamps, you will get:

ORA-00932: inconsistent datatypes: expected NUMBER got INTERVAL DAY TO SECOND


TIMESTAMP doesn't like that. you should CAST the TIMESTAMP to DATE:

rather than
bla - blu (where bla and blu are TIMESTAMP)
do
CAST (bla as DATE) - CAST (blu as DATE)

and you will get a NUMBER (multiply it by 3600 * 24 and you will turn it into seconds)

BUT
you will lose the millisecond info


Here you have the definition of the TIMESTAMP





Transformation from com.bea.wli.sb.sources.XmlObjectSource to com.bea.wli.sb.sources.MessageContextSource is not supported

in OSB, when using a SOAP with Attachment in a Proxy with OWSM Policies, you get this error message:

Transformation from com.bea.wli.sb.sources.XmlObjectSource to com.bea.wli.sb.sources.MessageContextSource is not supported

This is described in Oracle DOC "Combined use of SOAP with Attachments and OWSM Policies now Supported in OSB 11.1.1.7 (Doc ID 1347447.1)".

The solution is: upgrade to OSB 11.1.1.7. Or give up attachments. Or give up OWSM. Ok don't tell me I am stupid because I already know.

Monday, June 23, 2014

git error: object directory does not exist; check .git/objects/info/alternates.

Being an idiot, I have deleted the entire content of the C:\Users\nnkrepelja\.r10k\git folder.... with dire consequences, each time i do a "git add" I get the error
git error: object directory does not exist; check .git/objects/info/alternates.
Restoring the content of the cache fixes the issue.
Question: what is this mysterious .git/objects/info/alternates ?
Here http://git-scm.com/docs/gitrepository-layout.html it claims that you can borrow objects from other repositories.... sounds a bit iffy, and it's not my case really...
Git is very powerful but it takes a looong learning curve....in the meantime, don't play around too much, you touch it you break it...


Sunday, June 22, 2014

Book: Integration Testing from the Trenches


A colleague of mine, Nicolas Frankel, wrote this handbook illustrating several popular Testing frameworks (Mockito, DBUnit, TestNG....) . It provides some sample code for each technology, which makes it easy to get started on the topic.

Also on InfoQ!



Book: Jews against the Ghetto

I have just read the excellent book "Anarchists against the Wall"  - it's really a pity that they chose this unpalatable name for an organization which should be first of all a-political, I really believe that the original denomination "Jews against the Ghetto" would have been a lot more marketable....



I think everyone who is interested in Jewish history should read this book - and learn how hard it is to dissent in a country so ideologically, religiously and militarily biased - ok comparison with Nazi Germany is all too easy so let's skip it this time.


Uri Gordon is author of another book "Anarchy alive".




Saturday, June 21, 2014

Book: creating development environments with vagrant

http://www.packtpub.com/creating-development-environments-with-vagrant/book



This is a really cool, down to earth, getting started tutorial guiding you through some simple configurations with Vagrant and Puppet/Chef.

A few recipes are shown to
- create your own Virtual Box
- configure the network
- install apache, mysql etc etc

One should  really go through this book when getting started with Vagrant and Puppet, great value for the money.



 




Top ten security breaches , by OWASP

Recently a friend of mine took an interview as IT Architect. He was hammered with questions about top security breaches, and they are all listed here on the OWASP site.

  • Injection
  • Broken Authentication and Session Management
  • Cross-Site Scripting (XSS)
  • Insecure Direct Object References
  • Security Misconfiguration
  • Sensitive Data Exposure
  • Missing Function Level Access Control
  • Cross-Site Request Forgery (CSRF)
  • Using Components with Known Vulnerabilities
  • Unvalidated Redirects and Forwards


I found it very useful reading those wikis...

OWASP is a non-profit organization (Open Web Application Security Project) . I found it interesting that there are many women at its head.... men probably are too busy making money or hacking code...



Book: Storms of My Grandchildren



Author James Hansen does a decent job at unraveling the intricacies of climate science, sometimes one gets lost in the intricacies of the technicalities.... yet especially at the end of the book the message is loud and clear: "business as usual" means "total destruction of any form of life on this planet, within less than 100 years". And, most of all, stay away from COAL.

The short science-fiction story at the end of the book is really remarkable, Asimov could not have done better.

I don't share the author's opinion that Nuclear Power can be a solution, to me the only mitigation we can apply is a) population control 2) return to an essential, frugal lifestyle less focused on consumerism and more on social, cultural and spiritual values. Of course this shall never happen and Earth will become like Venus within 100 years - we are just at the beginning of a very accelerated warming path which will take a really dramatic pace within 10-15 years.



Thursday, June 19, 2014

Cygwin tricks with Ruby

First thing, install the Cygwin version of Ruby:



during Cygwin setup, otherwise if you try to run the Windows version of Ruby you will get:

C:\Ruby200-x64\bin\ruby.exe: No such file or directory -- /cygdrive/c/Ruby200-x64/bin/r10k (LoadError)

This is BAD:
$ ruby --version
ruby 2.0.0p353 (2013-11-22) [x64-mingw32]

This is GOOD:
$ ruby --version
ruby 1.9.3p545 (2014-02-24) [x86_64-cygwin]

And, to be on the safe side, remove the Windows Ruby from the PATH.

If when you try to install r10k

gem install r10k

you get this error:


/usr/lib/ruby/1.9.1/net/http.rb:763: [BUG] rb_sys_fail(connect(2)) - errno == 0

...
You may have encountered a bug in the Ruby interpreter or extension libraries.
Bug reports are welcome.
For details: http://www.ruby-lang.org/bugreport.html
Aborted (core dumped)



then try first to configure the proxy inside Cygwin:

export http_proxy=http://yourusername:yourpassword@proxy.acme.com:8080


Wednesday, June 18, 2014

bash scripts, success and failure

All this might be trivial, but it's good to have it written down somewhere.


false and true can be used to simulate a command which failed or succeeded(respectively)

false
echo $?
1
true
echo $?
0

You can use || to run a command only if the previous failed:

false || echo "it failed"
it failed 

true || echo "it didn't fail"
(nothing is printed here) 


You can use && to run a command only if the previous succeeded:
true && echo "It succeeded"
It succeeded
false && echo "It succeeded"
(nothing is printed here)  


You can chain as many command as you want with && and || (normally you do this only with &&).
 
set -e  will make sure your script stops execution as soon as a command returns something != 0
 



Sunday, June 8, 2014

Lucene and ElasticSearch clustering

https://blog.liip.ch/archive/2013/07/19/on-elasticsearch-performance.html this is a concise and excellent presentation on clustering and LB

https://vimeo.com/26710663 Shay Banon talks about Shards and Replica synchronization (ElasticSearch is at minute 25)

Friday, June 6, 2014

Resistance is futile

Today a friend told me he went for a permanent position in a large, bureaucratic, very political organization. I immediately thought of....

The Borg...



We are Borg
You will be assimilated
I am Locutus, a Borg
resistance is futile
Your life as it has been
is over
from this time forward
you will service US
You will be assimilated
resistance is futile
We are Borg
Lower your shields and surrender your ships
we will add your biological and technological distinctiveness to our own
Your culture will adapt to service us.
Resistance is futile.


http://en.wikipedia.org/wiki/Borg_%28Star_Trek%29

Thursday, June 5, 2014

SQLDeveloper: exporting/importing of CLOB data (XML)

Ok, I know that SQLDeveloper should not be the tool for this job, but it's so convenient, readily available and flexible.


One can export to a SQL INSERT file, but the CLOB is not exported.
One can use TEXT export to a TSV file, but if the XML contains quotes, you are screwed.
One can use XML export, it's perfect because each field is wrapped in a CDATA, but when importing it fails with a "there are no readers registered for the xml data type" (what??? then why do you offer me a XML export if you can't import???)

I find the XML format very convenient, so I think I will write a Python script to import it to a DB.

Here I read that the XML reader is in the pipe for Oracle to develop.... the issue is that I will probably retire sooner that that.

Surge of site traffic from Ukraine



What is going on.... surely nothing "normal"... anyway our friends from Ukraine are welcome, unless they are part of some paramilitary forces involved in the killing of innocent civilians.



Tuesday, June 3, 2014

unpack of a OSB domain fails if OSB is not installed (of course...)

The unpack of a OSB domain template will fail with:

Required component home not found for component 'oracle.as.jrf' version '11.1.1.7.0'"

I think at the origin of the issue is that it's expecting to find the component home in this file:
/opt/oracle/middleware11g/Oracle_OSB1/inventory/ContentsXML/comps.xml

which of course is not available of OSB binaries are not installed.

One would have appreciated a message like "unable to proceed, OSB not installed", but no, life would be too boring of all was perfect clean and intuitive.



sql connect by level

Sometimes it's useful to generate in SQL a series of data responding to a certain criteria.

Like "the integers between 0 and N", or "the dates between today and 5 days ago"

You can do it with "connect by level":

SELECT level from DUAL connect by level <= 10 ;

SELECT TRUNC((sysdate-5 + (LEVEL))) AS DATES FROM DUAL connect by level <= ( sysdate-(sysdate-5) ) order by level;


Monday, June 2, 2014

Weird Hiera related error messages in Puppet

A colleague had a YAML file not properly formatted, and he kept getting this error message:

Could not retrieve catalog from remote server: Error 400 on SERVER: syntax error on line 16, col -1: `' at /etc/puppetlabs/puppet/environments/dev/modules/introscope/manifests/pippo.pp:22 on node mynode.acme.com

at line pippo.pp:22 there is a call to hiera

One would say that hiera is so poorly coded that it doesn't even bother to log that there is a YAML parsing error... I remember working in GWBasic 30 years ago and most errors were reported in the same way, "syntax error"... but it was 30 years ago.... in fact working with Puppet I have often this 1980 feeling...

Sunday, June 1, 2014

Really cool "getting started with Scala" videos

I really like the style of this guy: no blabla, straight to the point, show me the code.