Installing Openshift Origin on your CentOS 7 VM

sudo yum install centos-release-openshift-origin

sudo yum install wget git net-tools bind-utils iptables-services bridge-utils bash-completion origin-clients

sudo oc cluster up

at this point, you get a terrifying

-- Checking Docker daemon configuration ... FAIL
   Error: did not detect an --insecure-registry argument on the Docker daemon
   Solution:

     Ensure that the Docker daemon is running with the following argument:
      --insecure-registry 172.30.0.0/16

after some googling, I start with :

sudo oc cluster up --skip-registry-check=true

Starting OpenShift using openshift/origin:v3.6.0 ...
OpenShift server started.

The server is accessible via web console at:
    https://127.0.0.1:8443

You are logged in as:
    User:     developer
    Password: 

To login as administrator:
    oc login -u system:admin

I open the console at https://127.0.0.1:8443 (add security exception) and login with system/admin

go to overview

If you see an error in the logs "Could not resolve host: github.com", you are screwed !
Haha no, just "sudo systemctl restart docker" , then "oc start-build --from-build=yourbuildid"




create a new project, java, wildfly, copy git url, create project pvproject01

oc login https://127.0.0.1:8443

system/admin

oc project pvproject01

oc status




If you get this

[centos@localhost ~]$ oc cluster up --skip-registry-check=true 
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... FAIL
   Error: Minor number must not contain leading zeroes "09"

it simply means that OpenShift developers are morons, and you have to wait the next release 1.5 for a fix. What a pathetic mess.



I have also followed these instructions https://fedoramagazine.org/experimenting-docker-openshift/ and they seem to work:

(at the beginning I am troubleshooting VirtualBox guest additions, which is also a good thing to record)


1 sudo shutdown now
2 cd /run/media/centos/VBOXADDITIONS_5.1.30_118389/
3 ls
4 sudo ./autorun.sh
5 less /var/log/VBoxGuestAdditions.log
6 less /var/log/vboxadd-install.log
7 sudo ./autorun.sh
8 sudo reboot now
9 less /var/log/vboxadd-install.log
10 sudo yum update
11 ping google.com
12 sudo vi /etc/sysconfig/network-scripts/ifcfg-enp0s3
13 sudo reboot now
14 sudo yum update
15 cd /run/media/centos/VBOXADDITIONS_5.1.30_118389
16 sudo ./autorun.sh
17 yum install kernel-devel gcc make patch
18 sudo yum install kernel-devel gcc make patch
19 sudo reboot now
20 cd /run/media/centos/VBOXADDITIONS_5.1.30_118389
21 sudo ./autorun.sh
22 sudo reboot now
23 sudo shutdown now
24 sudo yum install docker
25 sudo systemctl enable docker
26 sudo yum install origin docker-registry
27 vi /etc/sysconfig/docker
28 sudo vi /etc/sysconfig/docker
29 sudo systemctl daemon-reload
30 sudo systemctl restart docker
31 sudo systemctl status docker
32 sudo oc cluster up
33 cd
41 sudo ln -s /home/centos/oc /usr/sbin/oc
43 sudo oc cluster up
44 sudo oc login -u system:admin
45 sudo shutdown now
46 sudo oc cluster up

Maven deploy-file for batch upload

Unfortunately in the Nexus 3.5 and 3.6 version there is no batch upload of artifacts (in Nexus 2.X it was much easier: just rsync your Maven repo and "rebuild index"

http://maven.apache.org/plugins/maven-deploy-plugin/deploy-file-mojo.html

git clone https://github.com/gabrielf/maven-samples

git clone https://github.com/ronbadur/maven-artifacts-uploader

I start nexus with https://github.com/sonatype/docker-nexus3

sample command:

mvn -e -X deploy:deploy-file -q -DpomFile=/home/centos/myrepo/org/vafer/jdependency/1.1/jdependency-1.1.pom -Dfile=/home/centos/myrepo/org/vafer/jdependency/1.1/jdependency-1.1.jar -DrepositoryId=nexus -Durl=http://localhost:8081/repository/maven-releases/ -Dpackaging=jar


maven's settings.xml should contain

<server>
      <id>nexus</id>
      <username>admin</username>
      <password>admin123</password>
    </server>

If you get "ReasonPhrase: Repository does not allow updating assets: maven-releases." , make sure you set "allow redeploy" in the Deployment policy"

If you get "Cannot deploy artifact from the local repository:" it's because your source file is inside the .m2/repository folder - which is forbidden

See Sonatype help on this topic https://support.sonatype.com/hc/en-us/articles/213465818-How-can-I-programmatically-upload-an-artifact-into-Nexus-2-

Books: Docker in Action

It's an excellent book, highly recommended, Jeff Nickoloff is a great author.

Here some notes from the exercises

docker run dockerinaction/hello_world

docker help

Usage: docker COMMAND

A self-sufficient runtime for containers

Options:
--config string Location of client config files (default "/home/centos/.docker")
-D, --debug Enable debug mode
--help Print usage
-H, --host list Daemon socket(s) to connect to
-l, --log-level string Set the logging level ("debug"|"info"|"warn"|"error"|"fatal") (default "info")
--tls Use TLS; implied by --tlsverify
--tlscacert string Trust certs signed only by this CA (default "/home/centos/.docker/ca.pem")
--tlscert string Path to TLS certificate file (default "/home/centos/.docker/cert.pem")
--tlskey string Path to TLS key file (default "/home/centos/.docker/key.pem")
--tlsverify Use TLS and verify the remote
-v, --version Print version information and quit

Management Commands:
config Manage Docker configs
container Manage containers
image Manage images
network Manage networks
node Manage Swarm nodes
plugin Manage plugins
secret Manage Docker secrets
service Manage services
stack Manage Docker stacks
swarm Manage Swarm
system Manage Docker
volume Manage volumes

Commands:
attach Attach local standard input, output, and error streams to a running container
build Build an image from a Dockerfile
commit Create a new image from a container's changes
cp Copy files/folders between a container and the local filesystem
create Create a new container
diff Inspect changes to files or directories on a container's filesystem
events Get real time events from the server
exec Run a command in a running container
export Export a container's filesystem as a tar archive
history Show the history of an image
images List images
import Import the contents from a tarball to create a filesystem image
info Display system-wide information
inspect Return low-level information on Docker objects
kill Kill one or more running containers
load Load an image from a tar archive or STDIN
login Log in to a Docker registry
logout Log out from a Docker registry
logs Fetch the logs of a container
pause Pause all processes within one or more containers
port List port mappings or a specific mapping for the container
ps List containers
pull Pull an image or a repository from a registry
push Push an image or a repository to a registry
rename Rename a container
restart Restart one or more containers
rm Remove one or more containers
rmi Remove one or more images
run Run a command in a new container
save Save one or more images to a tar archive (streamed to STDOUT by default)
search Search the Docker Hub for images
start Start one or more stopped containers
stats Display a live stream of container(s) resource usage statistics
stop Stop one or more running containers
tag Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
top Display the running processes of a container
unpause Unpause all processes within one or more containers
update Update configuration of one or more containers
version Show the Docker version information
wait Block until one or more containers stop, then print their exit codes

Run 'docker COMMAND --help' for more information on a command.






interesting demo of Portainer https://portainer.io/overview.html#demo , I have installed and run locally to manage my environment

docker run --detach --name web nginx:latest

docker run --interactive --tty --link web:web --name web_test busybox:latest /bin/sh

wget -O - http://web:80/

exit
(CTRL-P CTRL-Q to run interactively a process and then detach)

#this example is WRONG in the book!
docker run -d --name mailer dockerinaction/ch2_mailer

docker run -it --name agent --link web:insideweb --link mailer:insidemailer dockerinaction/ch2_agent

#connect to a running instance in interactive mode
docker start -i "CID"

docker restart web
docker restart mailer
docker restart agent

docker run -d --name namespaceA busybox:latest /bin/sh -c "sleep 30000"
docker run -d --name namespaceB busybox:latest /bin/sh -c "nc -l -p 0.0.0.0:80"

docker exec namespaceA ps
docker exec namespaceB ps

#creating a conflict by NOT using namespaces
docker run -d --name webConflict nginx:latest
docker logs webConflict
docker exec webConflict nginx -g 'daemon off;'

#avoiding conflicts by using namespaces
docker run -d --name webA nginx:latest
docker logs webA
docker run -d --name webB nginx:latest
docker logs webB


docker rename webA webPippo

#create is like run, but it's created in stopped state
CID=$(docker create nginx)
echo $CID

docker create --cidfile /tmp/web.cid nginx

#running 3 containers linked to each other - in reverse order
MAILER_CID=$(docker run -d dockerinaction/ch2_mailer)
WEB_CID=$(docker run -d nginx)
AGENT_CID=$(docker run -d --link $WEB_CID:insideweb --link $MAILER_CID:insidemailer dockerinaction/ch2_agent)

#check status of container
docker inspect $CID


docker search postgres


docker pull busybox:latest
docker save -o myfile.tar busybox:latest
docker rmi busybox
docker load -i myfile.tar

Kubernetes essentials

https://dzone.com/articles/under-the-hood-an-intro-to-kubernetes-architecture?edition=327492 general overview

Installing Docker on CentOS 7

from the excellent guide https://docs.docker.com/engine/installation/linux/docker-ce/centos/

sudo yum remove docker docker-common docker-selinux docker-engine
sudo yum install -y yum-utils   device-mapper-persistent-data   lvm2
sudo yum-config-manager --add-repo  https://download.docker.com/linux/centos/docker-ce.repo
sudo yum-config-manager --enable docker-ce-edge
sudo yum-config-manager --enable docker-ce-test
sudo yum install docker-ce
yum list docker-ce.x86_64  --showduplicates | sort -r
sudo systemctl start docker
sudo docker run hello-world
sudo docker run -it ubuntu bash
sudo yum makecache fast
sudo groupadd docker
echo $USER
sudo usermod -aG docker $USER
#(perform logout/login here)
groups
#"centos docker"
sudo systemctl enable docker
#"Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service."
sudo chkconfig docker on
env | grep DOCKER_HOST
#DOCKER_HOST should not be defined if you want to connect to local daemon
#to restart daemon:
sudo service docker restart

useful commands (from the presentation below):



https://www.youtube.com/watch?v=VeiUjkiqo9E&feature=youtu.be

docker ps -a
docker version
docker info
docker images

docker run ubuntu echo "hello world"

docker run -i -t ubuntu
exit

docker diff 82af4da88bd7
docker rm 042cbb043587

docker run -d ubuntu /bin/sh -c "while true; do echo hello world; date; sleep 2; done"
docker logs 7939ad46d57c
docker attach 7939ad46d57c
docker stop 7939ad46d57c

docker inspect 7939ad46d57c

docker commit -m "installed apache" 7939ad46d57c mynamespace/myimage
docker images

#login into https://hub.docker.com
docker login
docker push mynamespace/myimage

docker rmi 77bde6a39eda

#execute the Dockerfile in myfolder and build a new image
docker build -t mynamespace/myimage github.com/myrepo/myfolder

#run assigning a non default port
docker run -d -p :8000 mynamespace/myimage

here the link to shipyard https://github.com/shipyard/shipyard

Cool Tools (from JBoss Hacks)

be aware that the Kindle edition is royally scr...wed up, missing ALL the sample script in the text... what a pity... maybe the PDF version available here http://www.itbuzzpress.com/ebooks/jboss-org-hacks.html is better, no idea.

https://github.com/mgm3746/garbagecat to analyze gc logs

http://fastthread.io/index.jsp thread dump analysis

https://github.com/rparree/jboss-bash-completion jboss/wildfly bash completion

http://hawt.io/ web console rich of managing plugins

http://byteman.jboss.org/index.html byteman , bytecode manipulation tool

https://hub.docker.com/r/jboss/wildfly/ dockerhub wildfly image

https://developers.redhat.com/jboss-docker/?referrer=jbd docker jboss images

https://docs.jboss.org/wildfly/plugins/maven/latest/ wildfly maven plugin


Jboss Forge examples available here https://github.com/fmarchioni/jbosshacks

Wildfly Swarm Project Generator http://wildfly-swarm.io/generator/


https://developers.redhat.com/products/rhamt/download/ windup / migration tool to migrate from other AS to JBoss

JBoss CLI sucks

Frankly the CLI hurts the eyes... someone should make it a bit more groovish... oh in fact here there is a guide how to do it http://www.mastertheboss.com/jboss-server/jboss-script/managing-wildfly-with-groovy-shell but it's a bit raw...

some recipes here:

https://docs.jboss.org/author/display/AS71/CLI+Recipes

at the end of the book "WildFly Configuration, Deployment, and Administration - Second Edition" there is also a useful CLI cheat sheet.

All Day DevOps on October 24, 2017 starting at 8:00am GMT

https://www.alldaydevops.com

register here https://www.alldaydevops.com/register

you can even watch the recorded session later (2016 recordings are available on their side...)

WildFly Configuration, Deployment, and Administration - Second Edition

standalone.conf

http://localhost:8080/

jboss-cli.sh

./jboss-cli.sh --connect command=:shutdown

red hat system administration guide

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/pdf/System_Administrators_Guide/Red_Hat_Enterprise_Linux-7-System_Administrators_Guide-en-US.pdf



My notes:

vi /etc/locale.conf
localectl

Nexus Repository Migration

MIGRATING FROM 2.12.1 TO 2.14.5

Migrating from Nexus Repository Manager OSS 2.12.1-01 to the latest 3.0 version:

http://www.sonatype.org/nexus/2016/05/11/migrating-to-nexus-repository-3-easy-peasy/

https://www.youtube.com/watch?v=ba4VtpusRxo

beware: upgrade agent is part of 2.14 only! You should first upgrade to 2.14 then migrate to 3.0, as shown here https://www.youtube.com/watch?v=q6qkj8UiSRY

Very detailed instructions here https://help.sonatype.com/display/NXRM3/Upgrading

Download old releases here https://support.sonatype.com/hc/en-us/articles/218238798-Nexus-Repository-Manager-OSS-Download-Archives

I have installed Nexus 2.12 here: /home/centos/nexus2/nexus-2.12.0-01/bin

cd /home/centos/nexus2/nexus-2.12.0-01/bin
./nexus start

The URL is http://localhost:8081/nexus/ (in 3.X it's simply http://localhost:8081 )



In maven's settings.xml I had this

  <mirror>
        <!--This sends everything else to /public -->
        <id>nexus</id>
        <mirrorOf>*</mirrorOf>
        <url>http://localhost:8081/repository/maven-public/</url>
     </mirror>

this was valid for 3.5, in 2.12 the URL should be changed to http://localhost:8081/nexus/content/repositories/central/

I run a sample "mvn package" from "https://github.com/gabrielf/maven-samples" and I can observe that the repository "Maven Central" is filled with stuff: http://localhost:8081/nexus/service/local/feeds/recentlyCachedReleaseArtifacts, then I select the "Central" repo, "Browse Index" and I can see stuff in it (junit, hamcrest, codehaus...)

I want now to upgrade from 2.12.1 to 2.14.5 https://support.sonatype.com/hc/en-us/articles/213464198, which is the supported version to late upgrade to 3.5

I extract the 2.14.5 zip file in a temporary location and I copy the nexus-2.14.5-02 folder (NOT the sonatype-work folder!!!) to /home/centos/nexus2, so that the nexus-2.12 and nexus-2.14 share the same sonatype-work folder.

[centos@localhost nexus2]$ pwd
/home/centos/nexus2

[centos@localhost nexus2]$ ls -ltra
total 4
drwxr-xr-x.  8 centos centos  113 Dec 16  2015 nexus-2.12.0-01
drwxr-xr-x.  3 centos centos   37 Dec 16  2015 sonatype-work
drwxr-xr-x.  8 centos centos  113 Jul 25 12:39 nexus-2.14.5-02
drwx------. 33 centos centos 4096 Sep 10 19:28 ..
drwxrwxr-x.  5 centos centos   73 Sep 10 19:34 .

apparently the sonatype-work format is binary compatible among the 2.12 and 2.14.5 versions.

I stop nexus, make a backup copy of the conf folder:

cd /home/centos/nexus2
cp -R nexus-2.12.0-01/conf/ nexus-2.12.0-01/confBACKUP

apparently no further manual steps are required https://support.sonatype.com/hc/en-us/articles/213464338-Sonatype-Nexus-Upgrade-and-Compatibility-Notes

and now I start the new version of Nexus

cd nexus-2.14.5-02/bin
./nexus start

At a quick look, the content of the repository and an extra user I had created are preserved in the migration...

MIGRATING TO 3.5

Now I start nexus 3.5 side by side, on the same host, making sure I use a different number:

cd /home/centos/nexus3
grep -R 8081 *
nexus-3.5.1-02/etc/nexus-default.properties:application-port=8081

and I change that port to 18081

cd /home/centos/nexus3/nexus-3.5.1-02/bin
./nexus start

localhost:18081

things are working. I delete all pre-existing repositories

I follow all the steps as in https://help.sonatype.com/display/NXRM3/Upgrading and things work perfectly - using the "download" method (slowest)

I have tested the 3 methods and they all work, of course the file copy (hard link or not) is much faster than HTTP. If you wonder what a hard link is, read here https://askubuntu.com/questions/108771/what-is-the-difference-between-a-hard-link-and-a-symbolic-link

Nexus and Maven (ST): setup

I have installed Maven (ST) in /home/centos/apache-maven-3.5.0/.

My m2 repo is in /home/centos/.m2/repository

To test the m2 repo, I "git clone https://github.com/gabrielf/maven-samples", then "mvn package"

"ls /home/centos/.m2/repository" shows that stuff is actually pulled from the maven central repo https://repo.maven.apache.org/maven2/ , you should see plenty of this in the logs :

"Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.5/maven-resources-plugin-2.5.jar"

I edit the settings in /home/centos/apache-maven-3.5.0/conf/settings.xml to incorporate this setup https://books.sonatype.com/nexus-book/3.0/reference/maven.html#maven-sect-single-group

I have installed Nexus in /home/centos/nexus30/. Make sure you increase the file descriptors before you start.

"cd /home/centos/nexus30/nexus-3.5.1-02/bin/" and "./nexus start".

Login at http://localhost:8081 using admin/admin123 , go to http://localhost:8081/#admin/repository/repositories and check that "maven central" is already preconfigured with http://localhost:8081/repository/maven-central/ URL. You can browse the Nexus content at http://localhost:8081/#browse/browse/components:maven-central (it should be empty at this stage)

Clear the local Maven (ST) repo "rm -rf /home/centos/.m2/repository/*" and run again "mvn package". This time Maven should retrieve the artifacts from Nexus:

Downloaded: http://localhost:8081/repository/maven-public/org/apache/maven/plugins/maven-resources-plugin/2.5/maven-resources-plugin-2.5.pom

Browsing again http://localhost:8081/#browse/browse/components:maven-central should show that all artifacts are cached in Nexus.

In case of issues, check the logs /home/centos/nexus30/sonatype-work/nexus3/log (nexus.log, request.log, jvm.log) for details.

Of course you can use the nexus docker way https://hub.docker.com/r/sonatype/nexus3/

PS: (ST) stands for STINKS . Maven STINKS like a wet rotten dirty skunk. Use Gradle instead.

Software Vulnerability Control with Sonatype products

Interesting introductory vide on the topic of Security in Software Supply Chain https://vimeo.com/217179090

Software Factory | Sonatype from Sonatype Nexus on Vimeo.

A really detailed presentation of the "Nexus Lifecycle" and "Nexus IQ server"





Software Supply Chain
Continuous Integration
Continuous Delivery
Release Automation Tool
DevOps Native Software Development
Nexus Firewall
Public repositories: Maven Central,

Sonar Security Rules: https://docs.sonarqube.org/display/SONAR/Security-related+rules

Software Weakness

Software Vulnerability

http://cwe.mitre.org/ Common Weakness Enumeration (common software security weaknesses) - very educational FAQ on Software Weakness here http://cwe.mitre.org/about/faq.html#A.1

https://www.first.org/cvss/ CVSS Common Vulnerability Scoring System, scores explained: https://www.first.org/cvss/specification-document

https://www.sonatype.com/nexus-lifecycle-tour-software-supply-chain-automation-sonatype Nexus lifecycle

https://nvd.nist.gov/ NVD is National Vulnerability Database - try searching for Bouncy Castle

Using "Application Health Check" to scan vulnerabilities:



I have read the devsecops Gartner report https://www.sonatype.com/devsecops also available here https://cdn2.hubspot.net/hubfs/1958393/White_Papers/devsecops_how_to_seamlessly__315283.pdf?t=1482418124868

Very interesting also the https://www.sonatype.com/ssc2017?hsCtaTracking=d915532d-28ac-4818-a2db-d4e8feb07036%7C61ff62a0-da6f-47d1-9591-c89200a00454 2017 State of the Open Software

See also the famous https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/owasptop10/OWASP%20Top%2010%20-%202013.pdf OWASP top 10 web application security risks. Number A9 says: "Components, such as libraries, frameworks, and other software modules, almost always run with full privileges. If a vulnerable component is exploited, such an attack can facilitate serious data loss or server takeover. Applications using components with known vulnerabilities may undermine application defenses and enable a range of possible attacks and impacts."

Repository Health Check RHC demo video here https://sonatype.wistia.com/medias/77jh7h47av

Result of a WebGoat Health Check https://clm.sonatype.com/reports/Sonatype/71fe66f5c3b540f09caa9ebf1f103e7a-8fae0/

Comparison of Free and Opensource Software Licenses https://en.wikipedia.org/wiki/Comparison_of_free_and_open-source_software_licenses

more videos on:

Brian Fox, Integration of Nexus Health Check with Eclipse https://www.youtube.com/watch?v=_wqxy5A7-oA&feature=youtu.be

Brian Fox, Nexus IQ Server email alerts on Weak Security https://www.youtube.com/watch?v=UHulp3PWqFk

Brian Fox, Nexus IQ Server, Define security policies https://www.youtube.com/watch?v=EnKHimjNjvQ




Q: Do I really need IQ Server? Can't I simply do a "health check" on a Nexus Repository and check manually each software vulnerability?

http://www.sonatype.org/nexus/2017/06/29/is-manual-remediation-with-repository-health-check-as-good-as-it-gets/

When you run a Health Check on a Nexus Repository, all you get is a high level report,
flagging the vulnerabilities but without pointers to the Vulnerability Database, nor indication of the newest version without vulnerability. All you get is this:

" Last generated Tue Sep 19 2017 at 5:18:09 AM
Health report for your central repository
Out of 74 components in central, 74 (100%) are known, and of these, 2 (3%) are vulnerable.
Download trends
Insufficient trend data
As you download components from central, we will show the percentage of vulnerable downloads over time.
The most vulnerable downloads over the last 30 days are listed below.
Component Vulnerabilities Last 30 Days Suggestion
com.thoughtworks.xstream : xstream : 1.3.1 Critical (3)
Update version
org.codehaus.plexus : plexus-archiver : 2.1 Severe (1)
Update version"




and then you are on your own googling for a solution

Dependency trees in Nexus and Maven: who uses what ?

Nexus plugin to display who uses a given artifact https://github.com/saleemshafi/nexus-artifact-usage-plugin (apparently the same info is built-in in Apache Archiva https://stackoverflow.com/questions/19392236/who-is-using-my-maven-artifact ). As suggested in SO, you should build a über-POM containing all your projects, then generate the dependency tree.

Useful commands:

mvn dependency:tree -DoutputType=graphml -DoutputFile=dependency.graphml

(use also --debug in case of errors)

Here the documentation on Dependency Management in Maven https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html

Transitive Dependencies

Spring Framework Essentials course by Ken Kousen in Safari Books Online

Here is the code https://github.com/kousen/Spring-Framework-Essentials

Ken is a very talented teacher, who focuses on concepts rather than sheer data. Highly recommended.

The course is available only if you register in Safari Books Online https://www.safaribooksonline.com/, but there is a short term FREE evaluation account (no credit car required).