Hammering Nexus 3.14

first, create an entry in your settings.xml for mvn to be able to authenticate in nexus

<server>
<id>nexus</id>
<username>admin</username>
<password>admin123</password>
</server>

then write this testnexus.sh script:

NEXUSHOME=/home/centos/nexus314
NEXUSLOG=$NEXUSHOME/sonatype-work/nexus3/log/nexus.log
rm $NEXUSLOG

cd $NEXUSHOME/nexus-3.14.0-04/bin

for i in `seq 1 1000`;
do
 echo "Iteration number " $i
 #make sure nexus is not running before you start it
 ./nexus stop
 ./nexus start
 while [ ! -f $NEXUSLOG ] ;
 do
  sleep 2
 done
 
 echo "logfile found"
 tail -f $NEXUSLOG | while read LOGLINE
 do
  [[ "$LOGLINE" == *"Started Sonatype Nexus OSS 3.14.0-04"* ]] && pkill -P $$ tail
 done
 #now that it's up, let's wait for some time
 
 echo "nexus started"
 NEXUSPID=`ps -e -o pid,command | grep org.sonatype.nexus.karaf.NexusMain | grep -v grep | awk '{print $1}'`
 echo NEXUSPID = $NEXUSPID
 #wait for a random time before we kill
 sleep $[ ( $RANDOM % 100 )  + 10 ]
 ps -ef | grep nexus
 echo killing $NEXUSPID
 kill -9 $NEXUSPID
 mv $NEXUSLOG  ${NEXUSLOG}.${i}
done

and here the 2 scripts to upload continuously stuff to nexus

testnexusupload1.sh

NEXUSHOME=/home/centos/nexus314

curl -X DELETE -u admin:admin123 http://localhost:8081/repository/maven-releases/commons-beanutils/commons-beanutils/1.9.2/commons-beanutils-1.9.2.pom
curl -X DELETE -u admin:admin123 http://localhost:8081/repository/maven-releases/commons-beanutils/commons-beanutils/1.9.2/commons-beanutils-1.9.2.jar
curl -X DELETE -u admin:admin123 http://localhost:8081/repository/maven-releases/commons-beanutils/commons-beanutils/1.9.2/commons-beanutils-1.9.2.md5
curl -X DELETE -u admin:admin123 http://localhost:8081/repository/maven-releases/commons-beanutils/commons-beanutils/1.9.2/commons-beanutils-1.9.2.sha1
curl -X DELETE -u admin:admin123 http://localhost:8081/repository/maven-releases/commons-beanutils/commons-beanutils/1.9.2/commons-beanutils-1.9.2.pom.sha1
curl -X DELETE -u admin:admin123 http://localhost:8081/repository/maven-releases/commons-beanutils/commons-beanutils/1.9.2/commons-beanutils-1.9.2.pom.md5
curl -X DELETE -u admin:admin123 http://localhost:8081/repository/maven-releases/commons-beanutils/commons-beanutils/1.9.2/commons-beanutils-1.9.2.jar.sha1
curl -X DELETE -u admin:admin123 http://localhost:8081/repository/maven-releases/commons-beanutils/commons-beanutils/1.9.2/commons-beanutils-1.9.2.jar.md5


mvn deploy:deploy-file -DgroupId=commons-beanutils -DartifactId=commons-beanutils -Dversion=1.9.2 -DgeneratePom=true -Dpackaging=jar -DrepositoryId=nexus -Durl=http://localhost:8081/repository/maven-releases -Dfile=$NEXUSHOME/nexus-3.14.0-04/system/commons-beanutils/commons-beanutils/1.9.2/commons-beanutils-1.9.2.jar

testnexusupload2.sh

NEXUSHOME=/home/centos/nexus314

curl -X DELETE -u admin:admin123 http://localhost:8081/repository/maven-releases/commons-codec/commons-codec/1.10/commons-codec-1.10.pom
curl -X DELETE -u admin:admin123 http://localhost:8081/repository/maven-releases/commons-codec/commons-codec/1.10/commons-codec-1.10.jar
curl -X DELETE -u admin:admin123 http://localhost:8081/repository/maven-releases/commons-codec/commons-codec/1.10/commons-codec-1.10.md5
curl -X DELETE -u admin:admin123 http://localhost:8081/repository/maven-releases/commons-codec/commons-codec/1.10/commons-codec-1.10.sha1
curl -X DELETE -u admin:admin123 http://localhost:8081/repository/maven-releases/commons-codec/commons-codec/1.10/commons-codec-1.10.pom.sha1
curl -X DELETE -u admin:admin123 http://localhost:8081/repository/maven-releases/commons-codec/commons-codec/1.10/commons-codec-1.10.pom.md5
curl -X DELETE -u admin:admin123 http://localhost:8081/repository/maven-releases/commons-codec/commons-codec/1.10/commons-codec-1.10.jar.sha1
curl -X DELETE -u admin:admin123 http://localhost:8081/repository/maven-releases/commons-codec/commons-codec/1.10/commons-codec-1.10.jar.md5


mvn deploy:deploy-file -DgroupId=commons-codec -DartifactId=commons-codec -Dversion=1.10 -DgeneratePom=true -Dpackaging=jar -DrepositoryId=nexus -Durl=http://localhost:8081/repository/maven-releases -Dfile=$NEXUSHOME/nexus-3.14.0-04/system/commons-codec/commons-codec/1.10/commons-codec-1.10.jar

testnexusupload3.sh (this will create always new jars)

NEXUSHOME=/home/centos/nexus314
for i in `seq 1 1000000`;
do
 mvn deploy:deploy-file -DgroupId=commons-beanutils -DartifactId=commons-beanutils -Dversion=1.10.${i} -DgeneratePom=true -Dpackaging=jar -DrepositoryId=nexus -Durl=http://localhost:8081/repository/maven-releases -Dfile=$NEXUSHOME/nexus-3.14.0-04/system/commons-beanutils/commons-beanutils/1.9.2/commons-beanutils-1.9.2.jar

 sleep 2
done

nohup ./testnexusupload1.sh > ./testnexusupload1.log 2>&1 &
nohup ./testnexusupload2.sh > ./testnexusupload2.log 2>&1 &
nohup ./testnexusupload3.sh > ./testnexusupload3.log 2>&1 &

or

setsid ./testnexusupload1.sh > ./testnexusupload1.log 2>&1 &
setsid ./testnexusupload2.sh > ./testnexusupload2.log 2>&1 &
setsid ./testnexusupload3.sh > ./testnexusupload3.log 2>&1 &


This way with a continuous cycle of restarts I can test the stability of the repository.
If the DB or shards are corrupted we will find out...



In fact, after some iteration I keep getting this error at startup:




2018-10-28 19:04:49,571+0100 INFO [elasticsearch[C1896386-DD1B2DA2-9E77236C-5051BC20-E1D39AE4][clusterService#updateTask][T#1]] *SYSTEM org.elasticsearch.cluster.metadata - [C1896386-DD1B2DA2-9E77236C-5051BC20-E1D39AE4] [2e9a1e67e8a325bcd6ee9f6790ff6c769e791d56] creating index, cause [api], templates [], shards [1]/[0], mappings [component]
2018-10-28 19:04:49,955+0100 INFO [elasticsearch[C1896386-DD1B2DA2-9E77236C-5051BC20-E1D39AE4][clusterService#updateTask][T#1]] *SYSTEM org.elasticsearch.cluster.routing.allocation - [C1896386-DD1B2DA2-9E77236C-5051BC20-E1D39AE4] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[2e9a1e67e8a325bcd6ee9f6790ff6c769e791d56][0]] ...]).
2018-10-28 19:04:50,065+0100 INFO [quartz-2-thread-1] *SYSTEM org.sonatype.nexus.repository.search.SearchFacetImpl - Rebuilding index of repository maven-central
2018-10-28 19:04:50,501+0100 INFO [elasticsearch[C1896386-DD1B2DA2-9E77236C-5051BC20-E1D39AE4][clusterService#updateTask][T#1]] *SYSTEM org.elasticsearch.cluster.metadata - [C1896386-DD1B2DA2-9E77236C-5051BC20-E1D39AE4] [73ae44bc066b6a7a33b4435641d8229b9b66495a] creating index, cause [api], templates [], shards [1]/[0], mappings [component]
2018-10-28 19:04:50,943+0100 ERROR [Thread-44 <command>sql.select from component where bucket = :bucket</command>] *SYSTEM com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage - Exception `32DA9B7B` in storage `plocal:/home/centos/nexus314/sonatype-work/nexus3/db/component`: 2.2.36 (build d3beb772c02098ceaea89779a7afd4b7305d3788, branch 2.2.x)
com.orientechnologies.orient.core.exception.OCommandExecutionException: Error on execution of command: sql.select from component where bucket = :bucket
DB name="component"
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3421)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.command(OAbstractPaginatedStorage.java:3318)
at com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:78)
at com.orientechnologies.orient.core.sql.query.OSQLAsynchQuery.run(OSQLAsynchQuery.java:74)
at com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:33)
at com.orientechnologies.orient.core.sql.query.OSQLNonBlockingQuery$1.run(OSQLNonBlockingQuery.java:283)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: null
at com.orientechnologies.orient.core.serialization.serializer.binary.impl.index.OCompositeKeySerializer.deserializeFromByteBufferObject(OCompositeKeySerializer.java:347)
at com.orientechnologies.orient.core.serialization.serializer.binary.impl.index.OCompositeKeySerializer.deserializeFromByteBufferObject(OCompositeKeySerializer.java:46)
at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurablePage.deserializeFromDirectMemory(ODurablePage.java:166)
at com.orientechnologies.orient.core.index.sbtree.local.OSBTreeBucket.getKey(OSBTreeBucket.java:266)
at com.orientechnologies.orient.core.index.sbtree.local.OSBTreeBucket.find(OSBTreeBucket.java:134)
at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.findBucket(OSBTree.java:1679)
at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.access$2800(OSBTree.java:69)
at com.orientechnologies.orient.core.index.sbtree.local.OSBTree$OSBTreeCursorForward.next(OSBTree.java:1991)
at com.orientechnologies.orient.core.index.engine.OSBTreeIndexEngine$OSBTreeIndexCursor.nextEntry(OSBTreeIndexEngine.java:271)
at com.orientechnologies.orient.core.index.OIndexAbstractCursor.hasNext(OIndexAbstractCursor.java:83)
at com.orientechnologies.orient.core.index.OIndexChangesWrapper.hasNext(OIndexChangesWrapper.java:138)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.serialIterator(OCommandExecutorSQLSelect.java:1636)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchFromTarget(OCommandExecutorSQLSelect.java:1585)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.fetchValuesFromIndexCursor(OCommandExecutorSQLSelect.java:2466)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchForIndexes(OCommandExecutorSQLSelect.java:2280)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.searchInClasses(OCommandExecutorSQLSelect.java:1017)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLResultsetAbstract.assignTarget(OCommandExecutorSQLResultsetAbstract.java:203)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.assignTarget(OCommandExecutorSQLSelect.java:527)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.executeSearch(OCommandExecutorSQLSelect.java:509)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.execute(OCommandExecutorSQLSelect.java:485)
at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.execute(OCommandExecutorSQLDelegate.java:70)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.executeCommand(OAbstractPaginatedStorage.java:3400)
... 6 common frames omitted






This seems a known issue with a workaround/solution


https://issues.sonatype.org/browse/NEXUS-18036

java -jar ./nexus-3.14.0-04/lib/support/nexus-orient-console.jar

connect plocal:/home/centos/nexus314/sonatype-work/nexus3/db/component/ admin admin

Disconnecting from the database [null]...OK
Connecting to database [plocal:/home/centos/nexus314/sonatype-work/nexus3/db/component/] with user 'admin'...
2018-10-28 23:42:11:147 WARNI {db=component} Storage 'component' was not closed properly. Will try to recover from write ahead log... [OLocalPaginatedStorage]
2018-10-28 23:42:11:192 WARNI {db=component} Record com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OCheckpointEndRecord{lsn=LSN{segment=105, position=52}} will be skipped during data restore [OLocalPaginatedStorage]
2018-10-28 23:42:11:247 WARNI {db=component} Record OFuzzyCheckpointStartRecord{lsn=LSN{segment=105, position=59}} com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OFuzzyCheckpointStartRecord{lsn=null, previousCheckpoint=LSN{segment=105, position=28}} will be skipped during data restore [OLocalPaginatedStorage]
2018-10-28 23:42:11:248 WARNI {db=component} Record com.orientechnologies.orient.core.storage.impl.local.paginated.wal.OFuzzyCheckpointEndRecord{lsn=LSN{segment=105, position=99}} will be skipped during data restore [OLocalPaginatedStorage]OK



repair database --fix-links

Repairing database...
- Removing broken links...
-- Done! Fixed links: 2, modified documents: 2
Repair database complete (0 errors)


rebuild index *


Rebuilding index(es)...
Rebuilt index(es). Found 4354 link(s) in 36.727001 sec(s).


Index(es) rebuilt successfully


disconnect
exit

Importing and exporting dictionaries in XL-Deploy

XebiaLab XL-Deploy is a great tool, but definitely lacking important features as far as versioning and exporting/importing configuration.

If you need to programmatically generate a Dictionary (rather than using the painfully limited UI), you can take a look at this code https://github.com/xebialabs-community/xld-aws-ecs-plugin/blob/master/src/test/resources/docker/initialize/exportCIs.py


In my case, I create a dictionary "pippo.dict" with a couple of keys (pippo, paperino...)

then from the cli:

mypippo=repository.searchByName('pippo.dict')
print mypippo
['Environments/pippo/pippo.dict']

repository.exportCisAndWait('Environments/pippo/pippo.dict')
export/pippo.dict_2018-10-24_16-36-45_045.zip

then

cd /home/centos/xldeploy/xl-deploy-8.2.0-server/export
unzip pippo.dict_2018-10-24_16-36-45_045.zip
cat configuration-items.xml


there are several entries <entry key="pippo">pluto</entry>

you can add/delete/edit those entries, then create a new file (with a slightly different name, 56 instead of 36):

zip pippo.dict_2018-10-24_16-56-45_045.zip configuration-items.xml

and in the CLI:

repository.importCisAndWait('export/pippo.dict_2018-10-24_16-56-45_045.zip')

and lo and behold, the updated properties are in the dictionary!

Getting configuration information from Nexus FOR FREE

One VERY SIMPLE way to log all the Nexus 3 configuration in the nexus.log is to enable debug logging (Administration/Logging/ set the ROOT logger at DEBUG level), then navigate to the Repositories/Roles/Users administration page. You will see that precious JSON is logged in nexus.log, containing all the most minute details of the Repositories/Roles/User configuration.

Programmatically you can achieve the same with a HTTP POST to http://mynexusserver:8081/service/extdirect with this JSON request, Content-Type application/json

[{"action":"coreui_Blobstore","method":"read","data":null,"type":"rpc","tid":104},{"action":"coreui_Repository","method":"read","data":null,"type":"rpc","tid":105},{"action":"coreui_Repository","method":"readRecipes","data":null,"type":"rpc","tid":106},{"action":"coreui_Repository","method":"readReferences","data":[{"page":1,"start":0,"limit":25}],"type":"rpc","tid":107},{"action":"coreui_Repository","method":"read","data":null,"type":"rpc","tid":108}]

[{"action":"coreui_User","method":"read","data":[{"filter":[{"property":"source","value":"default"},{"property":"userId","value":""}],"page":1,"start":0,"limit":100}],"type":"rpc","tid":63},{"action":"coreui_User","method":"readSources","data":null,"type":"rpc","tid":64},{"action":"coreui_Role","method":"read","data":null,"type":"rpc","tid":65},{"action":"coreui_User","method":"read","data":[{"page":1,"start":0,"limit":100}],"type":"rpc","tid":66}]

It would be really cool if such API was made available through SwaggerUI... Nexus 3 is very very primitive as far as API is concerned.

Internally, those actions coreui_Blobstore , coreui_User, coreui_Repository are mapped this way:

action class 'org.sonatype.nexus.coreui.UserComponent' mapped to 'coreui_User'

action class 'org.sonatype.nexus.coreui.RepositoryComponent' mapped to 'coreui_Repository'

action class 'org.sonatype.nexus.coreui.BlobStoreComponent' mapped to 'coreui_Blobstore'


which are in the plugin/nexus-coreui-plugin area as Groovy classes (this for Nexus 3.10... I think in Nexus 3.13 things have changed..)

bintray endless redirection for repo.scala-sbt.org

The Scala folks running this repository don't seem to understand the importance of running their stuff behind a Proxy Server, so that all elements come from the same domain.... instead, they send you HTTP 302 (MOVED) and force you to hop from repo.scala-sbt.org to dl.bintray.com and then to akamai.bintray.com .... which is outrageously cumbersome if you run your Nexus repository from behind a Firewall, then you have to open individually each subdomain... otherwise you end up with the dreaded " HTTP/1.1 407 Proxy Authentication Required "



2018-10-16 16:21:19,993+0200 DEBUG [qtp72695066-17874] *UNKNOWN org.sonatype.nexus.httpclient.outbound - https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.typesafe.sbt/sbt-native-packager/scala_2.12/sbt_1.0/1.3.4/jars/sbt-native-packager.jar > GET /scalasbt/sbt-plugin-releases/com.typesafe.sbt/sbt-native-packager/scala_2.12/sbt_1.0/1.3.4/jars/sbt-native-packager.jar HTTP/1.1

2018-10-16 16:21:20,102+0200 DEBUG [qtp72695066-17874] *UNKNOWN org.sonatype.nexus.httpclient.outbound - https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.typesafe.sbt/sbt-native-packager/scala_2.12/sbt_1.0/1.3.4/jars/sbt-native-packager.jar < HTTP/1.1 302 Moved Temporarily @ 108.4 ms 2018-10-16 16:21:20,102+0200 DEBUG [qtp72695066-17874] *UNKNOWN org.sonatype.nexus.httpclient.outbound - https://dl.bintray.com/sbt/sbt-plugin-releases/com.typesafe.sbt/sbt-native-packager/scala_2.12/sbt_1.0/1.3.4/jars/sbt-native-packager.jar > GET /sbt/sbt-plugin-releases/com.typesafe.sbt/sbt-native-packager/scala_2.12/sbt_1.0/1.3.4/jars/sbt-native-packager.jar HTTP/1.1

2018-10-16 16:21:20,193+0200 DEBUG [qtp72695066-17874] *UNKNOWN org.sonatype.nexus.httpclient.outbound - https://dl.bintray.com/sbt/sbt-plugin-releases/com.typesafe.sbt/sbt-native-packager/scala_2.12/sbt_1.0/1.3.4/jars/sbt-native-packager.jar < HTTP/1.1 302 @ 90.77 ms 2018-10-16 16:21:20,195+0200 DEBUG [qtp72695066-17874] *UNKNOWN org.sonatype.nexus.httpclient.outbound - https://akamai.bintray.com/3f/3f786b85a05e14c7ed5c24fbefbee19fbaa345e1db5d16fa9fa348c2a4e4789e?__gda__=exp=1539700400~hmac=9d5b250bffebb7cedf93ef28933b588c8788e9a7b20415dbe12e255435bf5ede&response-content-disposition=attachment%3Bfilename%3D%22sbt-native-packager.jar%22&response-content-type=application%2Fjava-archive&requestInfo=U2FsdGVkX1_tU3jkSNxcBb5JrmKzZS9pVZUOhb41Zj_AapUhiUGbx2fl-2B2bTDZvRaC1DmJfpw37KETEYGfMyBkdRqY-gknijMaFBx-HSJPsgC4aYd6Ejwpe2uzTSrTmhIF7HIgBbnG6hm8PxLHTvaGWPkWDnEacMK9-YMvxz8zphO4eHB46kniHIOc-vop&response-X-Checksum-Sha1=f8ab68145e3a23da45be0847f114065aa2fc0f54&response-X-Checksum-Sha2=3f786b85a05e14c7ed5c24fbefbee19fbaa345e1db5d16fa9fa348c2a4e4789e > GET /3f/3f786b85a05e14c7ed5c24fbefbee19fbaa345e1db5d16fa9fa348c2a4e4789e?__gda__=exp=1539700400~hmac=9d5b250bffebb7cedf93ef28933b588c8788e9a7b20415dbe12e255435bf5ede&response-content-disposition=attachment%3Bfilename%3D%22sbt-native-packager.jar%22&response-content-type=application%2Fjava-archive&requestInfo=U2FsdGVkX1_tU3jkSNxcBb5JrmKzZS9pVZUOhb41Zj_AapUhiUGbx2fl-2B2bTDZvRaC1DmJfpw37KETEYGfMyBkdRqY-gknijMaFBx-HSJPsgC4aYd6Ejwpe2uzTSrTmhIF7HIgBbnG6hm8PxLHTvaGWPkWDnEacMK9-YMvxz8zphO4eHB46kniHIOc-vop&response-X-Checksum-Sha1=f8ab68145e3a23da45be0847f114065aa2fc0f54&response-X-Checksum-Sha2=3f786b85a05e14c7ed5c24fbefbee19fbaa345e1db5d16fa9fa348c2a4e4789e HTTP/1.1

2018-10-16 16:21:20,204+0200 DEBUG [qtp72695066-17874] *UNKNOWN org.sonatype.nexus.httpclient.outbound - https://akamai.bintray.com/3f/3f786b85a05e14c7ed5c24fbefbee19fbaa345e1db5d16fa9fa348c2a4e4789e?__gda__=exp=1539700400~hmac=9d5b250bffebb7cedf93ef28933b588c8788e9a7b20415dbe12e255435bf5ede&response-content-disposition=attachment%3Bfilename%3D%22sbt-native-packager.jar%22&response-content-type=application%2Fjava-archive&requestInfo=U2FsdGVkX1_tU3jkSNxcBb5JrmKzZS9pVZUOhb41Zj_AapUhiUGbx2fl-2B2bTDZvRaC1DmJfpw37KETEYGfMyBkdRqY-gknijMaFBx-HSJPsgC4aYd6Ejwpe2uzTSrTmhIF7HIgBbnG6hm8PxLHTvaGWPkWDnEacMK9-YMvxz8zphO4eHB46kniHIOc-vop&response-X-Checksum-Sha1=f8ab68145e3a23da45be0847f114065aa2fc0f54&response-X-Checksum-Sha2=3f786b85a05e14c7ed5c24fbefbee19fbaa345e1db5d16fa9fa348c2a4e4789e < HTTP/1.1 407 Proxy Authentication Required @ 9.761 ms

backing up the Nexus configuration

Using the built-in procedure is risky - to say the least. I was unable to make it work (the restore part, I mean).

You can do this (see https://stackoverflow.com/a/45986526/651288 ):

java -jar /opt/sonatype/nexus/lib/support/nexus-orient-console.jar

CONNECT create database plocal:/home/centos/nexus310/sonatype-work/nexus3/db/component admin admin

#to export, use this:
export database component-export



#to import, use this:
drop database
create database plocal:/home/centos/nexus310/sonatype-work/nexus3/db/component
import database component-export.json.gz

I also do this:

gunzip component-export.json.gz
python -m json.tool component-export.json > component-exportPRETTY.json


and the JSON file is quite readable, probably you can even stick it into a Git repo and relax.

just replace "component" with "config" and do all over again, to backup the configuration

From an operational point of view, Nexus 3 is simply too fragile.

Installing JIRA

https://www.atlassian.com/software/jira/download
Linux 64 bit

cd Downloads
sudo ./atlassian-jira-software-7.12.3-x64.bin

Unpacking JRE ...
Starting Installer ...

This will install JIRA Software 7.12.3 on your computer.
OK [o, Enter], Cancel [c]

Choose the appropriate installation or upgrade option.
Please choose one of the following:
Express Install (use default settings) [1], Custom Install (recommended for advanced users) [2, Enter], Upgrade an existing JIRA installation [3]
1
Details on where JIRA Software will be installed and the settings that will be used.
Installation Directory: /opt/atlassian/jira 
Home Directory: /var/atlassian/application-data/jira 
HTTP Port: 8080 
RMI Port: 8005 
Install as service: Yes 
Install [i, Enter], Exit [e]
i

Extracting files ...
                                                                           

Please wait a few moments while JIRA Software is configured.
Installation of JIRA Software 7.12.3 is complete
Start JIRA Software 7.12.3 now?
Yes [y, Enter], No [n]
y

Please wait a few moments while JIRA Software starts up.
Launching JIRA Software ...
Installation of JIRA Software 7.12.3 is complete
Your installation of JIRA Software 7.12.3 is now ready and can be accessed
via your browser.
JIRA Software 7.12.3 can be accessed at http://localhost:8080
Finishing installation ...

http://localhost:8080

generating the evaluation license key is very simple (you should create an account with Atlassian)

the product is installed for user "jira":
sudo su - jira

logs are in /opt/atlassian/jira/logs/


To troubleshoot webhooks, you should enable DEBUG loggin as default in http://localhost:8080/secure/admin/ViewLogging.jspa
then you see the webhook body being traced in /opt/atlassian/jira/logs/catalina.out

Start and stop:
sudo /etc/init.d/jira start
sudo /etc/init.d/jira stop

Spring Boot vault

https://dzone.com/articles/managing-secrets-with-vault

which is basically same as https://spring.io/blog/2016/06/24/managing-secrets-with-vault

More JIRA on Docker

See also https://www.javamonamour.org/2017/11/jira-on-docker-and-integration-with.html


docker run -d -p 8080:8080 cptactionhank/atlassian-jira:7.7.0


I need to install netstat, but I have no root access (root pw is not given), so I do:

docker exec -ti --user root elastic_agnesi /bin/sh

and I get this strange error:

OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "chdir to cwd (\"/var/atlassian/jira\") set in config.json failed: permission denied": unknown


woorkaround:

docker exec -ti elastic_agnesi /bin/sh
chmod 777 .
chmod 777 ..
exit

and try again connecting as root.... I have no idea why, but this time it works....

Then as root do

apt install net-tools

Then I discover that only tcp (not tcp6) ports are available, because the image doesn't support IPv6 : "cat /proc/net/if_inet6" returns nothing.

Now I start my webhook intercepting application with IPv4

java -Djava.net.preferIPv4Stack=true -jar bla.jar

before:

[centos@localhost ~]$ netstat -an | grep 8090
tcp6 0 0 :::8090 :::* LISTEN

after:

[centos@localhost ~]$ netstat -an | grep 8090
tcp 0 0 0.0.0.0:8090 0.0.0.0:* LISTEN

but still curl -X POST http://localhost:8090/greeting is not working... I am giving up and will install JIRA directly on host....

Material UI and Spring Boot

https://material-ui.com/getting-started/installation/

download latest NPM here https://nodejs.org/en/ (10.12.0-CURRENT)

npx create-react-app react-material-ui
cd react-material-ui/
npm install @material-ui/core
npm install contentful





One can use rthis sample https://github.com/mui-org/material-ui/tree/master/examples/cdn

https://github.com/mui-org/material-ui

https://material.io/design/components/data-tables.html#usage

Nexus Groovy scripting

Huge issue in Nexus is that you can't export/import your Users/Groups/Privileges and Repository configuration. All you can do is to take a "backup", which ends of in a folder as a completely unreadable/unversionable format.

So I was looking for

when you run a Groovy script in Nexus, you have available a predefined variable "repository", which is of type org.sonatype.nexus.script.plugin.internal.provisioning.RepositoryApiImpl

This in turn contains a reference to a blobStoreManager org.sonatype.nexus.blobstore.api.BlobStoreManager and to a repositoryManager org.sonatype.nexus.repository.manager.RepositoryManager , plus a series of convenience methods to create commonly used repository formats (you are not expected to create anything fancy, most properties are assigned by default) and groups.

Key element is a org.sonatype.nexus.repository.config.Configuration object, again very disappointing since the configuration is represented by a "attribute" map (String, Map(String, Object)) which is really a stupid idea, too generic interface.

RepositoryManager has a Iterable<Repository> browse(); which returns a collection of org.sonatype.nexus.repository.Repository

Sample script:

import org.sonatype.nexus.repository.Repository

repository.repositoryManager.browse().each { Repository repo ->
    log.info("Repository: $repo")
    log.info("Repository Configuration: $repo.configuration")
}

this dumps in nexus.log the following content:

Repository: RepositoryImpl$$EnhancerByGuice$$c5f0822b{type=proxy, format=nuget, name='nuget.org-proxy'}
Repository Configuration: Configuration{repositoryName='nuget.org-proxy', recipeName='nuget-proxy', attributes={proxy={strictContentTypeValidation=true, contentMaxAge=1440, remoteUrl=https://www.nuget.org/api/v2/, metadataMaxAge=1440}, negativeCache={}, storage={blobStoreName=default}, nugetProxy={}, httpclient={connection={blocked=false, autoBlock=true}}}}

Repository: RepositoryImpl$$EnhancerByGuice$$c5f0822b{type=hosted, format=maven2, name='maven-releases'}
Repository Configuration: Configuration{repositoryName='maven-releases', recipeName='maven2-hosted', attributes={maven={versionPolicy=RELEASE, layoutPolicy=STRICT}, storage={writePolicy=ALLOW_ONCE, strictContentTypeValidation=false, blobStoreName=default}}}

Repository: RepositoryImpl$$EnhancerByGuice$$c5f0822b{type=hosted, format=maven2, name='maven-snapshots'}
Repository Configuration: Configuration{repositoryName='maven-snapshots', recipeName='maven2-hosted', attributes={maven={versionPolicy=SNAPSHOT, layoutPolicy=STRICT}, storage={writePolicy=ALLOW, strictContentTypeValidation=false, blobStoreName=default}}}

Repository: RepositoryImpl$$EnhancerByGuice$$c5f0822b{type=proxy, format=maven2, name='maven-central'}
Repository Configuration: Configuration{repositoryName='maven-central', recipeName='maven2-proxy', attributes={proxy={contentMaxAge=-1, remoteUrl=https://repo1.maven.org/maven2/, metadataMaxAge=1440}, negativeCache={timeToLive=1440, enabled=true}, storage={strictContentTypeValidation=false, blobStoreName=default}, maven-indexer={}, httpclient={connection={blocked=false, autoBlock=true}}, maven={versionPolicy=RELEASE, layoutPolicy=PERMISSIVE}}}

Repository: RepositoryImpl$$EnhancerByGuice$$c5f0822b{type=group, format=nuget, name='nuget-group'}
Repository Configuration: Configuration{repositoryName='nuget-group', recipeName='nuget-group', attributes={storage={blobStoreName=default}, nugetProxy={}, httpclient={}, group={memberNames=[nuget-hosted, nuget.org-proxy]}}}

Repository: RepositoryImpl$$EnhancerByGuice$$c5f0822b{type=hosted, format=nuget, name='nuget-hosted'}
Repository Configuration: Configuration{repositoryName='nuget-hosted', recipeName='nuget-hosted', attributes={storage={writePolicy=ALLOW, blobStoreName=default}}}

Repository: RepositoryImpl$$EnhancerByGuice$$c5f0822b{type=group, format=maven2, name='maven-public'}
Repository Configuration: Configuration{repositoryName='maven-public', recipeName='maven2-group', attributes={maven={versionPolicy=MIXED}, group={memberNames=[maven-releases, maven-snapshots, maven-central]}, storage={blobStoreName=default}}}

which is pretty good result, at least you can capture in one go all the configuration of all your repositories.



This task will delete all your repos:

import org.sonatype.nexus.repository.Repository

repository.repositoryManager.browse().each { Repository repo ->
    log.info("DELETE Repository: $repo")
    repository.repositoryManager.delete("$repo.name")
}

All the predefined variables are

core which is a org.sonatype.nexus.internal.provisioning.CoreApiImpl

repository which is a org.sonatype.nexus.script.plugin.internal.provisioning.RepositoryApiImpl

blobStore which is a org.sonatype.nexus.internal.provisioning.BlobStoreApiImpl

createFileBlobStore(final String name, final String path)
org.sonatype.nexus.blobstore.api.BlobStoreManager blobStoreManager 

security which is a org.sonatype.nexus.security.internal.SecurityApiImpl

User addUser(final String id, final String firstName, final String lastName, final String email, final boolean active,
final String password, final List roleIds)
Role addRole(final String id, final String name, final String description, final List privileges,
final List roles)
User setUserRoles(final String userId, final List roleIds)

If you do security.getSecuritySystem() you get an instance of this:
https://github.com/sonatype/nexus-public/blob/master/components/nexus-security/src/main/java/org/sonatype/nexus/security/SecuritySystem.java


Good is that if you clone the github repo nexus-book-examples you can directly open the APIs in file:///home/centos/gitclones/nexus-book-examples/scripting/apidocs/index.html


List all users with their roles

import groovy.json.JsonOutput
users = security.getSecuritySystem().listUsers()
userjson = JsonOutput.toJson(users)
log.info("USERS $userjson")

or also

import org.sonatype.nexus.security.user.UserSearchCriteria

users = security.getSecuritySystem().searchUsers(new UserSearchCriteria())
log.info("users=" + users)

Add users and roles:

privileges = [
            "nx-search-read",
            "nx-repository-view-*-*-read",
            "nx-repository-view-*-*-browse",
            "nx-repository-view-*-*-add",
            "nx-repository-view-*-*-edit",
"nx-apikey-all"]

security.addRole("deployer", "deployer", "deployment on all repositories", privileges, [])

security.addUser(userName, firstName, lastName, email, true, password, ["deployer"])

creating blobstores:

def list = ["dockerGroup", "mynpm", "pippo", "ivy", "dockerhosted", "dockerProxy", "jcenter", "pythonProxy"]
for (item in list) {
    log.info("creating blobstore " + item)
    blobStore.createFileBlobStore(item, item)
}

creating hosted repositories

import org.sonatype.nexus.repository.storage.WritePolicy;
import org.sonatype.nexus.repository.maven.VersionPolicy;
import org.sonatype.nexus.repository.storage.WritePolicy
import org.sonatype.nexus.repository.maven.LayoutPolicy


repository.createDockerHosted(name = 'pippo', httpPort = 8123, httpsPort = null, blobStoreName = 'docker', strictContentTypeValidation=true, v1Enabled=true, writePolicy = WritePolicy.ALLOW, forceBasicAuth=false)

repository.createMavenHosted(name = 'ivyhosted', blobStoreName = 'ivy', strictContentTypeValidation = true, versionPolicy = VersionPolicy.RELEASE, writePolicy= WritePolicy.ALLOW_ONCE, layoutPolicy=LayoutPolicy.PERMISSIVE )

You cannot delete anonymous, the only way is to update it:

import org.sonatype.nexus.security.user.*
import org.sonatype.nexus.security.role.*

// the following 6 lines are not required
anonymous = security.getSecuritySystem().getUser("anonymous", "default")
log.info("Anonymous before=" + anonymous)
Set allRoles = security.getSecuritySystem().listRoles();
log.info("allRoles=" + allRoles)
advRole = allRoles.find{it.roleId=='ADVRole'}
log.info("advRole=" + advRole)

// these 2 lines below do the job
advRoleIdentifier = new RoleIdentifier('default', 'ADVRole');
security.getSecuritySystem().setUsersRoles("anonymous", "default", [advRoleIdentifier].toSet())

Script to create/update a role (without affecting existing users using that role, if existing)

import org.sonatype.nexus.security.user.*
import org.sonatype.nexus.security.role.*
import static org.sonatype.nexus.security.user.UserManager.DEFAULT_SOURCE

privileges = [
            "nx-search-read",
            "nx-repository-view-*-*-read",
            "nx-repository-view-*-*-browse",
            "nx-repository-view-*-*-add",
            "nx-repository-view-*-*-edit",
            "nx-apikey-all"
]

createOrUpdateRole("pippo", privileges, [])
createOrUpdateUser("pippouser", [ "pippo", "nx-admin" ])

def createOrUpdateRole(rolename, privileges, roles) {
    log.info("calling createOrUpdateRole with parameters rolename=" + rolename + " privileges=" + privileges + " roles=" + roles)
 Set allRoles = security.getSecuritySystem().listRoles()
 Role role = allRoles.find{it.roleId==rolename}
 if (role != null) {
     log.info("existing role=" + role)
  role.setPrivileges(privileges.toSet())
  role.setRoles(roles.toSet())
  log.info("updating role=" + role)
  // security.securitySystem.getAuthorizationManager(DEFAULT_SOURCE).deleteRole(role.roleId)
  security.securitySystem.getAuthorizationManager(DEFAULT_SOURCE).updateRole(role)
  log.info("role updated=" + role)
 }
 else {
  log.info("adding role " + rolename)
  security.addRole(rolename, rolename, rolename, privileges, roles)
  log.info("role " + rolename + " successfully added")
 }
}



def createOrUpdateUser(username, roles) {
 log.info("calling createOrUpdateUser with parameters username=" + username + " roles=" + roles)
 Set allUsers = security.getSecuritySystem().listUsers()
 User user = allUsers.find{it.userId == username}
 if (user != null) {
  log.info("updating existing user=" + user)
  Set roleIdentifiers = new HashSet()
  roles.each{ role -> roleIdentifiers.add(new RoleIdentifier("default", role))}
  security.getSecuritySystem().setUsersRoles(username, "default", roleIdentifiers)
 }
 else {
  security.addUser(username, username, username, username + "@gmail.com", true, username, roles)
 }
}

Eclipse jee-2018-09 hangs upon restarting

Eclipse jee-2018-09 hangs upon restarting

I do a "ps -ef | grep -i eclipse" then a jstack on the process, to find out that:

ModalContext" #136 prio=6 os_prio=0 tid=0x00007fbe7885f000 nid=0x2744 waiting for monitor entry [0x00007fbdc980f000]
   java.lang.Thread.State: BLOCKED (on object monitor)
 at org.eclipse.swt.widgets.Display.asyncExec(Display.java:866)
 - waiting to lock <0x00000000c030b110> (a java.lang.Class for org.eclipse.swt.graphics.Device)
 at org.eclipse.jface.operation.AccumulatingProgressMonitor.beginTask(AccumulatingProgressMonitor.java:146)
 at org.eclipse.core.runtime.ProgressMonitorWrapper.beginTask(ProgressMonitorWrapper.java:51)
 at org.eclipse.ui.internal.ide.application.IDEWorkbenchAdvisor$CancelableProgressMonitorWrapper.beginTask(IDEWorkbenchAdvisor.java:463)
 at org.eclipse.core.runtime.ProgressMonitorWrapper.beginTask(ProgressMonitorWrapper.java:51)
 at org.eclipse.core.internal.resources.SaveManager.save(SaveManager.java:1157)
 at org.eclipse.core.internal.resources.Workspace.save(Workspace.java:2335)
 at org.eclipse.ui.internal.ide.application.IDEWorkbenchAdvisor$5.run(IDEWorkbenchAdvisor.java:529)
 at org.eclipse.jface.operation.ModalContext$ModalContextThread.run(ModalContext.java:122)


"Worker-33: Building workspace" #133 prio=5 os_prio=0 tid=0x00007fbe54f7b800 nid=0x2731 waiting for monitor entry [0x00007fbdf8133000]
   java.lang.Thread.State: BLOCKED (on object monitor)
 at org.eclipse.swt.graphics.Device.isDisposed(Device.java:835)
 - waiting to lock <0x00000000c030b110> (a java.lang.Class for org.eclipse.swt.graphics.Device)
 at org.eclipse.ui.internal.UILockListener.isUI(UILockListener.java:188)
 at org.eclipse.ui.internal.UILockListener.aboutToRelease(UILockListener.java:123)
 at org.eclipse.core.internal.jobs.LockManager.aboutToRelease(LockManager.java:92)
 at org.eclipse.core.internal.jobs.OrderedLock.doRelease(OrderedLock.java:189)
 - locked <0x00000000c648be58> (a org.eclipse.core.internal.jobs.OrderedLock)
 at org.eclipse.core.internal.jobs.OrderedLock.release(OrderedLock.java:237)
 at org.eclipse.core.internal.resources.WorkManager.beginUnprotected(WorkManager.java:109)
 at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2291)
 at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2319)
 at org.eclipse.wst.jsdt.internal.core.JavaModelManager.initializeAllContainers(JavaModelManager.java:2162)
 at org.eclipse.wst.jsdt.internal.core.JavaModelManager.getJsGlobalScopeContainer(JavaModelManager.java:1536)
 at org.eclipse.wst.jsdt.core.JavaScriptCore.getJsGlobalScopeContainer(JavaScriptCore.java:1358)
 at org.eclipse.wst.jsdt.internal.core.JavaProject.resolveClasspath(JavaProject.java:2748)
 at org.eclipse.wst.jsdt.internal.core.JavaProject.getResolvedClasspath(JavaProject.java:2027)
 at org.eclipse.wst.jsdt.internal.core.DeltaProcessor.validateClasspaths(DeltaProcessor.java:2264)
 at org.eclipse.wst.jsdt.internal.core.DeltaProcessor.resourceChanged(DeltaProcessor.java:1950)
 at org.eclipse.wst.jsdt.internal.core.DeltaProcessingState.resourceChanged(DeltaProcessingState.java:389)
 at org.eclipse.core.internal.events.NotificationManager$1.run(NotificationManager.java:300)
 at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:45)
 at org.eclipse.core.internal.events.NotificationManager.notify(NotificationManager.java:290)
 at org.eclipse.core.internal.events.NotificationManager.broadcastChanges(NotificationManager.java:153)
 at org.eclipse.core.internal.resources.Workspace.broadcastBuildEvent(Workspace.java:365)
 at org.eclipse.core.internal.events.AutoBuildJob.doBuild(AutoBuildJob.java:141)
 at org.eclipse.core.internal.events.AutoBuildJob.run(AutoBuildJob.java:235)
 at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)

it seems related to this bug https://bugs.eclipse.org/bugs/show_bug.cgi?id=522174

Just kill the bastard and move on.


Eclipse is a total piece of shit - they should simply rewrite it from scratch using a better design. Time invested in maintaining this old elephant is totally wasted.

Nexus programmatically create Repositories

Nexus API sucks - big time.

This https://blog.sonatype.com/automated-setup-of-a-repository-manager is a desperate attempt to automate stuff.


git clone https://github.com/sonatype/nexus-book-examples.git
cd ./nexus-book-examples/scripting/complex-script

docker run -d -p 8081:8081 --name nexus sonatype/nexus3:3.10.0
see https://hub.docker.com/r/sonatype/nexus3/tags/

check the logs by
docker exec -ti nexus /bin/bash
more /opt/sonatype/sonatype-work/nexus3/log/nexus.log


rm -rf /home/centos/.groovy/grapes/
rm -rf /home/centos/.m2/repository/



You can easily test your scripts from the Nexus UI (create execute script UI) , and also debut them remotely from Intellij :

https://support.sonatype.com/hc/en-us/articles/115015812727-Nexus-3-Groovy-Script-development-environment-setup

Try running this script:

repository.createMavenHosted('pippo')

it can't be easier!


The RepositoryAPI is available here

https://github.com/sonatype/nexus-public/blob/master/plugins/nexus-script-plugin/src/main/java/org/sonatype/nexus/script/plugin/RepositoryApi.java

Repository createMavenHosted(final String name,
final String blobStoreName,
final boolean strictContentTypeValidation,
final VersionPolicy versionPolicy,
final WritePolicy writePolicy,
final LayoutPolicy layoutPolicy);

example:


repository.createMavenHosted('private-again', 'default', true, org.sonatype.nexus.repository.maven.policy.VersionPolicy.SNAPSHOT, org.sonatype.nexus.repository.storage.WritePolicy.ALLOW_ONCE, org.sonatype.nexus.repository.maven.LayoutPolicy.STRICT)



To list repositories (unfortunately it returns only partial information about the Repository):

curl -X GET --header 'Accept: application/json' 'http://localhost:8081/service/rest/beta/repositories'

[ {
  "name" : "nuget.org-proxy",
  "format" : "nuget",
  "type" : "proxy",
  "url" : "http://localhost:8081/repository/nuget.org-proxy"
}, {
  "name" : "maven-releases",
  "format" : "maven2",
  "type" : "hosted",
  "url" : "http://localhost:8081/repository/maven-releases"
}, {
  "name" : "maven-snapshots",
  "format" : "maven2",
  "type" : "hosted",
  "url" : "http://localhost:8081/repository/maven-snapshots"
}, {
  "name" : "maven-central",
  "format" : "maven2",
  "type" : "proxy",
  "url" : "http://localhost:8081/repository/maven-central"
}, {
  "name" : "nuget-group",
  "format" : "nuget",
  "type" : "group",
  "url" : "http://localhost:8081/repository/nuget-group"
}, {
  "name" : "nuget-hosted",
  "format" : "nuget",
  "type" : "hosted",
  "url" : "http://localhost:8081/repository/nuget-hosted"
}, {
  "name" : "maven-public",
  "format" : "maven2",
  "type" : "group",
  "url" : "http://localhost:8081/repository/maven-public"
} ]

Ref: https://help.sonatype.com/repomanager3/rest-and-integration-api/script-api/writing-scripts about Groovy Scripting in Nexus

centos filesystem ntfs not configured in kernel

trying to install an external HD (Windows formatted) on a Centos box...

centos@localhost ~]$ sudo fdisk -l /dev/sdc2

Disk /dev/sdc2: 4000.7 GB, 4000650887168 bytes, 7813771264 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 33553920 bytes
Disk label type: dos
Disk identifier: 0x6e697373

This doesn't look like a partition table
Probably you selected the wrong device.

     Device Boot      Start         End      Blocks   Id  System
/dev/sdc2p1   ?  1936269394  3772285809   918008208   4f  QNX4.x 3rd part
Partition 1 does not start on physical sector boundary.
/dev/sdc2p2   ?  1917848077  2462285169   272218546+  73  Unknown
Partition 2 does not start on physical sector boundary.
/dev/sdc2p3   ?  1818575915  2362751050   272087568   2b  Unknown
Partition 3 does not start on physical sector boundary.
/dev/sdc2p4   ?  2844524554  2844579527       27487   61  SpeedStor
Partition 4 does not start on physical sector boundary.

Partition table entries are not in disk order
[centos@localhost ~]$ 

sudo yum install epel-release

Package epel-release-7-11.noarch already installed and latest version
Nothing to do


find /lib/modules/ | grep ntfs


nothing is found!

sudo yum install ntfs-3g -y


and now it works! Great! New mountpoint is /run/media/centos/Seagate\ Expansion\ Drive/


REF:

https://www.rootusers.com/how-to-mount-a-windows-ntfs-disk-in-linux/