netdata/build: A small house keeping around coverity scan (no functional or process change) (#5656)

* netdata/build: a small house keeping around coverity scan (no functional or process change)

1) Split coverity install on seperate scriptlet, call it through travis when needed
2) Remove shellcheck exclussions, they dont seem to be raised any more (i might be using wrong version of shellcheck, will see)
3) Fix authors,copyrights etc
4) Make sure all echo goes to stderr
5) When installing, use mktemp for the temporary directory and put everything there. Remove afterwards

* netdata/build: missed - remove temp folder once done

* Rebumping with empty commit to trigger build

* Temp commit to force execution of modified code for coverity

* [ci skip] Revert "Temp commit to force execution of modified code for coverity"

This reverts commit 70e770cbbc.

* Temp commit to force execution of modified code for coverity

* Temp commit to force execution of modified code for coverity

* packaging/netdata/coverity-scan: Missed PATH export during scan phase

* netdata/packaging/coverity-scan: the source folder not so temporary after all

Use a more permanent path for the workdir and we will deep dive on the details of this if necessary at a later stage

* netdata/packaging: This is a more general change, push REPOSITORY variable up to travis layer to help testing against personal accounts

* netdata/packaging/coverity-scan: Missed folder creation (no longer using mktemp, duh), also make sure you do a couple of validations so that you know install is failed or not, otherwise we exit with success but nothing has happened

* netdata/coverity-scan: a few more changes towards abstraction
1) move for submit email to travis environment variables
2) Use REPOSITORY in project name
3) Add some new lines on the last log message, to separate it from the curl result

* netdata/coverity: missed parameter from echo

* Reinstate the flag that controls coverity scan execution (was removed for testing purposes, to run coverity in a controlled environment)

* netdata/coverity-scan: Control coverity scan results submission and report back the potential error without blocking the pipeline by failing the step completely

.travis.yml

@@ -27,8 +27,7 @@ jobs:
     script: ./tests/installer/checksums.sh
   - name: coverity
     install: sudo apt-get install -y zlib1g-dev uuid-dev libipmimonitoring-dev libmnl-dev libnetfilter-acct-dev
-    script: ./coverity-scan.sh || echo "Coverity failed :("
-    env: INSTALL_COVERITY="true"
+    script: ./coverity-install.sh && ./coverity-scan.sh || echo "Coverity failed :("
     if: type = cron
 
   - stage: build

.travis/nightlies.sh

@@ -36,7 +36,6 @@ else
 fi
 
 echo "--- BUILD & PUBLISH DOCKER IMAGES ---"
-export REPOSITORY="netdata/netdata"
 packaging/docker/build.sh || BAD_THING_HAPPENED=1
 
 echo "--- BUILD ARTIFACTS ---"

.travis/releaser.sh

@@ -64,7 +64,6 @@ if [[ $(git describe) =~ -rc* ]]; then
 fi
 
 echo "---- CREATING TAGGED DOCKER CONTAINERS ----"
-export REPOSITORY="netdata/netdata"
 ./packaging/docker/build.sh
 
 echo "---- CREATING RELEASE ARTIFACTS -----"

Makefile.am

@@ -71,6 +71,7 @@ dist_noinst_DATA= \
 # should be proper init.d/openrc/systemd usable
 dist_noinst_SCRIPTS= \
 	coverity-scan.sh \
+	coverity-install.sh \
 	packaging/installer/netdata-updater.sh \
 	packaging/installer/kickstart.sh \
 	packaging/installer/kickstart-static64.sh \

coverity-install.sh

@@ -0,0 +1,42 @@
+#!/usr/bin/env bash
+# Coverity installation script
+#
+# Copyright: SPDX-License-Identifier: GPL-3.0-or-later
+#
+# Author: Pavlos Emm. Katsoulakis (paul@netdata.cloud)
+
+token="${COVERITY_SCAN_TOKEN}"
+([ -z "${token}" ] && [ -f .coverity-token ]) && token="$(<.coverity-token)"
+if [ -z "${token}" ]; then
+	echo >&2 "Save the coverity token to .coverity-token or export it as COVERITY_SCAN_TOKEN."
+	exit 1
+fi
+
+covbuild="$(which cov-build 2>/dev/null || command -v cov-build 2>/dev/null)"
+([ -z "${covbuild}" ] && [ -f .coverity-build ]) && covbuild="$(<.coverity-build)"
+if [ ! -z "${covbuild}" ]; then
+	echo >&2 "Coverity already installed, nothing to do!"
+	exit 0
+fi
+
+echo >&2 "Installing coverity..."
+WORKDIR="/opt/coverity-source"
+mkdir -p "${WORKDIR}"
+
+curl -SL --data "token=${token}&project=${REPOSITORY}" https://scan.coverity.com/download/linux64 > "${WORKDIR}/coverity_tool.tar.gz"
+if [ -f "${WORKDIR}/coverity_tool.tar.gz" ]; then
+	tar -x -C "${WORKDIR}" -f "${WORKDIR}/coverity_tool.tar.gz"
+	sudo mv "${WORKDIR}/cov-analysis-linux64-2017.07" /opt/coverity
+	export PATH=${PATH}:/opt/coverity/bin/
+else
+	echo "Failed to download coverity tool tarball!"
+fi
+
+# Validate the installation
+covbuild="$(which cov-build 2>/dev/null || command -v cov-build 2>/dev/null)"
+if [ -z "$covbuild" ]; then
+	echo "Failed to install coverity!"
+	exit 1
+else
+	echo >&2 "Coverity scan installed!"
+fi

coverity-scan.sh

@@ -1,10 +1,15 @@
 #!/usr/bin/env bash
-# shellcheck disable=SC2235
-
+# Coverity scan script
+#
 # To run this script you need to provide API token. This can be done either by:
 #  - Putting token in ".coverity-token" file
 #  - Assigning token value to COVERITY_SCAN_TOKEN environment variable
-# Additionally script can install coverity tool on your computer. To do this just set environment variable INSTALL_COVERITY to "true"
+#
+# Copyright: SPDX-License-Identifier: GPL-3.0-or-later
+#
+# Author  : Costa Tsaousis (costa@netdata.cloud)
+# Author  : Pawel Krupa (paulfantom)
+# Author  : Pavlos Emm. Katsoulakis (paul@netdata.cloud)
 
 cpus=$(grep -c ^processor </proc/cpuinfo)
 [ -z "${cpus}" ] && cpus=1
@@ -16,28 +21,14 @@ if [ -z "${token}" ]; then
 	exit 1
 fi
 
-# shellcheck disable=SC2230
+export PATH=${PATH}:/opt/coverity/bin/
 covbuild="$(which cov-build 2>/dev/null || command -v cov-build 2>/dev/null)"
 ([ -z "${covbuild}" ] && [ -f .coverity-build ]) && covbuild="$(<.coverity-build)"
 if [ -z "${covbuild}" ]; then
-	echo "Cannot find 'cov-build' binary in \$PATH."
-	if [ "${INSTALL_COVERITY}" != "" ]; then
-		echo "Installing coverity..."
-		mkdir /tmp/coverity
-		curl -SL --data "token=${token}&project=netdata%2Fnetdata" https://scan.coverity.com/download/linux64 > /tmp/coverity_tool.tar.gz
-		tar -x -C /tmp/coverity/ -f /tmp/coverity_tool.tar.gz
-		sudo mv /tmp/coverity/cov-analysis-linux64-2017.07 /opt/coverity
-		export PATH=${PATH}:/opt/coverity/bin/
-		# shellcheck disable=SC2230
-		covbuild="$(which cov-build 2>/dev/null || command -v cov-build 2>/dev/null)"
-	else
-		echo "Save command the full filename of cov-build in .coverity-build"
-		exit 1
-	fi
-fi
-
-if [ ! -x "${covbuild}" ]; then
-	echo "The command ${covbuild} is not executable. Save command the full filename of cov-build in .coverity-build"
+	echo >&2 "Cannot find 'cov-build' binary in \$PATH."
+	exit 1
+elif [ ! -x "${covbuild}" ]; then
+	echo >&2 "The command ${covbuild} is not executable. Save command the full filename of cov-build in .coverity-build"
 	exit 1
 fi
 
@@ -45,7 +36,7 @@ version="$(grep "^#define PACKAGE_VERSION" config.h | cut -d '"' -f 2)"
 echo >&2 "Working on netdata version: ${version}"
 
 echo >&2 "Cleaning up old builds..."
-make clean || echo "Nothing to clean"
+make clean || echo >&2 "Nothing to clean"
 
 [ -d "cov-int" ] && rm -rf "cov-int"
 
@@ -59,9 +50,13 @@ echo >&2 "Compressing data..."
 tar czvf netdata-coverity-analysis.tgz cov-int || exit 1
 
 echo >&2 "Sending analysis for version ${version} ..."
-curl --progress-bar --form token="${token}" \
-  --form email=costa@tsaousis.gr \
+COVERITY_SUBMIT_RESULT=$(curl --progress-bar --form token="${token}" \
+  --form email=${COVERITY_SCAN_SUBMIT_MAIL} \
   --form file=@netdata-coverity-analysis.tgz \
   --form version="${version}" \
   --form description="netdata, real-time performance monitoring, done right." \
-  https://scan.coverity.com/builds?project=netdata%2Fnetdata
+  https://scan.coverity.com/builds?project=${REPOSITORY})
+
+echo ${COVERITY_SUBMIT_RESULT} | grep -q -e 'Build successfully submitted' || echo >&2 "scan results were not pushed to coverity. Message was: ${COVERITY_SUBMIT_RESULT}"
+
+echo >&2 "Coverity scan mechanism completed"