netdata/packaging: Binary distributions - clean up .DEB package generation process (#6465)

* netdata/packaging: [ci skip] reinstate dev branch to help testing

* netdata/packaging: [ci skip] Yes, make the right package choice on json-c for debian based distros..

* netdata/packaging: [ci skip] two more packages that differentiate on debian/ubuntu...............

* netdata/packaging: [ci skip] Add more package dependencies around building

* netdata/packaging: [ci skip] Introduce changelog generation technique for .DEB

1) Remove contrib/debian/changelog from gitignore, we want it in
2) Add variables to monitor latest release version and latest release date from git
3) Add a templated contrib/debian/changelog, that will be the starting point for our changelog generation.
   Either on stable release or nightly releases, we will be generating the logs since the previous release, to manage the content.
   Note: We might eventually take the effort to produce one large changelog and append each time the extra stuff
4) Add the steps to prepare the source and the changelog for building and execute the build

At this stage things should still be broken, but we should have come closer to a ready-to-build-the-package environment per distro

* netdata/packaging: [ci skip] Extract to specified directory, otherwise we dump it on a privileged folder using an unprivileged user (builder)

Also, a fix on sed command, missed the s keyword on the string
replacement

* netdata/packaging: [ci skip] Should be in-place replacement, use -i instead of -e

* netdata/packaging: [ci skip] wrong parameter usage

* netdata/packaging: [ci skip] factor out the two build commands to run as a single bundle. refactor trigger deb build script respectively

* netdata/packaging: [ci skip] Be sure to copy over the build script to the build path

* netdata/packaging: [ci skip] OCD - unnecessary s added

* netdata/packaging: [ci skip] Changelog generation should take place on the repo side, then copy it over to the destination (If that doesnt work well, will just provide a static link for changelog

* netdata/packaging: [ci skip] missed the ignore branch option
	Also, a nit on the gbp command

* netdata/packaging: [ci skip] link to the original path, then create symlink on build folder
                             Also, fix --since option

* netdata/packaging: [ci skip] Run replacement of changelog fields on the host, not on the container

* netdata/packaging: [ci skip] fix contrib path, you are no longer reading from container path

* netdata/packaging: [ci skip] fix version

* netdata/packaging: [ci skip] add dep

* netdata/packaging: [ci skip] fix changelog template

* netdata/packaging: [ci skip] Fix debian package build, we were looping through forever. Exclude contrib all together

* netdata/packaging: [ci skip] Add logic to gather all package related content from the container folder and push it for publishing

* netdata/packaging: [ci skip] fixes and nits
1) use the branch for the beta deployment
2) change the cp pattern

* netdata/packaging:[ci skip] add more debug info

* netdata/packaging: [ci skip] Adjust LXC directory permissions, so that the rest of the workflow can pull through

* netdata/packaging: [ci skip] Using expressions within double quotes doesnt work, use quotes only on the variable to avoid syntax warnings from shellchecking

* netdata/packaging: [ci skip] Update build deps

* netdata/packaging: Add runtime deps (hopefully)

* netdata/packaging: [ci skip] remove old distro

* netdata/packaging: [ci skip] dont let that space there, breaks the syntax

* netdata/packaging: [ci skip] missed one

* netdata/packaging: [ci skip] yank yank yank

* netdata/packaging: [ci skip] You missed www over there

* netdata/packaging: [ci skip] Remove broken distros (no container images). We will revisit the way to support multiple distros in a different way, lxc failed us

* netdata/packaging: Pull in go.d plugin, also fix chmod command to just pass the folder since with the -R doesnt require anything else

* netdata/packaging: [ci skip] Get cups, freeipmi and nfacct on

* netdata/packaging: [ci skip] pull different package in

* netdata/packaging: [ci skip] Install dependencies on the build machine

* netdata/packaging: [ci skip] dont do post-clean, not on all versions

* netdata/packaging: [ci skip] move go install seperately

* netdata/packaging: [ci skip] Towards a more appropriate go.d plugin install

* netdata/packaging: [ci skip] fix permissions setting

* netdata/packaging: [ci skip] fix freeipmi package install

* netdata/packaging: [ci skip] Fix build arch setting on the build command, also remove pre-clean, it breaks on debian/jessie

* netdata/packaging: [ci skip] fix postinst actions, we use libexec

* netdata/packaging: adjust www and go.d permissions

* netdata/packaging: [ci skip] Handle the different build tool version cases, also take care some dependency issues on older version jessie

* netdata/packaging: [ci skip] adjust the control file based on the distro

* netdata/packaging: [ci skip] Move control file preparation prior to packaging the source

* netdata/packaging: [ci skip] Adjust versioning scheme for the nightlies

* netdata/packaging: [ci skip] Let RPM be built too for testing

* netdata/packaging: [ci skip] Factor out version calculation

* netdata/packaging: [ci skip] Attempt to modify version properly

* netdata/packaging: [ci skip] Update documentationg regarding .DEB files

* netdata/packaging: [ci skip] autogen not available everywhere, also fix json-c deps

* netdata/packaging: Remove development settings, also remove beta deployment as its not needed
This commit is contained in:
Paul Emm. Katsoulakis 2019-07-15 22:00:31 +02:00 committed by GitHub
parent cbb0528d32
commit 079d71d6fb
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
16 changed files with 350 additions and 116 deletions

2
.gitignore vendored
View File

@ -139,8 +139,6 @@ cmake_install.cmake
.DS_Store
webcopylocal*
contrib/debian/changelog
# converted diagrams
diagrams/*.png
diagrams/*.svg

View File

@ -24,6 +24,8 @@ install:
- export NOTIF_CHANNEL="automation-beta"
- if [ "${TRAVIS_REPO_SLUG}" = "netdata/netdata" ]; then export NOTIF_CHANNEL="automation"; fi;
- export BUILD_VERSION="$(cat packaging/version | cut -d'-' -f1)"
- export LATEST_RELEASE_VERSION="$(cat packaging/version | cut -d'-' -f1)"
- export LATEST_RELEASE_DATE="$(git log -1 --format=%aD "${LATEST_RELEASE_VERSION}" | cat)"
- if [[ "${TRAVIS_COMMIT_MESSAGE}" = *"[Build latest]"* ]]; then export BUILD_VERSION="$(cat packaging/version | cut -d'-' -f1,2 | sed -e 's/-/./g').latest"; fi;
- export DEPLOY_REPO="netdata" # Default production packaging repository
- if [[ "${TRAVIS_COMMIT_MESSAGE}" = *"[Build latest]"* ]]; then export DEPLOY_REPO="netdata-edge"; fi;
@ -69,8 +71,6 @@ stages:
if: type != cron AND type != pull_request AND branch = master AND commit_message =~ /(\[Package arm64 DEB Ubuntu\]|\[Package arm64 DEB\]|\[Package i386 DEB Ubuntu\]|\[Package i386 DEB\]|\[Package amd64 DEB Ubuntu\]|\[Package amd64 DEB\])/
- name: "Package ubuntu/bionic"
if: type != cron AND type != pull_request AND branch = master AND commit_message =~ /(\[Package arm64 DEB Ubuntu\]|\[Package arm64 DEB\]|\[Package i386 DEB Ubuntu\]|\[Package i386 DEB\]|\[Package amd64 DEB Ubuntu\]|\[Package amd64 DEB\])/
- name: "Package ubuntu/artful"
if: type != cron AND type != pull_request AND branch = master AND commit_message =~ /(\[Package arm64 DEB Ubuntu\]|\[Package arm64 DEB\]|\[Package i386 DEB Ubuntu\]|\[Package i386 DEB\]|\[Package amd64 DEB Ubuntu\]|\[Package amd64 DEB\])/
# Debian
- name: "Package debian/buster"
@ -79,8 +79,6 @@ stages:
if: type != cron AND type != pull_request AND branch = master AND commit_message =~ /(\[Package arm64 DEB Debian\]|\[Package arm64 DEB\]|\[Package i386 DEB Debian\]|\[Package i386 DEB\]|\[Package amd64 DEB Debian\]|\[Package amd64 DEB\])/
- name: "Package debian/jessie"
if: type != cron AND type != pull_request AND branch = master AND commit_message =~ /(\[Package arm64 DEB Debian\]|\[Package arm64 DEB\]|\[Package i386 DEB Debian\]|\[Package i386 DEB\]|\[Package amd64 DEB Debian\]|\[Package amd64 DEB\])/
- name: "Package debian/wheezy"
if: type != cron AND type != pull_request AND branch = master AND commit_message =~ /(\[Package arm64 DEB Debian\]|\[Package arm64 DEB\]|\[Package i386 DEB Debian\]|\[Package i386 DEB\]|\[Package amd64 DEB Debian\]|\[Package amd64 DEB\])/
# Build RPM packages under special conditions
# Enterprise linux (Covers CentOS, Redhat, Amazon linux)
@ -129,7 +127,6 @@ stages:
before_deploy:
- .travis/package_management/yank_stale_rpm.sh "${PACKAGES_DIRECTORY}" "${BUILD_STRING}" || echo "No stale RPM found"
deploy:
# Beta packages deployment
- provider: packagecloud
repository: "${DEPLOY_REPO}"
username: "${PACKAGING_USER}"
@ -142,19 +139,6 @@ stages:
repo: ${TRAVIS_REPO_SLUG}
branch: "master"
condition: -d "${PACKAGES_DIRECTORY}"
# Production release packages deployment
- provider: packagecloud
repository: "netdata"
username: "netdata"
token: "${PKG_CLOUD_TOKEN}"
dist: "${BUILD_STRING}"
local_dir: "${PACKAGES_DIRECTORY}"
skip_cleanup: true
on:
# Only deploy on ${USER}/netdata, master branch, when packages directory is created
repo: "netdata/netdata"
branch: "master"
condition: -d "${PACKAGES_DIRECTORY}"
after_deploy:
- if [ -n "${BUILDER_NAME}" ]; then rm -rf /home/${BUILDER_NAME}/* && echo "Cleared /home/${BUILDER_NAME} directory" || echo "Failed to clean /home/${BUILDER_NAME} directory"; fi;
- if [ -d "${PACKAGES_DIRECTORY}" ]; then rm -rf "${PACKAGES_DIRECTORY}"; fi;
@ -165,7 +149,7 @@ stages:
- stage: &_DEB_TEMPLATE
name: "Build & Publish DEB package"
before_install:
- sudo apt-get install -y wget lxc lxc-templates
- sudo apt-get install -y wget lxc lxc-templates dh-make git-buildpackage build-essential libdistro-info-perl
- source tests/installer/slack.sh
before_script:
- post_message "TRAVIS_MESSAGE" "Starting package preparation and publishing for ${BUILD_STRING}.${BUILD_ARCH}" "${NOTIF_CHANNEL}"
@ -177,7 +161,8 @@ stages:
- echo "packaging/version:" && cat packaging/version
- echo "Creating LXC environment for the build" && sudo -E .travis/package_management/create_lxc_for_build.sh
- echo "Building package in container" && sudo -E .travis/package_management/build_package_in_container.sh
- sudo chmod -R 755 "/var/lib/lxc"
- sudo chown -R root:travis "/var/lib/lxc"
- sudo chmod -R 750 "/var/lib/lxc"
- echo "Preparing DEB packaging contents for upload" && sudo -E .travis/package_management/prepare_packages.sh
git:
depth: false
@ -185,7 +170,6 @@ stages:
before_deploy:
- .travis/package_management/yank_stale_rpm.sh "${PACKAGES_DIRECTORY}" "${BUILD_STRING}" || echo "No stale DEB found"
deploy:
# Beta packages deployment
- provider: packagecloud
repository: "${DEPLOY_REPO}"
username: "${PACKAGING_USER}"
@ -198,19 +182,6 @@ stages:
repo: ${TRAVIS_REPO_SLUG}
branch: "master"
condition: -d "${PACKAGES_DIRECTORY}"
# Production release packages deployment
- provider: packagecloud
repository: "netdata"
username: "netdata"
token: "${PKG_CLOUD_TOKEN}"
dist: "${BUILD_STRING}"
local_dir: "${PACKAGES_DIRECTORY}"
skip_cleanup: true
on:
# Only deploy on ${USER}/netdata, master branch, when build-area directory is created
repo: "netdata/netdata"
branch: "master"
condition: -d "${PACKAGES_DIRECTORY}"
after_deploy:
- if [ -n "${BUILDER_NAME}" ]; then rm -rf /home/${BUILDER_NAME}/* && echo "Cleared /home/${BUILDER_NAME} directory" || echo "Failed to clean /home/${BUILDER_NAME} directory"; fi;
- if [ -d "${PACKAGES_DIRECTORY}" ]; then rm -rf "${PACKAGES_DIRECTORY}"; fi;
@ -422,16 +393,6 @@ jobs:
- stage:
<<: *_DEB_TEMPLATE
stage: "Package ubuntu/artful"
env:
- BUILDER_NAME="builder" BUILD_DISTRO="ubuntu" BUILD_RELEASE="artful" BUILD_STRING="ubuntu/artful"
- PACKAGE_TYPE="deb" REPO_TOOL="apt-get"
- ALLOW_SOFT_FAILURE_HERE=true
# Debian distros build
- stage:
<<: *_DEB_TEMPLATE
@ -463,16 +424,6 @@ jobs:
- stage:
<<: *_DEB_TEMPLATE
stage: "Package debian/wheezy"
env:
- BUILDER_NAME="builder" BUILD_DISTRO="debian" BUILD_RELEASE="wheezy" BUILD_STRING="debian/wheezy"
- PACKAGE_TYPE="deb" REPO_TOOL="apt-get"
- ALLOW_SOFT_FAILURE_HERE=true
# Enterprise linux builds (Centos, Redhat, Amazon linux (el/6))
#
- stage:

View File

@ -0,0 +1,31 @@
#!/usr/bin/env bash
UNPACKAGED_NETDATA_PATH="$1"
LATEST_RELEASE_VERSION="$2"
if [ -z "${LATEST_RELEASE_VERSION}" ]; then
echo "Parameter 'LATEST_RELEASE_VERSION' not defined"
exit 1
fi
if [ -z "${UNPACKAGED_NETDATA_PATH}" ]; then
echo "Parameter 'UNPACKAGED_NETDATA_PATH' not defined"
exit 1
fi
echo "Running changelog generation mechanism since ${LATEST_RELEASE_VERSION}"
echo "Entering ${UNPACKAGED_NETDATA_PATH}"
cd "${UNPACKAGED_NETDATA_PATH}"
echo "Linking debian -> contrib/debian"
ln -sf contrib/debian debian
echo "Executing dpkg-buildpackage"
if dpkg-buildpackage --version 2> /dev/null | grep -q "1.18"; then
dpkg-buildpackage --post-clean --pre-clean --build=binary
else
dpkg-buildpackage -b
fi
echo "DEB build script completed!"

View File

@ -8,6 +8,21 @@ import lxc
import subprocess
import os
def fetch_version(orig_build_version):
tag = None
friendly_version = ""
# TODO: Checksum validations
if str(orig_build_version).count(".latest") == 1:
version_list=str(orig_build_version).replace('v', '').split('.')
friendly_version='.'.join(version_list[0:2]) + "." + version_list[3]
else:
friendly_version = orig_build_version.replace('v', '')
tag = friendly_version # Go to stable tag
print("Version set to %s from %s" % (friendly_version, orig_build_version))
return friendly_version, tag
def replace_tag(tag_name, spec, new_tag_content):
print("Fixing tag %s in %s" % (tag_name, spec))
@ -51,24 +66,36 @@ def install_common_dependendencies(container):
run_command(container, [os.environ["REPO_TOOL"], "clean", "-a"])
run_command(container, [os.environ["REPO_TOOL"], "--no-gpg-checks", "update", "-y"])
run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "json-glib-devel"])
run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "freeipmi-devel"])
run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "cups-devel"])
elif str(os.environ["REPO_TOOL"]).count("yum") == 1:
run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "json-c-devel"])
run_command(container, [os.environ["REPO_TOOL"], "clean", "all"])
run_command(container, [os.environ["REPO_TOOL"], "update", "-y"])
run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "json-c-devel"])
run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "freeipmi-devel"])
run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "cups-devel"])
if os.environ["BUILD_STRING"].count("el/7") == 1 and os.environ["BUILD_ARCH"].count("i386") == 1:
print ("Skipping epel-release install for %s-%s" % (os.environ["BUILD_STRING"], os.environ["BUILD_ARCH"]))
else:
run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "epel-release"])
elif str(os.environ["REPO_TOOL"]).count("apt-get") == 1:
run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "libipmimonitoring-dev"])
run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "libjson-c-dev"])
run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "libcups2-dev"])
else:
run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "cups-devel"])
run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "freeipmi-devel"])
run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "json-c-devel"])
run_command(container, [os.environ["REPO_TOOL"], "update", "-y"])
if os.environ["BUILD_STRING"].count("el/6") < 0:
run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "autogen"])
run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "sudo"])
run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "wget"])
run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "bash"])
run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "freeipmi-devel"])
run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "cups-devel"])
def prepare_version_source(dest_archive, pkg_friendly_version, tag=None):
print(".0 Preparing local implementation tarball for version %s" % pkg_friendly_version)

View File

@ -39,6 +39,8 @@ print("Waiting for container connectivity to start configuration sequence")
if not container.get_ips(timeout=30):
raise Exception("Timeout while waiting for container")
build_path = "/home/%s" % os.environ['BUILDER_NAME']
# Run the required activities now
# 1. Create the builder user
print("1. Adding user %s" % os.environ['BUILDER_NAME'])
@ -48,28 +50,43 @@ common.run_command(container, ["useradd", "-m", os.environ['BUILDER_NAME']])
print("2. Installing package dependencies within LXC container")
common.install_common_dependendencies(container)
print("2.1 Install .DEB build support packages")
common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "dpkg-dev"])
common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "libdistro-info-perl"])
common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "dh-make"])
common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "dh-systemd"])
common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "dh-autoreconf"])
common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "git-buildpackage"])
print("2.2 Add more dependencies")
common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "libnetfilter-acct-dev"])
common.run_command(container, [os.environ["REPO_TOOL"], "install", "-y", "libcups2-dev"])
print ("3. Run install-required-packages scriptlet")
common.run_command(container, ["wget", "-T", "15", "-O", "/home/%s/.install-required-packages.sh" % (os.environ['BUILDER_NAME']), "https://raw.githubusercontent.com/netdata/netdata-demo-site/master/install-required-packages.sh"])
common.run_command(container, ["bash", "/home/%s/.install-required-packages.sh" % (os.environ['BUILDER_NAME']), "netdata", "--dont-wait", "--non-interactive"])
common.run_command(container, ["wget", "-T", "15", "-O", "%s/.install-required-packages.sh" % build_path, "https://raw.githubusercontent.com/netdata/netdata-demo-site/master/install-required-packages.sh"])
common.run_command(container, ["bash", "%s/.install-required-packages.sh" % build_path, "netdata", "--dont-wait", "--non-interactive"])
friendly_version=""
dest_archive=""
download_url=""
tag = None
# TODO: Checksum validations
if str(os.environ['BUILD_VERSION']).count(".latest") == 1:
version_list=str(os.environ['BUILD_VERSION']).replace('v', '').split('.')
friendly_version='.'.join(version_list[0:3]) + "." + version_list[3]
else:
friendly_version = os.environ['BUILD_VERSION'].replace('v', '')
tag = friendly_version # Go to stable tag
friendly_version, tag = common.fetch_version(os.environ['BUILD_VERSION'])
tar_file="%s/netdata-%s.tar.gz" % (os.path.dirname(dest_archive), friendly_version)
print("5. I will be building version '%s' of netdata." % os.environ['BUILD_VERSION'])
dest_archive="/home/%s/netdata-%s.tar.gz" % (os.environ['BUILDER_NAME'], friendly_version)
dest_archive="%s/netdata-%s.tar.gz" % (build_path, friendly_version)
if str(os.environ["BUILD_STRING"]).count("debian/jessie") == 1:
print("5.1 We are building for Jessie, adjusting control file")
common.run_command_in_host(['sudo', 'rm', 'contrib/debian/control'])
common.run_command_in_host(['sudo', 'cp', 'contrib/debian/control.jessie', 'contrib/debian/control'])
common.prepare_version_source(dest_archive, friendly_version, tag=tag)
print("6. Installing build.sh script to build path")
common.run_command_in_host(['sudo', 'cp', '.travis/package_management/build.sh', "%s/%s/build.sh" % (os.environ['LXC_CONTAINER_ROOT'], build_path)])
common.run_command_in_host(['sudo', 'chmod', '777', "%s/%s/build.sh" % (os.environ['LXC_CONTAINER_ROOT'], build_path)])
common.run_command_in_host(['sudo', 'ln', '-sf', 'contrib/debian', 'debian'])
print("Done!")

View File

@ -79,15 +79,7 @@ dest_archive=""
download_url=""
spec_file="/home/%s/rpmbuild/SPECS/netdata.spec" % os.environ['BUILDER_NAME']
tag = None
# TODO: Checksum validations
if str(os.environ['BUILD_VERSION']).count(".latest") == 1:
version_list=str(os.environ['BUILD_VERSION']).replace('v', '').split('.')
rpm_friendly_version='.'.join(version_list[0:3]) + "." + version_list[3]
else:
rpm_friendly_version = os.environ['BUILD_VERSION'].replace('v', '')
tag = rpm_friendly_version # Go to stable tag
rpm_friendly_version, tag = common.fetch_version(os.environ['BUILD_VERSION'])
tar_file="%s/netdata-%s.tar.gz" % (os.path.dirname(dest_archive), rpm_friendly_version)
print("5. I will be building version '%s' of netdata." % os.environ['BUILD_VERSION'])

View File

@ -27,29 +27,36 @@ for d in ${CREATED_CONTAINERS[@]}; do
# Pick up any RPMS from builder
RPM_BUILD_PATH="${LXC_ROOT}/${d}/rootfs/home/${BUILDER_NAME}/rpmbuild"
echo "Checking folder ${RPM_BUILD_PATH} for RPMS and SRPMS"
if [ -d "${RPM_BUILD_PATH}" ]; then
echo "Checking folder ${RPM_BUILD_PATH} for RPMS and SRPMS"
if [ -d "${RPM_BUILD_PATH}/RPMS" ]; then
echo "Copying any RPMS in '${RPM_BUILD_PATH}', copying over the following:"
ls -ltrR "${RPM_BUILD_PATH}/RPMS"
[[ -d "${RPM_BUILD_PATH}/RPMS/x86_64" ]] && cp -r "${RPM_BUILD_PATH}"/RPMS/x86_64/* "${PACKAGES_DIRECTORY}"
[[ -d "${RPM_BUILD_PATH}/RPMS/i386" ]] && cp -r "${RPM_BUILD_PATH}"/RPMS/i386/* "${PACKAGES_DIRECTORY}"
[[ -d "${RPM_BUILD_PATH}/RPMS/i686" ]] && cp -r "${RPM_BUILD_PATH}"/RPMS/i686/* "${PACKAGES_DIRECTORY}"
if [ -d "${RPM_BUILD_PATH}/RPMS" ]; then
echo "Copying any RPMS in '${RPM_BUILD_PATH}', copying over the following:"
ls -ltrR "${RPM_BUILD_PATH}/RPMS"
[[ -d "${RPM_BUILD_PATH}/RPMS/x86_64" ]] && cp -r "${RPM_BUILD_PATH}"/RPMS/x86_64/* "${PACKAGES_DIRECTORY}"
[[ -d "${RPM_BUILD_PATH}/RPMS/i386" ]] && cp -r "${RPM_BUILD_PATH}"/RPMS/i386/* "${PACKAGES_DIRECTORY}"
[[ -d "${RPM_BUILD_PATH}/RPMS/i686" ]] && cp -r "${RPM_BUILD_PATH}"/RPMS/i686/* "${PACKAGES_DIRECTORY}"
fi
if [ -d "${RPM_BUILD_PATH}/SRPMS" ]; then
echo "Copying any SRPMS in '${RPM_BUILD_PATH}', copying over the following:"
ls -ltrR "${RPM_BUILD_PATH}/SRPMS"
[[ -d "${RPM_BUILD_PATH}/SRPMS/x86_64" ]] && cp -r "${RPM_BUILD_PATH}"/SRPMS/x86_64/* "${PACKAGES_DIRECTORY}"
[[ -d "${RPM_BUILD_PATH}/SRPMS/i386" ]] && cp -r "${RPM_BUILD_PATH}"/SRPMS/i386/* "${PACKAGES_DIRECTORY}"
[[ -d "${RPM_BUILD_PATH}/SRPMS/i686" ]] && cp -r "${RPM_BUILD_PATH}"/SRPMS/i686/* "${PACKAGES_DIRECTORY}"
fi
else
DEB_BUILD_PATH="${LXC_ROOT}/${d}/rootfs/home/${BUILDER_NAME}"
echo "Checking folder ${DEB_BUILD_PATH} for DEB packages"
if [ -d "${DEB_BUILD_PATH}" ]; then
cp "${DEB_BUILD_PATH}"/netdata*.ddeb "${PACKAGES_DIRECTORY}" || echo "Could not copy any .ddeb files"
cp "${DEB_BUILD_PATH}"/netdata*.deb "${PACKAGES_DIRECTORY}" || echo "Could not copy any .deb files"
cp "${DEB_BUILD_PATH}"/netdata*.buildinfo "${PACKAGES_DIRECTORY}" || echo "Could not copy any .buildinfo files"
cp "${DEB_BUILD_PATH}"/netdata*.changes "${PACKAGES_DIRECTORY}" || echo "Could not copy any .changes files"
else
echo "Folder ${DEB_BUILD_PATH} does not exist or not a directory, nothing to do for package preparation"
fi
fi
if [ -d "${RPM_BUILD_PATH}/SRPMS" ]; then
echo "Copying any SRPMS in '${RPM_BUILD_PATH}', copying over the following:"
ls -ltrR "${RPM_BUILD_PATH}/SRPMS"
[[ -d "${RPM_BUILD_PATH}/SRPMS/x86_64" ]] && cp -r "${RPM_BUILD_PATH}"/SRPMS/x86_64/* "${PACKAGES_DIRECTORY}"
[[ -d "${RPM_BUILD_PATH}/SRPMS/i386" ]] && cp -r "${RPM_BUILD_PATH}"/SRPMS/i386/* "${PACKAGES_DIRECTORY}"
[[ -d "${RPM_BUILD_PATH}/SRPMS/i686" ]] && cp -r "${RPM_BUILD_PATH}"/SRPMS/i686/* "${PACKAGES_DIRECTORY}"
fi
# Pick up any DEBs from builder
DEB_BUILD_PATH="${d}/home/${BUILDER_NAME}/build-area"
echo "Checking folder ${DEB_BUILD_PATH} for DEB packages"
#TODO: During debian clean up we 'll fill this up
done
chmod -R 777 "${PACKAGES_DIRECTORY}"

View File

@ -37,15 +37,37 @@ if not container.running or not container.state == "RUNNING":
if not container.get_ips(timeout=30):
raise Exception("Timeout while waiting for container")
build_path = "/home/%s" % os.environ['BUILDER_NAME']
print("Setting up EMAIL and DEBFULLNAME variables required by the build tools")
os.environ["EMAIL"] = "bot@netdata.cloud"
os.environ["DEBFULLNAME"] = "Netdata builder"
# Run the build process on the container
print("Starting DEB build process, running dh-make")
new_version = os.environ["BUILD_VERSION"].replace('v', '')
new_version, tag = common.fetch_version(os.environ['BUILD_VERSION'])
print("Starting DEB build process for version %s" % new_version)
print("Building the package")
common.run_command(container, ["sudo", "-u", os.environ['BUILDER_NAME'], "dpkg-buildpackage", "--host-arch", "amd64", "--target-arch", "amd64", "--post-clean", "--pre-clean", "--build=binary", "--release-by=\"Netdata Builder\"", "--build-by=\"Netdata Builder\""])
netdata_tarball = "%s/netdata-%s.tar.gz" % (build_path, new_version)
unpacked_netdata = netdata_tarball.replace(".tar.gz", "")
print("Extracting tarball %s" % netdata_tarball)
common.run_command(container, ["sudo", "-u", os.environ['BUILDER_NAME'], "tar", "xf", netdata_tarball, "-C", build_path])
print("Fixing changelog tags")
changelog_in_host = "contrib/debian/changelog"
common.run_command_in_host(['sed', '-i', 's/PREVIOUS_PACKAGE_VERSION/%s-1/g' % os.environ["LATEST_RELEASE_VERSION"].replace("v", ""), changelog_in_host])
common.run_command_in_host(['sed', '-i', 's/PREVIOUS_PACKAGE_DATE/%s/g' % os.environ["LATEST_RELEASE_DATE"], changelog_in_host])
print("Executing gbp dch command..")
common.run_command_in_host(['gbp', 'dch', '--release', '--ignore-branch', '--spawn-editor=snapshot', '--since=%s' % os.environ["LATEST_RELEASE_VERSION"], '--new-version=%s' % new_version])
print("Copying over changelog to the destination machine")
common.run_command_in_host(['sudo', 'cp', 'debian/changelog', "%s/%s/netdata-%s/contrib/debian/" % (os.environ['LXC_CONTAINER_ROOT'], build_path, new_version)])
print("Running debian build script since %s" % os.environ["LATEST_RELEASE_VERSION"])
common.run_command(container, ["sudo", "-u", os.environ['BUILDER_NAME'], "%s/build.sh" % build_path, unpacked_netdata, new_version])
print("Listing contents on build path")
common.run_command(container, ["sudo", "-u", os.environ['BUILDER_NAME'], "ls", "-ltr", build_path])
print('Done!')

View File

@ -21,7 +21,7 @@ fi
PACKAGES_DIR="$1"
DISTRO="$2"
PACKAGES_LIST="$(ls -AR "${PACKAGES_DIR}" | grep '\.rpm')"
PACKAGES_LIST="$(ls -AR "${PACKAGES_DIR}" | grep -e '\.rpm' -e '\.deb' -e '\.ddeb' )"
if [ ! -d "${PACKAGES_DIR}" ] || [ -z "${PACKAGES_LIST}" ]; then
echo "Folder ${PACKAGES_DIR} does not seem to be a valid directory or is empty. No packages to check for yanking"

5
contrib/debian/changelog Normal file
View File

@ -0,0 +1,5 @@
netdata (PREVIOUS_PACKAGE_VERSION) unstable; urgency=medium
* Initial Release
-- Netdata Builder <bot@netdata.cloud> PREVIOUS_PACKAGE_DATE

View File

@ -4,7 +4,22 @@ Build-Depends: debhelper (>= 9),
dh-systemd (>= 1.5),
dpkg-dev (>= 1.13.19),
zlib1g-dev,
uuid-dev
uuid-dev,
libuv1-dev,
liblz4-dev,
libjudy-dev,
libssl-dev,
libmnl-dev,
libjson-c-dev,
libcups2-dev,
libipmimonitoring-dev,
libnetfilter-acct-dev,
autogen,
autoconf,
automake,
pkg-config,
curl,
gcc
Section: net
Priority: optional
Maintainer: Costa Tsaousis <costa@tsaousis.gr>
@ -16,6 +31,17 @@ Architecture: any
Depends: adduser,
libcap2-bin (>= 1:2.0),
lsb-base (>= 3.1-23.2),
zlib1g,
libuuid1,
libuv1,
liblz4-1,
libjudydebian1,
openssl,
libmnl0,
libjson-c3,
cups,
freeipmi,
libnetfilter-acct1,
${misc:Depends},
${shlibs:Depends}
Description: real-time charts for system monitoring

View File

@ -0,0 +1,49 @@
Source: netdata
Build-Depends: debhelper (>= 9),
dh-autoreconf,
dh-systemd (>= 1.5),
dpkg-dev (>= 1.13.19),
zlib1g-dev,
uuid-dev,
liblz4-dev,
libjudy-dev,
libssl-dev,
libmnl-dev,
libjson-c-dev,
libcups2-dev,
libipmimonitoring-dev,
libnetfilter-acct-dev,
autogen,
autoconf,
automake,
pkg-config,
curl,
gcc
Section: net
Priority: optional
Maintainer: Costa Tsaousis <costa@tsaousis.gr>
Standards-Version: 3.9.6
Homepage: https://github.com/netdata/netdata/wiki
Package: netdata
Architecture: any
Depends: adduser,
libcap2-bin (>= 1:2.0),
lsb-base (>= 3.1-23.2),
zlib1g,
libuuid1,
liblz4-1,
libjudydebian1,
openssl,
libmnl0,
libjson-c3,
cups,
freeipmi,
libnetfilter-acct1,
${misc:Depends},
${shlibs:Depends}
Description: real-time charts for system monitoring
Netdata is a daemon that collects data in realtime (per second)
and presents a web site to view and analyze them. The presentation
is also real-time and full of interactive charts that precisely
render all collected values.

96
contrib/debian/install_go.sh Executable file
View File

@ -0,0 +1,96 @@
#!/usr/bin/env bash
LIB_DIR="$1"
LIBEXEC_DIR="$2"
# ############################################################
# Package Go within netdata (TBD: Package it separately)
safe_sha256sum() {
# Within the contexct of the installer, we only use -c option that is common between the two commands
# We will have to reconsider if we start non-common options
if command -v sha256sum >/dev/null 2>&1; then
sha256sum $@
elif command -v shasum >/dev/null 2>&1; then
shasum -a 256 $@
else
fatal "I could not find a suitable checksum binary to use"
fi
}
download_go() {
url="${1}"
dest="${2}"
if command -v curl >/dev/null 2>&1; then
curl -sSL --connect-timeout 10 --retry 3 "${url}" > "${dest}"
elif command -v wget >/dev/null 2>&1; then
wget -T 15 -O - "${url}" > "${dest}"
else
echo >&2
echo >&2 "Downloading go.d plugin from '${url}' failed because of missing mandatory packages."
echo >&2 "Either add packages or disable it by issuing '--disable-go' in the installer"
echo >&2
exit 1
fi
}
install_go() {
# When updating this value, ensure correct checksums in packaging/go.d.checksums
GO_PACKAGE_VERSION="v0.7.0"
ARCH_MAP=(
'i386::386'
'i686::386'
'x86_64::amd64'
'aarch64::arm64'
'armv64::arm64'
'armv6l::arm'
'armv7l::arm'
'armv5tel::arm'
)
if [ -z "${NETDATA_DISABLE_GO+x}" ]; then
echo >&2 "Install go.d.plugin"
ARCH=$(uname -m)
OS=$(uname -s | tr '[:upper:]' '[:lower:]')
for index in "${ARCH_MAP[@]}" ; do
KEY="${index%%::*}"
VALUE="${index##*::}"
if [ "$KEY" = "$ARCH" ]; then
ARCH="${VALUE}"
break
fi
done
tmp=$(mktemp -d /tmp/netdata-go-XXXXXX)
GO_PACKAGE_BASENAME="go.d.plugin-${GO_PACKAGE_VERSION}.${OS}-${ARCH}"
download_go "https://github.com/netdata/go.d.plugin/releases/download/${GO_PACKAGE_VERSION}/${GO_PACKAGE_BASENAME}" "${tmp}/${GO_PACKAGE_BASENAME}"
download_go "https://github.com/netdata/go.d.plugin/releases/download/${GO_PACKAGE_VERSION}/config.tar.gz" "${tmp}/config.tar.gz"
if [ ! -f "${tmp}/${GO_PACKAGE_BASENAME}" ] || [ ! -f "${tmp}/config.tar.gz" ] || [ ! -s "${tmp}/config.tar.gz" ] || [ ! -s "${tmp}/${GO_PACKAGE_BASENAME}" ]; then
echo >&2 "Either check the error or consider disabling it by issuing '--disable-go' in the installer"
echo >&2
return 1
fi
grep "${GO_PACKAGE_BASENAME}\$" "packaging/go.d.checksums" > "${tmp}/sha256sums.txt" 2>/dev/null
grep "config.tar.gz" "packaging/go.d.checksums" >> "${tmp}/sha256sums.txt" 2>/dev/null
# Checksum validation
if ! (cd "${tmp}" && safe_sha256sum -c "sha256sums.txt"); then
echo >&2 "go.d plugin checksum validation failure."
echo >&2 "Either check the error or consider disabling it by issuing '--disable-go' in the installer"
echo >&2
echo "go.d.plugin package files checksum validation failed."
exit 1
fi
# Install files
tar -xf "${tmp}/config.tar.gz" -C "${LIB_DIR}/conf.d/"
mv "${tmp}/$GO_PACKAGE_BASENAME" "${LIBEXEC_DIR}/plugins.d/go.d.plugin"
fi
return 0
}
install_go

View File

@ -28,9 +28,10 @@ case "$1" in
fi
dpkg-statoverride --update --add --force root netdata 0775 /var/lib/netdata/registry
chown -R root:netdata /usr/share/netdata/*
chown -R root:netdata /usr/lib/@DEB_HOST_MULTIARCH@/netdata/plugins.d
setcap cap_dac_read_search,cap_sys_ptrace+ep /usr/lib/@DEB_HOST_MULTIARCH@/netdata/plugins.d/apps.plugin
chown -R root:netdata /usr/share/netdata
chown -R root:netdata /usr/libexec/netdata/plugins.d
chown -R root:netdata /var/lib/netdata/www
setcap cap_dac_read_search,cap_sys_ptrace+ep /usr/libexec/netdata/plugins.d/apps.plugin
#PERMS#
;;

View File

@ -17,7 +17,9 @@ TOP = $(CURDIR)/debian/netdata
#dh $@ --with autoreconf
override_dh_auto_configure:
dh_auto_configure -- --with-math --with-webdir=/var/lib/netdata/www
autoreconf -ivf
dh_auto_configure -- --prefix=/usr --sysconfdir=/etc --localstatedir=/var \
--libexecdir=/usr/libexec --with-user=netdata --with-math --with-webdir=/var/lib/netdata/www
debian/%.postinst: debian/%.postinst.in
sed 's/@DEB_HOST_MULTIARCH@/$(DEB_HOST_MULTIARCH)/g' $< > $@
@ -34,7 +36,7 @@ override_dh_install: debian/netdata.postinst
mkdir -p "$(TOP)/usr/share/netdata"
for D in $$(find "$(TOP)/var/lib/netdata/www/" -maxdepth 1 -type d -printf '%f '); do \
echo Relocating $$D; \
mv "$(TOP)/var/lib/netdata/www/$$D" "$(TOP)/usr/share/netdata/$$D"; \
mv "$(TOP)/var/lib/netdata/www/$$D" "$(TOP)/usr/share/netdata/www/$$D"; \
ln -s "/usr/share/netdata/$$D" "$(TOP)/var/lib/netdata/www/$$D"; \
done
@ -51,6 +53,10 @@ override_dh_install: debian/netdata.postinst
done
sed -i "/^#PERMS#/d" $(CURDIR)/debian/netdata.postinst
# Install go
#
debian/install_go.sh $(TOP)/usr/lib/$(DEB_HOST_MULTIARCH)/netdata/ $(TOP)/usr/libexec/netdata
override_dh_installdocs:
dh_installdocs
@ -58,10 +64,11 @@ override_dh_installdocs:
-name README.md \
-not -path './.travis/*' \
-not -path './debian/*' \
-not -path './contrib/*' \
-exec cp \
--parents \
--target $(TOP)/usr/share/doc/netdata/ \
{} \;
--parents \
--target $(TOP)/usr/share/doc/netdata/ \
{} \;
override_dh_fixperms:
dh_fixperms
@ -69,7 +76,10 @@ override_dh_fixperms:
# apps.plugin should only be runnable by the netdata user. It will be
# given extra capabilities in the postinst script.
#
chmod 0754 $(TOP)/usr/lib/$(DEB_HOST_MULTIARCH)/netdata/plugins.d/apps.plugin
chmod 0754 $(TOP)/usr/libexec/netdata/plugins.d/apps.plugin
chmod 0754 $(TOP)/usr/libexec/netdata/plugins.d/freeipmi.plugin
chmod 0754 $(TOP)/usr/libexec/netdata/plugins.d/perf.plugin
chmod 0750 $(TOP)/usr/libexec/netdata/plugins.d/go.d.plugin
override_dh_installlogrotate:
cp system/netdata.logrotate debian/netdata.logrotate

View File

@ -298,7 +298,9 @@ To apply the changes you made, you have to restart Netdata.
We provide our own flavour of binary packages for the most common operating systems that comply with .RPM and .DEB packaging formats.
We have currently released .RPM versions with version [1.16.0](https://github.com/netdata/netdata/releases/tag/v1.16.0). We are planning to release packages following the .DEB format on one of the following releases. Our current packaging infrastructure provider is [Package Cloud](https://packagecloud.io).
We have currently released packages following the .RPM format with version [1.16.0](https://github.com/netdata/netdata/releases/tag/v1.16.0).
We have planned to release packages following the .DEB format with version [1.17.0](https://github.com/netdata/netdata/releases/tag/v1.17.0).
Early adopters may experiment with our .DEB formatted packages using our nightly releases. Our current packaging infrastructure provider is [Package Cloud](https://packagecloud.io).
Netdata is committed to support installation of our solution to all operating systems. This is a constant battle for Netdata, as we strive to automate and make things easier for our users. For the operating system support matrix, please visit our [distributions](../../packaging/DISTRIBUTIONS.md) support page.