bots: Move bots and related tasks over to cockpit-project/bots

Closes #12872
This commit is contained in:
Sanne Raymaekers 2019-09-23 16:39:40 +02:00 committed by Martin Pitt
parent 2c8be2591a
commit 91d73df7df
296 changed files with 1 additions and 12884 deletions

6
.gitignore vendored
View File

@ -99,11 +99,7 @@ depcomp
/src/selinux/cockpit.pp
/pkg/*/test-*.log
/pkg/*/test-*.trs
/bots/images/*.qcow2
/bots/images/*.partial
/bots/images/*.xz
/bots/images/*.img
/bots/make-checkout-workdir
/bots/
/test/images/*
/test/verify/naughty-*/*
/test/container-probe*

2
.tasks
View File

@ -23,8 +23,6 @@ fi
# File issues for these tasks
if [ $chance -gt 9 ]; then
bots/po-trigger
bots/image-trigger
bots/npm-trigger
bots/naughty-trigger
bots/learn-trigger
fi

6
bots/.gitignore vendored
View File

@ -1,6 +0,0 @@
*.pyc
*.qcow2
*.partial
*.xz
/*.log
/build-results/

View File

@ -1,38 +0,0 @@
# Hacking on the Cockpit Bots
These are automated bots and testing that works on the Cockpit project. This
includes updating operating system images, bringing in changes from other
projects, releasing Cockpit and more.
## Environment for the bots
The bots work in containers that are built in the [cockpituous](https://github.com/cockpit-project/cockpituous)
repository. New dependencies should be added there in the `tests/Dockerfile`
file in that repository.
## Invoking the bots
1. The containers in the `cockpitous` repository invoke the `.tasks` file
at root of this repository.
1. The ```.tasks``` file prints out a list of possible tasks on standard out.
1. The printed tasks are sorted in alphabetical reverse order, and one of the
first items in the list is executed.
## The bots themselves
Most bots are python scripts. They live in this `bots/` directory. Shared code
is in the `bots/tasks` directory.
## Bots filing issues
Many bots file or work with issues in GitHub repository. We can use issues to tell
bots what to do. Often certan bots will just file issues for tasks that are outstanding.
And in many cases other bots will then perform those tasks.
These bots are listed in the `bots/issue-scan` file. They are written using the
`bots/tasks/__init__.py` code, and you can see `bots/example-task` for an
example of one.
## Bots printing output
The bot output is posted using the cockpitous [sink](https://github.com/cockpit-project/cockpituous/tree/master/sink) code. See that link for how it works.

View File

@ -1,122 +0,0 @@
# Cockpit Bots
These are automated bots and tools that work on Cockpit. This
includes updating operating system images, testing changes,
releasing Cockpit and more.
## Images
In order to test Cockpit-related projects, they are staged into an operating
system image. These images are tracked in the ```bots/images``` directory.
These well known image names are expected to contain no ```.```
characters and have no file name extension.
For managing these images:
* image-download: Download test images
* image-upload: Upload test images
* image-create: Create test machine images
* image-customize: Generic tool to install packages, upload files, or run
commands in a test machine image
* image-prepare: Build and install Cockpit packages into a test machine image
(specific to the cockpit project itself, thus it is in test/, not bots/)
For debugging the images:
* bots/vm-run: Run a test machine image
* bots/vm-reset: Remove all overlays from image-customize, image-prepare, etc
from test/images/
In case of `qemu-system-x86_64: -netdev bridge,br=cockpit1,id=bridge0: bridge helper failed`
error, please [allow][1] `qemu-bridge-helper` to access the bridge settings.
To check when images will automatically be refreshed by the bots
use the image-trigger tool:
$ bots/image-trigger -vd
## Tests
The bots automatically run the tests as needed on pull requests
and branches. To check when and where tests will be run, use the
tests-scan tool:
$ bots/tests-scan -vd
#### Note on eslintrc interaction
As eslint looks for additional configurations, eslintrc.(json|yaml) files, in
parent directories, it is recommended to have `"root": true` in the eslint
configuration of any project which is using eslint and is tested through
cockpit-bots.
## Integration with GitHub
A number of machines are watching our GitHub repository and are
executing tests for pull requests as well as making new images.
Most of this happens automatically, but you can influence their
actions with the tests-trigger utility in this directory.
### Setup
You need a GitHub token in ~/.config/github-token. You can create one
for your account at
https://github.com/settings/tokens
When generating a new personal access token, the scope only needs to
encompass public_repo (or repo if you're accessing a private repo).
### Retrying a failed test
If you want to run the "verify/fedora-atomic" testsuite again for pull
request #1234, run tests-trigger like so:
$ bots/tests-trigger 1234 verify/fedora-atomic
### Testing a pull request by a non-whitelisted user
If you want to run all tests on pull request #1234 that has been
opened by someone who is not in our white-list, run tests-trigger
like so:
$ bots/tests-trigger -f 1234
Of course, you should make sure that the pull request is proper and
doesn't execute evil code during tests.
### Refreshing a test image
Test images are refreshed automatically once per week, and even if the
last refresh has failed, the machines wait one week before trying again.
If you want the machines to refresh the fedora-atomic image immediately,
run image-trigger like so:
$ bots/image-trigger fedora-atomic
### Creating new images for a pull request
If as part of some new feature you need to change the content of some
or all images, you can ask the machines to create those images.
If you want to have a new fedora-atomic image for pull request #1234, add
a bullet point to that pull request's description like so, and add the
"bot" label to the pull request.
* [ ] image-refresh fedora-atomic
The machines will post comments to the pull request about their
progress and at the end there will be links to commits with the new
images. You can then include these commits into the pull request in
any way you like.
If you are certain about the changes to the images, it is probably a
good idea to make a dedicated pull request just for the images. That
pull request can then hopefully be merged to master faster. If
instead the images are created on the main feature pull request and
sit there for a long time, they might cause annoying merge conflicts.
[1]: https://blog.christophersmart.com/2016/08/31/configuring-qemu-bridge-helper-after-access-denied-by-acl-file-error/

View File

@ -1,49 +0,0 @@
#!/usr/bin/env python3
# This file is part of Cockpit.
#
# Copyright (C) 2016 Red Hat, Inc.
#
# Cockpit is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Cockpit is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
# To use this example add a line to an issue with the "bot" label
#
# * [ ] example-bot 20
#
import os
import sys
import time
sys.dont_write_bytecode = True
import task
BOTS = os.path.abspath(os.path.dirname(__file__))
BASE = os.path.normpath(os.path.join(BOTS, ".."))
def run(argument, verbose=False, **kwargs):
try:
int(argument)
except (TypeError, ValueError):
return "Failed to parse argument"
sys.stdout.write("Example message to log\n")
# Attach the package.json script as an example
task.attach("./package.json")
time.sleep(20)
if __name__ == '__main__':
task.main(function=run, title="Example bot task")

View File

@ -1,186 +0,0 @@
#!/usr/bin/env python3
# This file is part of Cockpit.
#
# Copyright (C) 2018 Red Hat, Inc.
#
# Cockpit is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Cockpit is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
import sys
import time
import os
import urllib
import json
import re
import ssl
sys.dont_write_bytecode = True
import task
BOTS = os.path.dirname(os.path.realpath(__file__))
NUMBER_OPEN_ISSUES = 7 # How many issues do we want to have open at a given time?
# How far back does our data go? If a flake gets fixed but is still
# flaky after this many days, the bots open another issue.
WINDOW_DAYS = 21
# This parses the output JSONL format discussed here, where various
# values are grouped:
#
# https://github.com/cockpit-project/cockpituous/blob/master/learn/README.md
# Here we're looking for a field in a record that only has one value
def value(record, field):
values = record.get(field, [])
if len(values) == 1:
return values[0][0] or ""
return None
# Here we're looking for the count of a specific field/value in the record
def count(record, field, only):
values = record.get(field, [])
for value, count in values:
if value != only:
continue
return count
return 0
# For linking flakes to test logs
def slurp_one(url, n, logs):
items_url = url + str(n) + "/items.jsonl"
try:
with urllib.request.urlopen(items_url) as f:
for line in f.readlines():
try:
record = json.loads(line.decode('utf-8'))
logs.setdefault(record["test"], [ ]).append(record["url"])
except ValueError as ex:
sys.stderr.write("{0}: {1}\n".format(url, ex))
except urllib.error.URLError as ex:
if ex.code == 404:
return False
raise
return True
def slurp_failure_logs(url):
logs = { }
n = 0
while slurp_one(url, n, logs):
n = n + 1
return logs
def get_failure_logs(failure_logs, name, context):
match = context.replace("/", "-")
return sorted(filter(lambda url: match in url, failure_logs[name]), reverse=True)[0:10]
# Main
def run(context, verbose=False, **kwargs):
api = task.github.GitHub()
open_issues = api.issues(labels=[ "flake" ])
create_count = NUMBER_OPEN_ISSUES - len(open_issues)
if create_count <= 0:
return 0
if verbose:
sys.stderr.write("Going to create %s new flake issue(s)\n" % create_count)
host = os.environ.get("COCKPIT_LEARN_SERVICE_HOST", "learn-cockpit.apps.ci.centos.org")
port = os.environ.get("COCKPIT_LEARN_SERVICE_PORT", "443")
url = "{0}://{1}:{2}/active/".format("https" if port == "443" else "http", host, port)
cafile = os.path.join(BOTS, "images", "files", "ca.pem")
context = ssl.create_default_context(cafile=cafile)
failure_logs = slurp_failure_logs(url)
# Retrieve the URL
statistics = [ ]
with urllib.request.urlopen(url + "statistics.jsonl", context=context) as f:
for line in f.readlines():
try:
record = json.loads(line.decode('utf-8'))
statistics.append(record)
except ValueError as ex:
sys.stderr.write("{0}: {1}\n".format(url, ex))
tests = { }
for record in statistics:
test = value(record, "test")
context = value(record, "context")
status = value(record, "status")
tracker = value(record, "tracker")
# Flaky tests only score on those that fail and are not tracked
if test is not None and status == "failure" and not tracker:
merged = count(record, "merged", True)
not_merged = count(record, "merged", False)
null_merged = count(record, "merged", None)
total = merged + not_merged + null_merged
# And the key is that they were merged anyway
if total > 10:
tests.setdefault(test, [ ]).append((merged / total, context, record))
scores = [ ]
for n, t in tests.items():
scores.append((sum(map(lambda f: f[0], t))/len(t), n, t))
closed_issues = api.issues(labels=["flake"], state="closed", since=(time.time() - (WINDOW_DAYS * 86400)))
def find_in_issues(issues, name):
for issue in issues:
if name in issue["title"]:
return True
return False
def url_desc(url):
m = re.search("pull-[0-9]+", url)
return m.group(0) if m else url
def failure_description(name, f, logs):
return ("%s%% on %s\n" % (int(f[0]*100), f[1]) +
"".join(map(lambda url: " - [%s](%s)\n" % (url_desc(url), url),
get_failure_logs(logs, name, f[1]))))
scores.sort(reverse=True)
for score, name, failures in scores:
if find_in_issues(open_issues, name) or find_in_issues(closed_issues, name):
continue
if verbose:
sys.stderr.write("Opening issue for %s\n" % name)
source = "<details><summary>Source material</summary>\n\n```json\n%s\n```\n</details>\n" % "\n".join(map(lambda f: json.dumps(f[2], indent=2), failures))
data = {
"title": "%s is flaky" % name,
"body": ("\n".join(map(lambda f: failure_description(name, f, failure_logs), failures)) +
"\n\n" + source),
"labels": [ "flake" ]
}
api.post("issues", data)
create_count -= 1
if create_count == 0:
break
return 0
if __name__ == '__main__':
task.main(function=run, title="Create issues for test flakes")

View File

@ -1,73 +0,0 @@
#!/usr/bin/env python3
# This file is part of Cockpit.
#
# Copyright (C) 2015 Red Hat, Inc.
#
# Cockpit is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Cockpit is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
# Shared GitHub code. When run as a script, we print out info about
# our GitHub interacition.
import argparse
import datetime
import sys
sys.dont_write_bytecode = True
from task import github
def httpdate(dt):
"""Return a string representation of a date according to RFC 1123
(HTTP/1.1).
The supplied date must be in UTC.
From: http://stackoverflow.com/a/225106
"""
weekday = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"][dt.weekday()]
month = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep",
"Oct", "Nov", "Dec"][dt.month - 1]
return "%s, %02d %s %04d %02d:%02d:%02d GMT" % (weekday, dt.day, month,
dt.year, dt.hour, dt.minute, dt.second)
def main():
parser = argparse.ArgumentParser(description='Test GitHub rate limits')
parser.parse_args()
# in order for the limit not to be affected by the call itself,
# use a conditional request with a timestamp in the future
future_timestamp = datetime.datetime.utcnow() + datetime.timedelta(seconds=3600)
api = github.GitHub()
headers = { 'If-Modified-Since': httpdate(future_timestamp) }
response = api.request("GET", "git/refs/heads/master", "", headers)
sys.stdout.write("Rate limits:\n")
for entry in ["X-RateLimit-Limit", "X-RateLimit-Remaining", "X-RateLimit-Reset"]:
entries = [t for t in response['headers'].items() if t[0].lower() == entry.lower()]
if entries:
if entry == "X-RateLimit-Reset":
try:
readable = datetime.datetime.utcfromtimestamp(float(entries[0][1])).isoformat()
except:
readable = "parse error"
pass
sys.stdout.write("{0}: {1} ({2})\n".format(entry, entries[0][1], readable))
else:
sys.stdout.write("{0}: {1}\n".format(entry, entries[0][1]))
if __name__ == '__main__':
sys.exit(main())

View File

@ -1,210 +0,0 @@
#!/usr/bin/env python3
# This file is part of Cockpit.
#
# Copyright (C) 2015 Red Hat, Inc.
#
# Cockpit is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Cockpit is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
# image-create -- Make a root image suitable for use with vm-run.
#
# Installs the OS indicated by TEST_OS into the image
# for test machine and tweaks it to be useable with
# vm-run and testlib.py.
import argparse
import os
import shutil
import subprocess
import sys
import time
import tempfile
BOTS = os.path.abspath(os.path.dirname(__file__))
BASE = os.path.normpath(os.path.join(BOTS, ".."))
from machine import testvm
parser = argparse.ArgumentParser(description='Create a virtual machine image')
parser.add_argument('-v', '--verbose', action='store_true', help='Display verbose progress details')
parser.add_argument('-s', '--sit', action='store_true', help='Sit and wait if setup script fails')
parser.add_argument('-n', '--no-save', action='store_true', help='Don\'t save the new image')
parser.add_argument('-u', '--upload', action='store_true', help='Upload the image after creation')
parser.add_argument('--no-build', action='store_true', dest='no_build',
help='Don''t build packages and create the vm without build capabilities')
parser.add_argument("--store", default=None, help="Where to send images")
parser.add_argument('image', help='The image to create')
args = parser.parse_args()
# default to --no-build for some images
if args.image in ["candlepin", "continuous-atomic", "fedora-atomic", "ipa", "rhel-atomic", "selenium", "openshift"]:
if not args.no_build:
if args.verbose:
print("Creating machine without build capabilities based on the image type")
args.no_build = True
class MachineBuilder:
def __init__(self, machine):
tempdir = testvm.get_temp_dir()
self.machine = machine
os.makedirs(tempdir, 0o750, exist_ok=True)
# Use a tmp filename
self.target_file = self.machine.image_file
fp, self.machine.image_file = tempfile.mkstemp(dir=tempdir, prefix=self.machine.image, suffix=".qcow2")
os.close(fp)
def bootstrap_system(self):
assert not self.machine._domain
os.makedirs(self.machine.run_dir, 0o750, exist_ok=True)
bootstrap_script = os.path.join(testvm.SCRIPTS_DIR, "%s.bootstrap" % (self.machine.image, ))
if os.path.isfile(bootstrap_script):
subprocess.check_call([ bootstrap_script, self.machine.image_file ])
else:
raise testvm.Failure("Unsupported OS %s: %s not found." % (self.machine.image, bootstrap_script))
def run_setup_script(self, script):
"""Prepare a test image further by running some commands in it."""
self.machine.start()
try:
self.machine.wait_boot(timeout_sec=120)
self.machine.upload([ os.path.join(testvm.SCRIPTS_DIR, "lib") ], "/var/lib/testvm")
self.machine.upload([script], "/var/tmp/SETUP")
self.machine.upload([ os.path.join(testvm.SCRIPTS_DIR, "lib", "base") ],
"/var/tmp/cockpit-base")
if "rhel" in self.machine.image:
self.machine.upload([ os.path.expanduser("~/.rhel") ], "/root/")
env = {
"TEST_OS": self.machine.image,
"DO_BUILD": "0" if args.no_build else "1",
}
self.machine.message("run setup script on guest")
try:
self.machine.execute(script="/var/tmp/SETUP " + self.machine.image,
environment=env, quiet=not self.machine.verbose, timeout=7200)
self.machine.execute(command="rm -f /var/tmp/SETUP")
self.machine.execute(command="rm -rf /root/.rhel")
if self.machine.image == 'openshift':
# update our local openshift kube config file to match the new image
self.machine.download("/root/.kube/config", os.path.join(BOTS, "images/files/openshift.kubeconfig"))
except subprocess.CalledProcessError as ex:
if args.sit:
sys.stderr.write(self.machine.diagnose())
input ("Press RET to continue... ")
raise testvm.Failure("setup failed with code {0}\n".format(ex.returncode))
finally:
self.machine.stop(timeout_sec=500)
def boot_system(self):
"""Start the system to make sure it can boot, then shutdown cleanly
This also takes care of any selinux relabeling setup triggered
Don't wait for an ip address during start, since the system might reboot"""
self.machine.start()
try:
self.machine.wait_boot(timeout_sec=120)
finally:
self.machine.stop(timeout_sec=120)
def build(self):
self.bootstrap_system()
# gather the scripts, separated by reboots
script = os.path.join(testvm.SCRIPTS_DIR, "%s.setup" % (self.machine.image, ))
if not os.path.exists(script):
return
self.machine.message("Running setup script %s" % (script))
self.run_setup_script(script)
tries_left = 3
successfully_booted = False
while tries_left > 0:
try:
# make sure we can boot the system
self.boot_system()
successfully_booted = True
break
except:
# we might need to wait for the image to become available again
# accessing it in maintain=True mode successively can trigger qemu errors
time.sleep(3)
tries_left -= 1
if not successfully_booted:
raise testvm.Failure("Unable to verify that machine boot works.")
def save(self):
data_dir = testvm.get_images_data_dir()
os.makedirs(data_dir, 0o750, exist_ok=True)
if not os.path.exists(self.machine.image_file):
raise testvm.Failure("Nothing to save.")
partial = os.path.join(data_dir, self.machine.image + ".partial")
# Copy image via convert, to make it sparse again
subprocess.check_call([ "qemu-img", "convert", "-c", "-O", "qcow2", self.machine.image_file, partial ])
# Hash the image here
(sha, x1, x2) = subprocess.check_output([ "sha256sum", partial ], universal_newlines=True).partition(" ")
if not sha:
raise testvm.Failure("sha256sum returned invalid output")
name = self.machine.image + "-" + sha + ".qcow2"
data_file = os.path.join(data_dir, name)
shutil.move(partial, data_file)
# Remove temp image file
os.unlink(self.machine.image_file)
# Update the images symlink
if os.path.islink(self.target_file):
os.unlink(self.target_file)
os.symlink(name, self.target_file)
# Handle alternate images data directory
image_file = os.path.join(testvm.IMAGES_DIR, name)
if not os.path.exists(image_file):
os.symlink(os.path.abspath(data_file), image_file)
try:
testvm.VirtMachine.memory_mb = 2048
machine = testvm.VirtMachine(verbose=args.verbose, image=args.image, maintain=True)
builder = MachineBuilder(machine)
builder.build()
if not args.no_save:
print("Saving...")
builder.save()
if args.upload:
print("Uploading...")
cmd = [ os.path.join(BOTS, "image-upload") ]
if args.store:
cmd += [ "--store", args.store ]
cmd += [ args.image ]
subprocess.check_call(cmd)
except testvm.Failure as ex:
sys.stderr.write("image-create: %s\n" % ex)
sys.exit(1)

View File

@ -1,147 +0,0 @@
#!/usr/bin/env python3
# This file is part of Cockpit.
#
# Copyright (C) 2015 Red Hat, Inc.
#
# Cockpit is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Cockpit is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
import argparse
import os
import re
import sys
import subprocess
BOTS = os.path.abspath(os.path.dirname(__file__))
BASE = os.path.normpath(os.path.join(BOTS, ".."))
TEST = os.path.join(BASE, "test")
os.environ["PATH"] = "{0}:{1}".format(os.environ.get("PATH"), BOTS)
from machine import testvm
parser = argparse.ArgumentParser(
description='Run command inside or install packages into a Cockpit virtual machine',
formatter_class=argparse.ArgumentDefaultsHelpFormatter
)
parser.add_argument('-v', '--verbose', action='store_true', help='Display verbose progress details')
parser.add_argument('-i', '--install', action='append', dest="packagelist", default=[], help='Install packages')
parser.add_argument('-I', '--install-command', action='store', dest="installcommand",
default="yum --setopt=skip_missing_names_on_install=False -y install",
help="Command used to install packages in machine")
parser.add_argument('-r', '--run-command', action='append', dest="commandlist",
default=[], help='Run command inside virtual machine')
parser.add_argument('-s', '--script', action='append', dest="scriptlist",
default=[], help='Run selected script inside virtual machine')
parser.add_argument('-u', '--upload', action='append', dest="uploadlist",
default=[], help='Upload file/dir to destination file/dir separated by ":" example: -u file.txt:/var/lib')
parser.add_argument('--base-image', help='Base image name, if "image" does not match a standard Cockpit VM image name')
parser.add_argument('--resize', help="Resize the image. Size in bytes with using K, M, or G suffix.")
parser.add_argument('image', help='The image to use (destination name when using --base-image)')
args = parser.parse_args()
if not args.base_image:
args.base_image = os.path.basename(args.image)
args.base_image = testvm.get_test_image(args.base_image)
# Create the necessary layered image for the build/install
def prepare_install_image(base_image, install_image):
if "/" not in base_image:
base_image = os.path.join(testvm.IMAGES_DIR, base_image)
if "/" not in install_image:
install_image = os.path.join(os.path.join(TEST, "images"), os.path.basename(install_image))
# In vm-customize we don't force recreate images
if not os.path.exists(install_image):
install_image_dir = os.path.dirname(install_image)
os.makedirs(install_image_dir, exist_ok=True)
base_image = os.path.realpath(base_image)
qcow2_image = "{0}.qcow2".format(install_image)
subprocess.check_call([ "qemu-img", "create", "-q", "-f", "qcow2",
"-o", "backing_file={0},backing_fmt=qcow2".format(base_image), qcow2_image ])
if os.path.lexists(install_image):
os.unlink(install_image)
os.symlink(os.path.basename(qcow2_image), install_image)
if args.resize:
subprocess.check_call(["qemu-img", "resize", install_image, args.resize])
return install_image
def run_command(machine_instance, commandlist):
"""Run command inside image"""
for foo in commandlist:
try:
machine_instance.execute(foo, timeout=1800)
except subprocess.CalledProcessError as e:
sys.stderr.write("%s\n" % e)
sys.exit(e.returncode)
def run_script(machine_instance, scriptlist):
"""Run script inside image"""
for foo in scriptlist:
if os.path.isfile(foo):
pname = os.path.basename(foo)
uploadpath = "/var/tmp/" + pname
machine_instance.upload([os.path.abspath(foo)], uploadpath)
machine_instance.execute("chmod a+x %s" % uploadpath)
try:
machine_instance.execute(uploadpath, timeout=1800)
except subprocess.CalledProcessError as e:
sys.stderr.write("%s\n" % e)
sys.exit(e.returncode)
else:
sys.stderr.write("Bad path to script: %s\n" % foo)
def upload_files(machine_instance, uploadfiles):
"""Upload files/directories inside image"""
for foo in uploadfiles:
srcfile, dest = foo.split(":")
src_absolute = os.path.join(os.getcwd(), srcfile)
machine_instance.upload([src_absolute], dest)
def install_packages(machine_instance, packagelist, install_command):
"""Install packages into a test image
It could be done via local rpms or normal package installation
"""
allpackages = []
for foo in packagelist:
if os.path.isfile(foo):
pname = os.path.basename(foo)
machine_instance.upload([foo], "/var/tmp/" + pname)
allpackages.append("/var/tmp/" + pname)
elif not re.search("/", foo):
allpackages.append(foo)
else:
sys.stderr.write("Bad package name or path: %s\n" % foo)
if allpackages:
machine_instance.execute(install_command + " " + ' '.join(allpackages), timeout=1800)
if args.commandlist or args.packagelist or args.scriptlist or args.uploadlist or args.resize:
if '/' not in args.base_image:
subprocess.check_call(["image-download", args.base_image])
machine = testvm.VirtMachine(maintain=True,
verbose=args.verbose, image=prepare_install_image(args.base_image, args.image))
machine.start()
machine.wait_boot()
try:
if args.uploadlist:
upload_files(machine, args.uploadlist)
if args.commandlist:
run_command(machine, args.commandlist)
if args.packagelist:
install_packages(machine, args.packagelist, args.installcommand)
if args.scriptlist:
run_script(machine, args.scriptlist)
finally:
machine.stop()

View File

@ -1,305 +0,0 @@
#!/usr/bin/env python3
# This file is part of Cockpit.
#
# Copyright (C) 2013 Red Hat, Inc.
#
# Cockpit is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Cockpit is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
#
# Download images or other state
#
# Images usually have a name specific link committed to git. These
# are referred to as 'committed'
#
# Other state is simply referenced by name without a link in git
# This is referred to as 'state'
#
# The stores are places to look for images or other state
#
import argparse
import email
import io
import os
import shutil
import socket
import stat
import subprocess
import sys
import tempfile
import time
import fcntl
import urllib.parse
from machine import testvm
from task import REDHAT_STORE
CONFIG = "~/.config/image-stores"
DEFAULT = [
"http://cockpit-images.verify.svc.cluster.local",
"https://images-cockpit.apps.ci.centos.org/",
"https://209.132.184.41:8493/",
REDHAT_STORE
]
BOTS = os.path.dirname(os.path.realpath(__file__))
DEVNULL = open("/dev/null", "r+")
EPOCH = "Thu, 1 Jan 1970 00:00:00 GMT"
def find(name, stores, latest, quiet):
found = [ ]
ca = os.path.join(testvm.IMAGES_DIR, "files", "ca.pem")
for store in stores:
url = urllib.parse.urlparse(store)
defport = url.scheme == 'http' and 80 or 443
try:
ai = socket.getaddrinfo(url.hostname, url.port or defport, socket.AF_INET, 0, socket.IPPROTO_TCP)
except socket.gaierror:
ai = [ ]
message = store
for (family, socktype, proto, canonname, sockaddr) in ai:
message = "{scheme}://{0}:{1}{path}".format(*sockaddr, scheme=url.scheme, path=url.path)
def curl(*args):
try:
cmd = ["curl"] + list(args) + ["--head", "--silent", "--fail", "--cacert", ca, source]
start = time.time()
output = subprocess.check_output(cmd, universal_newlines=True)
found.append((cmd, output, message, time.time() - start))
if not quiet:
sys.stderr.write(" > {0}\n".format(message))
return True
except subprocess.CalledProcessError:
return False
# first try with stores that accept the "cockpit-tests" host name
resolve = "cockpit-tests:{1}:{0}".format(*sockaddr)
source = urllib.parse.urljoin("{0}://cockpit-tests:{1}{2}".format(url.scheme, sockaddr[1], url.path), name)
if curl("--resolve", resolve):
continue
# fall back for OpenShift proxied stores which send their own SSL cert initially; host name has to match that
source = urllib.parse.urljoin(store, name)
if curl():
continue
if not quiet:
sys.stderr.write(" x {0}\n".format(message))
# If we couldn't find the file, but it exists, we're good
if not found:
return None, None
# Find the most recent version of this file
def header_date(args):
cmd, output, message, latency = args
try:
reply_line, headers_alone = output.split('\n', 1)
last_modified = email.message_from_file(io.StringIO(headers_alone)).get("Last-Modified", "")
return time.mktime(time.strptime(last_modified, '%a, %d %b %Y %H:%M:%S %Z'))
except ValueError:
return ""
if latest:
found.sort(reverse=True, key=header_date)
else:
found.sort(reverse=False, key=lambda x: x[3])
# Return the command and message
return found[0][0], found[0][2]
def download(dest, force, state, quiet, stores):
if not stores:
config = os.path.expanduser(CONFIG)
if os.path.exists(config):
with open(config, 'r') as fp:
stores = fp.read().strip().split("\n")
else:
stores = []
stores += DEFAULT
# The time condition for If-Modified-Since
exists = not force and os.path.exists(dest)
if exists:
since = dest
else:
since = EPOCH
name = os.path.basename(dest)
cmd, message = find(name, stores, latest=state, quiet=quiet)
# If we couldn't find the file, but it exists, we're good
if not cmd:
if exists:
return
raise RuntimeError("image-download: couldn't find file anywhere: {0}".format(name))
# Choose the first found item after sorting by date
if not quiet:
sys.stderr.write(" > {0}\n".format(urllib.parse.urljoin(message, name)))
temp = dest + ".partial"
# Adjust the command above that worked to make it visible and download real stuff
cmd.remove("--head")
cmd.append("--show-error")
if not quiet and os.isatty(sys.stdout.fileno()):
cmd.remove("--silent")
cmd.insert(1, "--progress-bar")
cmd.append("--remote-time")
cmd.append("--time-cond")
cmd.append(since)
cmd.append("--output")
cmd.append(temp)
if os.path.exists(temp):
if force:
os.remove(temp)
else:
cmd.append("-C")
cmd.append("-")
# Always create the destination file (because --state)
else:
open(temp, 'a').close()
curl = subprocess.Popen(cmd)
ret = curl.wait()
if ret != 0:
raise RuntimeError("curl: unable to download %s (returned: %s)" % (message, ret))
os.chmod(temp, stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH)
# Due to time-cond the file size may be zero
# A new file downloaded, put it in place
if not exists or os.path.getsize(temp) > 0:
shutil.move(temp, dest)
# Calculate a place to put images where links are not committed in git
def state_target(path):
data_dir = testvm.get_images_data_dir()
os.makedirs(data_dir, mode=0o775, exist_ok=True)
return os.path.join(data_dir, path)
# Calculate a place to put images where links are committed in git
def committed_target(image):
link = os.path.join(testvm.IMAGES_DIR, image)
if not os.path.islink(link):
raise RuntimeError("image link does not exist: " + image)
dest = os.readlink(link)
relative_dir = os.path.dirname(os.path.abspath(link))
full_dest = os.path.join(relative_dir, dest)
while os.path.islink(full_dest):
link = full_dest
dest = os.readlink(link)
relative_dir = os.path.dirname(os.path.abspath(link))
full_dest = os.path.join(relative_dir, dest)
dest = os.path.join(testvm.get_images_data_dir(), dest)
# We have the file but there is not valid link
if os.path.exists(dest):
try:
os.symlink(dest, os.path.join(testvm.IMAGES_DIR, os.readlink(link)))
except FileExistsError:
pass
# The image file in the images directory, may be same as dest
image_file = os.path.join(testvm.IMAGES_DIR, os.readlink(link))
# Double check that symlink in place but never make a cycle.
if os.path.abspath(dest) != os.path.abspath(image_file):
try:
os.symlink(os.path.abspath(dest), image_file)
except FileExistsError:
pass
return dest
def wait_lock(target):
lockfile = os.path.join(tempfile.gettempdir(), ".cockpit-test-resources", os.path.basename(target) + ".lock")
os.makedirs(os.path.dirname(lockfile), exist_ok=True)
# we need to keep the lock fd open throughout the entire runtime, so remember it in a global-scoped variable
wait_lock.f = open(lockfile, "w")
for retry in range(360):
try:
fcntl.flock(wait_lock.f, fcntl.LOCK_NB | fcntl.LOCK_EX)
return
except BlockingIOError:
if retry == 0:
print("Waiting for concurrent image-download of %s..." % os.path.basename(target))
time.sleep(10)
else:
raise TimeoutError("timed out waiting for concurrent downloads of %s\n" % target)
def download_images(image_list, force, quiet, state, store):
data_dir = testvm.get_images_data_dir()
os.makedirs(data_dir, exist_ok=True)
# A default set of images are all links in git. These links have
# no directory part. Other links might exist, such as the
# auxiliary links created by committed_target above, and we ignore
# them.
if not image_list:
image_list = []
if not state:
for filename in os.listdir(testvm.IMAGES_DIR):
link = os.path.join(testvm.IMAGES_DIR, filename)
if os.path.islink(link) and os.path.dirname(os.readlink(link)) == "":
image_list.append(filename)
success = True
for image in image_list:
image = testvm.get_test_image(image)
try:
if state:
target = state_target(image)
else:
target = committed_target(image)
# don't download the same thing multiple times in parallel
wait_lock(target)
if force or state or not os.path.exists(target):
download(target, force, state, quiet, store)
except Exception as ex:
success = False
sys.stderr.write("image-download: {0}\n".format(str(ex)))
return success
def main():
parser = argparse.ArgumentParser(description='Download a bot state or images')
parser.add_argument("--force", action="store_true", help="Force unnecessary downloads")
parser.add_argument("--store", action="append", help="Where to find state or images")
parser.add_argument("--quiet", action="store_true", help="Make downloading quieter")
parser.add_argument("--state", action="store_true", help="Images or state not recorded in git")
parser.add_argument('image', nargs='*')
args = parser.parse_args()
if not download_images(args.image, args.force, args.quiet, args.state, args.store):
return 1
return 0
if __name__ == '__main__':
sys.exit(main())

View File

@ -1,218 +0,0 @@
#!/usr/bin/env python3
# This file is part of Cockpit.
#
# Copyright (C) 2013 Red Hat, Inc.
#
# Cockpit is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Cockpit is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
# Days after which images expire if not in use
IMAGE_EXPIRE = 14
import argparse
import os
import subprocess
import sys
import time
from contextlib import contextmanager
from task import github
from machine import testvm
BOTS = os.path.dirname(os.path.realpath(__file__))
# threshold in G below which unreferenced qcow2 images will be pruned, even if they aren't old
PRUNE_THRESHOLD_G = float(os.environ.get("PRUNE_THRESHOLD_G", 15))
def enough_disk_space():
"""Check if available disk space in our data store is sufficient
"""
st = os.statvfs(testvm.get_images_data_dir())
free = st.f_bavail * st.f_frsize / (1024*1024*1024)
return free >= PRUNE_THRESHOLD_G;
def get_refs(open_pull_requests=True, offline=False):
"""Return dictionary for available refs of the format {'rhel-7.4': 'ad50328990e44c22501bd5e454746d4b5e561b7c'}
Expects to be called from the top level of the git checkout
If offline is true, git show-ref is used instead of listing the remote
"""
# get all remote heads and filter empty lines
# output of ls-remote has the format
#
# d864d3792db442e3de3d1811fa4bc371793a8f4f refs/heads/master
# ad50328990e44c22501bd5e454746d4b5e561b7c refs/heads/rhel-7.4
refs = { }
considerable = {}
g = github.GitHub()
if open_pull_requests:
if offline:
raise Exception("Unable to consider open pull requests when in offline mode")
for p in g.pulls():
files = g.get("pulls/{0}/files".format(p["number"]))
images = []
for fl in files:
fl_name = fl['filename']
if fl_name.startswith("bots/images/"):
fl_name_split = fl_name.split("/", 2)
if "/" not in fl_name_split[2]:
images.append(fl_name_split[2])
if images:
sha = p["head"]["sha"]
considerable[sha] = images
subprocess.call(["git", "fetch", "origin", "pull/{0}/head".format(p["number"])])
refs["pull request #{} ({})".format(p["number"], p["title"])] = sha
git_cmd = "show-ref" if offline else "ls-remote"
ref_output = subprocess.check_output(["git", git_cmd], universal_newlines=True).splitlines()
# filter out the "refs/heads/" prefix and generate a dictionary
prefix = "refs/heads"
for ln in ref_output:
[ref, name] = ln.split()
if name.startswith(prefix):
refs[name[len(prefix):]] = ref
return (refs, considerable)
def get_image_links(ref, git_path):
"""Return all image links for the given git ref
Expects to be called from the top level of the git checkout
"""
# get all the links we have first
# trailing slash on path is important
if not git_path.endswith("/"):
git_path = "{0}/".format(git_path)
try:
entries = subprocess.check_output(["git", "ls-tree", "--name-only", ref, git_path], universal_newlines=True).splitlines()
except subprocess.CalledProcessError as e:
if e.returncode == 128:
sys.stderr.write("Skipping {0} due to tree error.\n".format(ref))
return []
raise
links = [subprocess.check_output(["git", "show", "{0}:{1}".format(ref, entry)], universal_newlines=True) for entry in entries]
return [link for link in links if link.endswith(".qcow2")]
@contextmanager
def remember_cwd():
curdir = os.getcwd()
try:
yield
finally:
os.chdir(curdir)
def get_image_names(quiet=False, open_pull_requests=True, offline=False):
"""Return all image names used by all branches and optionally in open pull requests
"""
images = set()
# iterate over visible refs (mostly branches)
# this hinges on being in the top level directory of the the git checkout
with remember_cwd():
os.chdir(os.path.join(BOTS, ".."))
(refs, considerable) = get_refs(open_pull_requests, offline)
# list images present in each branch / pull request
for name, ref in refs.items():
if not quiet:
sys.stderr.write("Considering images from {0} ({1})\n".format(name, ref))
for link in get_image_links(ref, "bots/images"):
if ref in considerable:
for consider in considerable[ref]:
if link.startswith(consider):
images.add(link)
else:
images.add(link)
return images
def prune_images(force, dryrun, quiet=False, open_pull_requests=True, offline=False, checkout_only=False):
"""Prune images
"""
now = time.time()
# everything we want to keep
if checkout_only:
targets = set()
else:
targets = get_image_names(quiet, open_pull_requests, offline)
# what we have in the current checkout might already have been added by its branch, but check anyway
for filename in os.listdir(testvm.IMAGES_DIR):
path = os.path.join(testvm.IMAGES_DIR, filename)
# only consider original image entries as trustworthy sources and ignore non-links
if path.endswith(".qcow2") or path.endswith(".partial") or not os.path.islink(path):
continue
target = os.readlink(path)
targets.add(target)
expiry_threshold = now - IMAGE_EXPIRE * 86400
for filename in os.listdir(testvm.get_images_data_dir()):
path = os.path.join(testvm.get_images_data_dir(), filename)
if not force and (enough_disk_space() and os.lstat(path).st_mtime > expiry_threshold):
continue
if os.path.isfile(path) and (path.endswith(".xz") or path.endswith(".qcow2") or path.endswith(".partial")) and filename not in targets:
if not quiet or dryrun:
sys.stderr.write("Pruning {0}\n".format(filename))
if not dryrun:
os.unlink(path)
# now prune broken links
for filename in os.listdir(testvm.IMAGES_DIR):
path = os.path.join(testvm.IMAGES_DIR, filename)
# don't prune original image entries and ignore non-links
if not path.endswith(".qcow2") or not os.path.islink(path):
continue
# if the link isn't valid, prune
if not os.path.isfile(path):
if not quiet or dryrun:
sys.stderr.write("Pruning link {0}\n".format(path))
if not dryrun:
os.unlink(path)
def every_image():
result = []
for filename in os.listdir(testvm.IMAGES_DIR):
link = os.path.join(testvm.IMAGES_DIR, filename)
if os.path.islink(link):
result.append(filename)
return result
def main():
parser = argparse.ArgumentParser(description='Prune downloaded images')
parser.add_argument("--force", action="store_true", help="Delete images even if they aren't old")
parser.add_argument("--quiet", action="store_true", help="Make downloading quieter")
parser.add_argument("-d", "--dry-run-prune", dest="dryrun", action="store_true", help="Don't actually delete images and links")
parser.add_argument("-b", "--branches-only", dest="branches_only", action="store_true", help="Don't consider pull requests on GitHub, only look at branches")
parser.add_argument("-c", "--checkout-only", dest="checkout_only", action="store_true", help="Consider neither pull requests on GitHub nor branches, only look at the current checkout")
parser.add_argument("-o", "--offline", dest="offline", action="store_true", help="Don't access external sources such as GitHub")
args = parser.parse_args()
try:
prune_images(args.force, args.dryrun, quiet=args.quiet, open_pull_requests=(not args.branches_only), offline=args.offline, checkout_only=args.checkout_only)
except RuntimeError as ex:
sys.stderr.write("image-prune: {0}\n".format(str(ex)))
return 1
return 0
if __name__ == '__main__':
sys.exit(main())

View File

@ -1,169 +0,0 @@
#!/usr/bin/env python3
# This file is part of Cockpit.
#
# Copyright (C) 2016 Red Hat, Inc.
#
# Cockpit is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Cockpit is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
import os
import subprocess
import sys
import task
from task import github, REDHAT_STORE
TRIGGERS = {
"centos-7": [
"centos-7@cockpit-project/starter-kit",
],
"continuous-atomic": [
"continuous-atomic@cockpit-project/cockpit-ostree",
"continuous-atomic@cockpit-project/cockpit/rhel-7.7",
],
"debian-testing": [
"debian-testing"
],
"debian-stable": [
"debian-stable"
],
"fedora-29": [
"fedora-atomic",
"fedora-29@cockpit-project/cockpit-podman",
],
"fedora-30": [
"fedora-30",
"fedora-30/selenium-chrome",
"fedora-30/selenium-firefox",
"fedora-30/selenium-edge",
"fedora-30/container-bastion",
"fedora-30@cockpit-project/starter-kit",
"fedora-30@cockpit-project/cockpit-podman",
"fedora-30@weldr/lorax",
"fedora-30/live-iso@weldr/lorax",
"fedora-30/qcow2@weldr/lorax",
"fedora-30/chrome@weldr/cockpit-composer",
"fedora-30/firefox@weldr/cockpit-composer",
"fedora-30/edge@weldr/cockpit-composer",
],
"fedora-31": [
"fedora-31",
"fedora-31@cockpit-project/cockpit-podman",
],
"fedora-atomic": [
"fedora-atomic",
"fedora-atomic@cockpit-project/cockpit-ostree",
],
"fedora-testing": [
"fedora-testing"
],
"fedora-i386": [
"fedora-i386"
],
"ubuntu-1804": [
"ubuntu-1804"
],
"ubuntu-stable": [
"ubuntu-stable"
],
"openshift": [
"rhel-7-7@cockpit-project/cockpit/rhel-7.7",
"rhel-7-8@cockpit-project/cockpit/rhel-7.8",
],
"ipa": [
"fedora-30",
"ubuntu-1804",
"debian-stable"
],
"selenium": [
"fedora-30/selenium-chrome",
"fedora-30/selenium-firefox",
"fedora-30/chrome@weldr/cockpit-composer",
"fedora-30/firefox@weldr/cockpit-composer",
"rhel-7-7/firefox@weldr/cockpit-composer",
"rhel-8-1/chrome@weldr/cockpit-composer",
],
"rhel-7-7": [
"rhel-7-7/firefox@weldr/cockpit-composer",
"rhel-7-7@cockpit-project/cockpit/rhel-7.7",
"rhel-atomic@cockpit-project/cockpit/rhel-7.7",
"continuous-atomic@cockpit-project/cockpit/rhel-7.7",
],
"rhel-7-8": [
"rhel-7-8@cockpit-project/cockpit/rhel-7.8",
],
"rhel-8-0": [
"rhel-8-0@cockpit-project/cockpit/rhel-8.0",
],
"rhel-8-1": [
"rhel-8-1",
"rhel-8-1-distropkg",
"rhel-8-1@cockpit-project/cockpit/rhel-8.1",
"rhel-8-1@cockpit-project/cockpit/rhel-8-appstream",
"rhel-8-1/chrome@weldr/cockpit-composer",
"rhel-8-1@cockpit-project/cockpit-podman",
],
"rhel-atomic": [
"rhel-atomic@cockpit-project/cockpit-ostree",
"rhel-atomic@cockpit-project/cockpit/rhel-7.7",
]
}
STORES = {
"rhel-7-7": REDHAT_STORE,
"rhel-7-8": REDHAT_STORE,
"rhel-8-0": REDHAT_STORE,
"rhel-8-1": REDHAT_STORE,
"rhel-atomic": REDHAT_STORE,
"windows-10": REDHAT_STORE,
}
BOTS = os.path.abspath(os.path.dirname(__file__))
BASE = os.path.normpath(os.path.join(BOTS, ".."))
sys.dont_write_bytecode = True
def run(image, verbose=False, **kwargs):
if not image:
raise RuntimeError("no image specified")
triggers = TRIGGERS.get(image, [ ])
store = STORES.get(image, None)
# Cleanup any extraneous disk usage elsewhere
subprocess.check_call([ os.path.join(BOTS, "vm-reset") ])
cmd = [ os.path.join(BOTS, "image-create"), "--verbose", "--upload" ]
if store:
cmd += [ "--store", store ]
cmd += [ image ]
os.environ['VIRT_BUILDER_NO_CACHE'] = "yes"
ret = subprocess.call(cmd)
if ret:
return ret
branch = task.branch(image, "images: Update {0} image".format(image), pathspec="bots/images", **kwargs)
if branch:
pull = task.pull(branch, labels=['bot', 'no-test'], run_tests=False, **kwargs)
# Trigger this pull request
api = github.GitHub()
head = pull["head"]["sha"]
for trigger in triggers:
api.post("statuses/{0}".format(head), { "state": "pending", "context": trigger,
"description": github.NOT_TESTED_DIRECT })
if __name__ == '__main__':
task.main(function=run, title="Refresh image")

View File

@ -1,111 +0,0 @@
#!/usr/bin/env python3
# This file is part of Cockpit.
#
# Copyright (C) 2015 Red Hat, Inc.
#
# Cockpit is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Cockpit is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
DAYS = 7
REFRESH = {
"candlepin": { "refresh-days": 120 },
"centos-7": { },
"continuous-atomic": { },
"debian-testing": { },
"debian-stable": { },
"fedora-29": { },
"fedora-30": { },
"fedora-31": { },
"fedora-atomic": { },
"fedora-testing": { },
"fedora-i386": { },
"ipa": { "refresh-days": 120 },
"ubuntu-1804": { },
"ubuntu-stable": { },
"openshift": { "refresh-days": 30 },
'rhel-7-7': { },
'rhel-8-0': { },
'rhel-8-1': { },
'rhel-atomic': { },
"selenium": { "refresh-days": 30 },
}
import argparse
import os
import sys
import tempfile
import time
import subprocess
sys.dont_write_bytecode = True
import task
from task import github
def main():
parser = argparse.ArgumentParser(description='Ensure necessary issue exists for image refresh')
parser.add_argument('-v', '--verbose', action="store_true", default=False,
help="Print verbose information")
parser.add_argument("image", nargs="?")
opts = parser.parse_args()
api = github.GitHub()
try:
results = scan(api, opts.image, opts.verbose)
except RuntimeError as ex:
sys.stderr.write("image-trigger: " + str(ex) + "\n")
return 1
for result in results:
if result:
sys.stdout.write(result + "\n")
return 0
# Prepare an image prune command
def scan_for_prune():
tasks = [ ]
stamp = os.path.join(tempfile.gettempdir(), "cockpit-image-prune.stamp")
# Don't prune more than once per hour
try:
mtime = os.stat(stamp).st_mtime
except OSError:
mtime = 0
if mtime < time.time() - 3600:
tasks.append("PRIORITY=0000 touch {0} && bots/image-prune".format(stamp))
return tasks
def scan(api, force, verbose):
subprocess.check_call([ "git", "fetch", "origin", "master" ])
for (image, options) in REFRESH.items():
perform = False
if force:
perform = image == force
else:
days = options.get("refresh-days", DAYS)
perform = task.stale(days, os.path.join("bots", "images", image), "origin/master")
if perform:
text = "Image refresh for {0}".format(image)
issue = task.issue(text, text, "image-refresh", image)
sys.stderr.write("#{0}: image-refresh {1}\n".format(issue["number"], image))
return scan_for_prune()
if __name__ == '__main__':
sys.exit(main())

View File

@ -1,120 +0,0 @@
#!/usr/bin/env python3
# This file is part of Cockpit.
#
# Copyright (C) 2013 Red Hat, Inc.
#
# Cockpit is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Cockpit is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
# The default settings here should match one of the default download stores
DEFAULT_UPLOAD = [
"https://images-cockpit.apps.ci.centos.org/",
"https://209.132.184.41:8493/",
]
TOKEN = "~/.config/github-token"
import argparse
import getpass
import errno
import os
import socket
import subprocess
import sys
import urllib.parse
from machine import testvm
BOTS = os.path.dirname(__file__)
def upload(store, source):
ca = os.path.join(BOTS, "images", "files", "ca.pem")
url = urllib.parse.urlparse(store)
# Start building the command
cmd = ["curl", "--progress-bar", "--cacert", ca, "--fail", "--upload-file", source ]
def try_curl(cmd):
print("Uploading to", cmd[-1])
# Passing through a non terminal stdout is necessary to make progress work
curl = subprocess.Popen(cmd, stdout=subprocess.PIPE)
cat = subprocess.Popen(["cat"], stdin=curl.stdout)
curl.stdout.close()
ret = curl.wait()
cat.wait()
if ret != 0:
sys.stderr.write("image-upload: unable to upload image: {0}\n".format(cmd[-1]))
return ret
# Parse the user name and token, if present
user = url.username or getpass.getuser()
try:
with open(os.path.expanduser(TOKEN), "r") as gt:
token = gt.read().strip()
cmd += [ "--user", user + ":" + token ]
except IOError as exc:
if exc.errno == errno.ENOENT:
pass
# First try to use the original store URL, for stores with valid SSL cert on an OpenShift proxy
if try_curl(cmd + [store]) == 0:
return 0
# Fall back for stores that use our self-signed cockpit certificate
# Parse out the actual address to connect to and override certificate info
defport = url.scheme == 'http' and 80 or 443
ai = socket.getaddrinfo(url.hostname, url.port or defport, socket.AF_INET, 0, socket.IPPROTO_TCP)
for (family, socktype, proto, canonname, sockaddr) in ai:
resolve = "cockpit-tests:{1}:{0}".format(*sockaddr)
curl_url = "https://cockpit-tests:{0}{1}".format(url.port or defport, url.path)
ret = try_curl(cmd + ["--resolve", resolve, curl_url])
if ret == 0:
return 0
return 1
def main():
parser = argparse.ArgumentParser(description='Upload bot state or images')
parser.add_argument("--store", action="append", default=[], help="Where to send state or images")
parser.add_argument("--state", action="store_true", help="Images or state not recorded in git")
parser.add_argument('image', nargs='*')
args = parser.parse_args()
data_dir = testvm.get_images_data_dir()
sources = []
for image in args.image:
if args.state:
source = os.path.join(data_dir, image)
else:
link = os.path.join(testvm.IMAGES_DIR, image)
if not os.path.islink(link):
parser.error("image link does not exist: " + image)
source = os.path.join(data_dir, os.readlink(link))
if not os.path.isfile(source):
parser.error("image does not exist: " + image)
sources.append(source)
for source in sources:
for store in (args.store or DEFAULT_UPLOAD):
ret = upload(store, source)
if ret == 0:
return ret
else:
# all stores failed, so return last exit code
return ret
if __name__ == '__main__':
sys.exit(main())

View File

@ -1 +0,0 @@
candlepin-3a39cecb7d2fea2e75b0093a891b3c476141406e20f332cb2a12f2dfb6e9d275.qcow2

View File

@ -1 +0,0 @@
centos-7-b12881afba5b51520073d9633295a89121e4f52b2e8aee4e2c422a95064a902f.qcow2

View File

@ -1 +0,0 @@
cirros-d5fcb44e05f2dafc7eaab6bce906ba9cc06af51f84f1e7a527fe12102e34bbcf.qcow2

View File

@ -1 +0,0 @@
continuous-atomic-a4c3407c689e53e6864f7ba92b95a0d7eea42b04545a7aeb77271b6f5521bd08.qcow2

View File

@ -1 +0,0 @@
debian-stable-980197e60a14278239e4c06b39421e9bd16d9ad97e1d4bb7d405b8fb2a8b15a0.qcow2

View File

@ -1 +0,0 @@
debian-testing-da77a67318002005f72e5f988978198fe6a53bea4bd60f2727a4123e98d6dd26.qcow2

View File

@ -1 +0,0 @@
fedora-23-stock-1a7ce615dcf1772ff6514148513fc88e420b9179f32c5395e3a27dab3b107dcc.qcow2

View File

@ -1 +0,0 @@
fedora-29-7191db4290794ba7bcf8eb30739d07a6cf59b072513b813b97755adb69162a95.qcow2

View File

@ -1 +0,0 @@
fedora-30-219724b87c59cbb21378c3f1e5fe80e8963072dc0fa97c15c809cf19118f8433.qcow2

View File

@ -1 +0,0 @@
fedora-31-632704fccac608d265572c0500eeecae3d61e0d932683adb5b5092980032b3ef.qcow2

View File

@ -1 +0,0 @@
fedora-atomic-6a63990b2443f568bb3321efe55fe8bad8891d128ec4c5b818303cd52a34e1e0.qcow2

View File

@ -1 +0,0 @@
fedora-i386-f5c6c9730facd6b7d00d5c07f59cf7bf3a9ce3de1270f174cf5d9aefcd86a297.qcow2

View File

@ -1 +0,0 @@
stock-fedora-22-x86_64-2.qcow2

View File

@ -1 +0,0 @@
fedora-testing-fedf1d06768b7cb69efbb2ef27ae665161c938d94f844d63de3c3fb20f509b8f.qcow2

View File

@ -1,21 +0,0 @@
# This is the CA for cockpit-tests images and data
-----BEGIN CERTIFICATE-----
MIIDDDCCAfSgAwIBAgIJANdoyGJiUz+8MA0GCSqGSIb3DQEBCwUAMDUxEDAOBgNV
BAoMB0NvY2twaXQxFDASBgNVBAsMC0NvY2twaXR1b3VzMQswCQYDVQQDDAJDQTAg
Fw0xOTAyMDcxMDE4NDNaGA8zMDE4MDYxMDEwMTg0M1owNTEQMA4GA1UECgwHQ29j
a3BpdDEUMBIGA1UECwwLQ29ja3BpdHVvdXMxCzAJBgNVBAMMAkNBMIIBIjANBgkq
hkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAnvIZetd5yEhdE0c/9lYp1mC4M6qiu6E2
wVMbJLwsOuCyCSaZs5eDap1kremHz7ms+Fq07TUsN/o5U7PBnNgM3z6Zbv78QN6R
wn6ovLHfCyVqpg0nPMh3Hzpd0HDZQ+3eBayL2xfmBhU8p1+/vWVBOe49SDO15YDM
/Ian7I/HRsnprz5PH3atquSf+B8/Q+lgbO0dHKhXlbnTsSy/Esee82HhYrDlxD3p
Ow7EcZ7HACh/2dvF70BQpjnxTEc//4LNgP7hiqk4phsGzM/9QSFHW8ol4XlBDUi0
F5nNXZTs3jKITTOeda5mppuKoZoC+7iFk8dLvV0Y187xD38X2XgGnwIDAQABox0w
GzAMBgNVHRMEBTADAQH/MAsGA1UdDwQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAQEA
PHaVKb97ZN2m/sEVU+TGepVhCZ15frIaCJRuBPEs5rwcJjIctyRF4H6R6ec2b2lB
6ni9eqU6pPgS+rVJPsxqCpelQiCZALR7FYoA6+FtfpLkB5+zwJUfexr7Q6I7llWI
8OBOmtEADRv//2D+Iu6mM6nkzUK1K/wCcFS//roLjK/nKH2xd2lWbYk2Ro+nTPIm
slwgk6fAUXQcd5v/XqrySZ5jny73jMqo7SRVC5suNuAfiT0/YGvE5N99+I5AkD5I
R/R80/w1bDExfcqtx5UPBitMG2bx/gA07k4XbAGsEH5zvIdgsV9S5uYQEDjIRZys
ScLMpNOd3JyD7ncvr6Ga6g==
-----END CERTIFICATE-----

View File

@ -1,37 +0,0 @@
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM2akNDQWRLZ0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREFtTVNRd0lnWURWUVFEREJ0dmNHVnUKYzJocFpuUXRjMmxuYm1WeVFERTFOamMxTWpjMk56UXdIaGNOTVRrd09UQXpNVFl5TVRFeldoY05NalF3T1RBeApNVFl5TVRFMFdqQW1NU1F3SWdZRFZRUUREQnR2Y0dWdWMyaHBablF0YzJsbmJtVnlRREUxTmpjMU1qYzJOelF3CmdnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUURQOUVKZ05NdEk0TG1KSk9wRGNiUlQKbWNDazF1amJiQVVydTU3bjFFY0ZnTEhDNzZCQmxsME9sbzNQcnRUVWVTR0d5c3R3eW5HT0J3MytGOFBzL1hjZApwWmwrdGkySkMweGVVSnFNVDMyclRwTHVwWG1tczZGMi90d0dTc0lSTWx5NjgyZ01NaHlmV0wrT2FRZERaM3NWCjd6aDBPd2E3cE5wWWUwSE1VbUM5QzFPaEltc254YVF2Mzh4TGU5SjgvQXZxSXZMV21Wc1J3cnEveWhVakphOWkKMkZYV20wa2Z4WjBNckpRQWI2cGlTZzVEbE5pc2htbURPbU1QR21mWlI5ZkY2SHA5MWRuV3lId2NCVmpnVTJHagpRMmlqWWh5WGRMYS9RaDEzTG9BQUFiQ29aUjVKcVNUL3luQU1SWlFXTE1CRFYvSC95Wk53RFI1WUJ1YXI4LzlsCkFnTUJBQUdqSXpBaE1BNEdBMVVkRHdFQi93UUVBd0lDcERBUEJnTlZIUk1CQWY4RUJUQURBUUgvTUEwR0NTcUcKU0liM0RRRUJDd1VBQTRJQkFRQXdWSDk2SGtTQ1NWTlRXbmJOK0R1TE1Kb0l0M3ZRTW5aQ2hwcnAzWUxiZ0MvZgpxeGMxUmtXaXBTemNhYUJQSHU4RkR6aTlMOUpFcVIwVWhyeHN5Sk9iSGxJOVB5cnN2WnhpR05pc2UvOE1IKzRtCjFlMUVDZGNlT3pHOVJlK09SOGV4b25GaitJSk9ZNG9xanVtRFM2ZmdRS01Ja08vN29SZmhxRGZJREMveVUvTnUKc09xYmZnS1dMeWxxOWJKTEtQVUkwemw3YnUrSmNyK3g1anhQaTFLY05yY1BXaXNoVFpXNExrakh0Wkl5QkNyLwp5MTQ2eHZLc1hHMXY2aEJ4ZTBvRnFrcVNqMzhUYTRXV2NNUTVXd1lxbU5xMVhkRFN0T0UzMm1iZmNMZWJXSXRuCitDVGVOcldOSnQxQXE4Q3p2UDdOMEwwVkxrT1NPSmpoTVMzNVdKUnYKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
server: https://10.111.112.101:8443
name: 10-111-112-101:8443
contexts:
- context:
cluster: 10-111-112-101:8443
user: scruffy/10-111-112-101:8443
name: /10-111-112-101:8443/scruffy
- context:
cluster: 10-111-112-101:8443
namespace: default
user: system:admin/10-111-112-101:8443
name: default/10-111-112-101:8443/system:admin
- context:
cluster: 10-111-112-101:8443
namespace: marmalade
user: scruffy/10-111-112-101:8443
name: marmalade/10-111-112-101:8443/scruffy
- context:
cluster: 10-111-112-101:8443
namespace: pizzazz
user: scruffy/10-111-112-101:8443
name: pizzazz/10-111-112-101:8443/scruffy
current-context: default/10-111-112-101:8443/system:admin
kind: Config
preferences: {}
users:
- name: scruffy/10-111-112-101:8443
user:
token: S6xMoSAI-Rs4shOP0aU4Y4hMEqQJB2KlBAHU1q86M5Y
- name: system:admin/10-111-112-101:8443
user:
client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURKRENDQWd5Z0F3SUJBZ0lCQ0RBTkJna3Foa2lHOXcwQkFRc0ZBREFtTVNRd0lnWURWUVFEREJ0dmNHVnUKYzJocFpuUXRjMmxuYm1WeVFERTFOamMxTWpjMk56UXdIaGNOTVRrd09UQXpNVFl5TVRFNFdoY05NakV3T1RBeQpNVFl5TVRFNVdqQk9NVFV3RlFZRFZRUUtFdzV6ZVhOMFpXMDZiV0Z6ZEdWeWN6QWNCZ05WQkFvVEZYTjVjM1JsCmJUcGpiSFZ6ZEdWeUxXRmtiV2x1Y3pFVk1CTUdBMVVFQXhNTWMzbHpkR1Z0T21Ga2JXbHVNSUlCSWpBTkJna3EKaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF0UTJvT1NhaHlBSnpVdlYxMmtKdXc5ZFBjaDFiT29CNgpRNW1vdEM3b0k0V1F3SnhISnBWTWIxRkhWTm96dnNJQWZEMTZiaXZGd1VLWk5iQmRQNlVCZy90MlRST1JJTHZRCjBvYUg5OEd5NGZXUEE3TE9kekZ3THcxeW1BTk9UNjdYZFpUdzB0cEhOQUN6akgyd0t1cS9MRTFoVDNpTGFKSFEKQVRxdzRuOVNFSFQwbVJTYTBDcU5HRWUrZkxZQXNrd3FPOHE0UW5NOWY4QTdHUVdxY29lMFdlVDRha1VYOUVlbwpONUg1Vmt2V0VuMHhmTXBYWHQ3VGM0YUlPbk9Ba3lJbnkwQzU4TVhDNlJibHo0KzN1cHNwRlB3ZTJtS1o5bXliClZrWFcrZjJQcjJaSEZJdS9ORmx1MFJnOEdmWVVMRHgvTG9TSGFxbzBzRndNTFpwZ1RjTzdzd0lEQVFBQm96VXcKTXpBT0JnTlZIUThCQWY4RUJBTUNCYUF3RXdZRFZSMGxCQXd3Q2dZSUt3WUJCUVVIQXdJd0RBWURWUjBUQVFILwpCQUl3QURBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQUVGRzQyS3lyNkFBNCsrdjU4M1VablRlVE4xdFdLYWlLCjRURjdlWFNQNG1TRmtrYjF1RGpuekphZWUwejJUMTBpVkUvNGdsOXhKcXVUZ3k4b0RQSmFSUFJuc0kxbGVLTzkKMi90bkFJQ3kzbE5wL2U4MksybFZGVDcvdlEwL3Nqd1lVaVAzQnRoQTZkdHpMUWpFTE1abytNZm1ucnJCM0k2TgpCelk1a1pTN2hENTdZQ2FCZmRjZ0hlNWQzY0p4ZEQ4RGhMNVNBemFUTUsrcW54ZXEvU2U2TU42alRYOVBnamxIClJjK2liSnhYVkh2TmUwNC9sN0I3S2pGVW9qRG1aTE8rK290RFdldmx4SHRNRzNuV21POWo1Z0V4NVlLaXJPcjkKSTU0NHVEUGlINkdJQUdENytwaFVpMzFMVTFuYmIreWN2akpvWWVlTG1WRERzWGZ6Ri9xUzN3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBdFEyb09TYWh5QUp6VXZWMTJrSnV3OWRQY2gxYk9vQjZRNW1vdEM3b0k0V1F3SnhICkpwVk1iMUZIVk5venZzSUFmRDE2Yml2RndVS1pOYkJkUDZVQmcvdDJUUk9SSUx2UTBvYUg5OEd5NGZXUEE3TE8KZHpGd0x3MXltQU5PVDY3WGRaVHcwdHBITkFDempIMndLdXEvTEUxaFQzaUxhSkhRQVRxdzRuOVNFSFQwbVJTYQowQ3FOR0VlK2ZMWUFza3dxTzhxNFFuTTlmOEE3R1FXcWNvZTBXZVQ0YWtVWDlFZW9ONUg1Vmt2V0VuMHhmTXBYClh0N1RjNGFJT25PQWt5SW55MEM1OE1YQzZSYmx6NCszdXBzcEZQd2UybUtaOW15YlZrWFcrZjJQcjJaSEZJdS8KTkZsdTBSZzhHZllVTER4L0xvU0hhcW8wc0Z3TUxacGdUY083c3dJREFRQUJBb0lCQVFDR3ZQY29NUHZNNFNYNQo0dm9seDdLdXhCazNqMmxKRER2dyt2VjF3a0szekxxQTNNeUdoaTB2Mm9qL09MT3hqcWJWenRyQ0NvbE0zY2N2CkVXVVQ3RFJJaUdidHpWWC95a1lKcGx5aG9PRURENyt5dk9xeUFYUy9UMzZzYWlscFczQzA3SGFjTkIweE1pUnMKdFV6Wlk0R0o4cndzYkVVek9QQlhPZHBSZFBjWmp1OEF3ZDBkS2oxQmwvMGVCL2RxZCsrR3o0ZEFNR1h0b1V6WApSVGtBSWtLWGJEUHVON1oyck5BMzVmWnhBOE44b0NzalYyODRlUnhiUURQejVUQXB1RExEdHZqYk8xTVZvdVpECjZyZUtmSStnbDlIUnZ6YlMxekhXWjVSZ1F2RUFIV0pKcHBBMGF2NHdEU0NKZjhLNnl2R3NvbjJ0UzAveUErY3YKVVMxWEJIckJBb0dCQU1nNW5IM2twclBUUFRkcTRVQmhRWndLWG50Nm5kL2huRmQ2d1h1VVY5UmRrU1ZXbmxkcQpMOVcwVDU5ODFMeXdEb1ZXL2swUTkvelFMRG9ONTZDOGpTTnBkZHZtQlBHY2tkNU1JN1RtTEwvTVRtNWRLOGZHClZGNnorMVRvMm9BcW83czFFSWhseXlmckZPTVI0NThsSENWVEdPNlI3Y2MzQy8rVU01MVNQY0tMQW9HQkFPZDgKNFNwWFg0TjJHZmRZQTRyKzdqNHFDWXdLYVpyWjlmckxZQXI2cnBja2xScDRZL0M3b0Nja2VTM3BNMDB6SVlQQgpMQis0YVRJeXdBaVNxY3I1YVhnM0tZTkVKNFFpaEpiRjNqV1I2cUdtc3kxMzBCWVpHQVg1VDRMdHVJc0ZIV2NvCkdjNDJQSXhXREtxdzJzYUZBZTBFa2lkb2dsTit5eGt6WWJMQWVWaDVBb0dCQUxyaTQwbXl5Vktod2NyZkxQNTkKaU5MUDd0Nk1WWjJwcE5jV1VsQTU1enptVk5zb0hVVjBiTStvcklVdDdCZHVzUzhPUXZERi9PSnhvRVpUd2phSwpwNlk5QW5CTkk2SXRSUTNidlp4VkY4R3lQaWJQT2xVT3JxTnlsUTN0QmoySkR5aG00RmFmeE44dWttRmJ5ajA2ClV5b1hoUGJ4S0tMQW82ZGJ1azJHZlBUL0FvR0FZYlFPb1QxaGZlNENCYWlyVGlaTlRnV1dJL3BkR2xPMmc1VUYKUTMwTTVaUTJMb2J6djY2aGFRUDI5WTdBN1d1UVVMamVzOEMwL2MvM3gyYUhyYmpaY0Rqd0Y0eFRsV3l3UTZiZQpKQVFqWVBrb0ZSL0Z2eDMyU3Njd2JSV0MxNEpnSjZNQVNVNFEvalp2Z0RmSER4VWllL1I2NzVFbnVMQUNidStGCjQ5bGpIaGtDZ1lBUFA2RGM1SCtyMC8vVWUrSlNVTzY1N2pIdW5WYU8rZE81bVJZTzZqR1NHc2JpWnZ3RmZjZEIKZXZBTFJQQ0NOS3F4U1dGU2RXc3p0WUNtZ3BHSVFpaXowMncvSk54bERHTVd1T002WnlsODl2WmdPYk1SWnBWWQpzZ0pMNGZJMFFLZmdhMkZoYUdML1Z4L2c0RE91Vy8zWjZ6NEI3YzYySXE1OVFhaEZURERERlE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=

View File

@ -1 +0,0 @@
ipa-94b1c71c3c7d0eb739170278f537225fb347530ecc2a3282fb9224dbe334b84a.qcow2

View File

@ -1 +0,0 @@
openshift-c65dc46c4885d5b7cd1198d310cb0a62a515e0cf67aba4a0ba55caf2b7fddd39.qcow2

View File

@ -1 +0,0 @@
ovirt-f033c4457fecb1e9078eb16d7ac5239fe79455ca6b533f2a37de4f965cf174e7.qcow2

View File

@ -1 +0,0 @@
rhel-7-7-9d4481970308e14f9af75151118b6ffc9bf0fb2c1c9bda2f6c19d70b3cc612d4.qcow2

View File

@ -1 +0,0 @@
rhel-7-8-90e79cf4377714b69929e8613375987c0d7b6a8e5deff38bbfa9662ff583d5e0.qcow2

View File

@ -1 +0,0 @@
rhel-8-0-a7af23080d6f6d5595b5fbc5331a2f7148069223659701c1870c844babc56833.qcow2

View File

@ -1 +0,0 @@
rhel-8-1-25b42538c7798dda35e910167a9de2af924ff28d81e8612bb5318469c2962c06.qcow2

View File

@ -1 +0,0 @@
rhel-atomic-a1388f58b093cb83a15f4260f2fc3bbb42b85aae102e5f4e467f2635faeca4c9.qcow2

View File

@ -1,78 +0,0 @@
#! /bin/bash
# This file is part of Cockpit.
#
# Copyright (C) 2015 Red Hat, Inc.
#
# Cockpit is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Cockpit is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
set -ex
out="$1"
base="$2"
redirect_base=$(curl -s -w "%{redirect_url}" "$base" -o /dev/null)
if [ -n "$redirect_base" ]; then
base="$redirect_base"
fi
# Lookup the newest base image recursively
url="$base"
while [ $# -gt 2 ]; do
fragment="$3"
if [ "$fragment" = "sort" ]; then
backref="$4"
pattern="$5"
result="`wget -q -O- $url | grep -oE "$pattern" | sed -E "s/${pattern}/\\\\${backref} \\0/" | sort -V -k1 | tail -1`"
fragment="`echo $result | cut -f2 -d' '`"
if [ -z "$fragment" ]; then
echo "Could not find '$pattern' at: $url" >&2
exit 1
fi
shift; shift
fi
base="$url"
url="$base/$fragment"
shift
done
# we link to the file so wget can properly detect if we have already downloaded it
# note that due to mirroring, timestamp comparison can result in unnecessary downloading
out_base="`dirname $out`"
intermediate="$out_base/$fragment"
if [ "$intermediate" != "$out" ]; then
wget --no-clobber --directory-prefix="$out_base" "$base/$fragment"
cp "$intermediate" "$out"
else
rm -f "$out"
wget --directory-prefix="$out_base" "$base/$fragment"
fi
# Make the image be at least 12 Gig. During boot, docker-storage-setup
# will grow the partitions etc as appropriate, and atomic.setup will
# explicitly grow the docker pool.
vsize=$(qemu-img info "$out" --output=json | python3 -c 'import json, sys; print(json.load(sys.stdin)["virtual-size"])')
if [ "$vsize" -lt 12884901888 ]; then
qemu-img resize "$out" 12884901888
fi

View File

@ -1 +0,0 @@
centos-7.bootstrap

View File

@ -1,65 +0,0 @@
#!/bin/bash
set -ex
YUM_INSTALL="yum --setopt=skip_missing_names_on_install=False -y install"
# We deploy candlepin via ansible
$YUM_INSTALL epel-release
# Install dependencies
CANDLEPIN_DEPS="\
ansible \
git \
openssl \
"
$YUM_INSTALL $CANDLEPIN_DEPS
mkdir -p playbookdir; cd playbookdir;
mkdir -p roles
git clone https://github.com/candlepin/ansible-role-candlepin.git roles/candlepin
# Run the playbook
cat > inventory <<- EOF
[dev]
localhost
EOF
useradd -m admin
echo admin:foobar | chpasswd
echo 'admin ALL=(ALL) NOPASSWD: ALL' > /etc/sudoers.d/admin
cat > playbook.yml <<- EOF
- hosts: dev
environment:
JAVA_HOME: /usr/lib/jvm/java-1.8.0/
roles:
- role: candlepin
candlepin_git_pull: True
candlepin_deploy_args: "-g -a -f -t"
candlepin_user: admin
candlepin_user_home: /home/admin
candlepin_checkout: /home/admin/candlepin
EOF
ansible-playbook -i inventory -c local -v --skip-tags 'system_update' playbook.yml
rm -rf playbookdir
# reduce image size
yum clean all
/var/lib/testvm/zero-disk.setup
# Final tweaks
rm -rf /var/log/journal/*
echo "kernel.core_pattern=|/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e" > /etc/sysctl.d/50-coredump.conf
# Audit events to the journal
rm -f '/etc/systemd/system/multi-user.target.wants/auditd.service'
rm -rf /var/log/audit/

View File

@ -1,4 +0,0 @@
#! /bin/bash
BASE=$(dirname $0)
$BASE/virt-install-fedora "$1" x86_64 "http://mirror.centos.org/centos/7/os/x86_64/"

View File

@ -1,8 +0,0 @@
#! /bin/bash
set -e
# remove cockpit distro packages, testing with upstream master
rpm --erase --verbose cockpit cockpit-ws cockpit-bridge cockpit-system
/var/lib/testvm/fedora.install "$@"

View File

@ -1 +0,0 @@
rhel.setup

View File

@ -1,28 +0,0 @@
#!/bin/sh
set -eux
OUTPUT="$1"
curl https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-i386-disk.img > "$OUTPUT"
# prepare a cloud-init iso for disabling network source, to avoid a 90s timeout at boot
WORKDIR=$(mktemp -d)
trap "rm -rf '$WORKDIR'" EXIT INT QUIT PIPE
cd "$WORKDIR"
cat > meta-data <<EOF
{ "instance-id": "nocloud" }
EOF
cat > user-data <<EOF
#!/bin/sh
set -ex
sed -i 's/configdrive *//; s/ec2 *//' /etc/cirros-init/config
(sleep 1; poweroff) &
EOF
genisoimage -input-charset utf-8 -output cloud-init.iso -volid cidata -joliet -rock user-data meta-data
# boot it once with the cloud-init ISO
qemu-system-x86_64 -enable-kvm -nographic -net none \
-drive file="$OUTPUT",if=virtio -cdrom cloud-init.iso

View File

@ -1,9 +0,0 @@
#! /bin/bash
set -e
url="https://cloud.centos.org/centos/7/atomic/images"
prefix="CentOS-Atomic-Host-GenericCloud.qcow2"
BASE=$(dirname $0)
$BASE/atomic.bootstrap "$1" "$url" "$prefix"

View File

@ -1,5 +0,0 @@
#! /bin/bash
set -e
/var/lib/testvm/atomic.install --skip cockpit-sosreport --extra "/root/rpms/libssh*" --extra "/var/tmp/build-results/cockpit-dashboard*" "$@"

View File

@ -1,72 +0,0 @@
#!/bin/bash
# This file is part of Cockpit.
#
# Copyright (C) 2016 Red Hat, Inc.
#
# Cockpit is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Cockpit is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
set -ex
# The docker pool should grow automatically as needed, but we grow it
# explicitly here anyway. This is hopefully more reliable.
# HACK: docker falls over regularly, print its log if it does
systemctl start docker || journalctl -u docker
lvresize atomicos/root -l+50%FREE -r
if lvs atomicos/docker-pool 2>/dev/null; then
lvresize atomicos/docker-pool -l+100%FREE
elif lvs atomicos/docker-root-lv; then
lvresize atomicos/docker-root-lv -l+100%FREE
fi
# Get the centos cockpit/ws image
docker pull registry.centos.org/cockpit/ws:latest
docker tag registry.centos.org/cockpit/ws cockpit/ws
# docker images that we need for integration testing
/var/lib/testvm/docker-images.setup
# Configure core dumps
echo "kernel.core_pattern=|/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e" > /etc/sysctl.d/50-coredump.conf
# Download the libssh RPM plus dependencies which we'll use for
# package overlay. The only way to do this is via a container
. /etc/os-release
REPO="updates"
if [ "$ID" = "rhel" ]; then
subscription-manager repos --enable rhel-7-server-extras-rpms
REPO="rhel-7-server-extras-rpms"
ID="rhel7"
fi
docker run --rm --volume=/etc/yum.repos.d:/etc/yum.repos.d:z --volume=/root/rpms:/tmp/rpms:rw,z "$ID:$VERSION_ID" /bin/sh -cex "yum install -y findutils createrepo_c && yum install -y --downloadonly --enablerepo=$REPO libssh && find /var -name '*.rpm' | while read rpm; do mv -v \$rpm /tmp/rpms; done; createrepo_c /tmp/rpms"
rm -f /etc/yum.repos.d/*
cat >/etc/yum.repos.d/deps.repo <<EOF
[deps]
baseurl=file:///root/rpms
enabled=1
EOF
# Switch to continuous stream
ostree remote add --set=gpg-verify=false centos-atomic-continuous https://ci.centos.org/artifacts/sig-atomic/rdgo/centos-continuous/ostree/repo/
rpm-ostree rebase centos-atomic-continuous:centos-atomic-host/7/x86_64/devel/continuous
ostree checkout centos-atomic-continuous:centos-atomic-host/7/x86_64/devel/continuous /var/local-tree
# reduce image size
/var/lib/testvm/zero-disk.setup
# Prevent SSH from hanging for a long time when no external network access
echo 'UseDNS no' >> /etc/ssh/sshd_config
# Final tweaks
rm -rf /var/log/journal/*

View File

@ -1,7 +0,0 @@
#! /bin/sh -ex
RELEASE=buster
RELEASENUM=10
LATEST_DAILY=$(curl -s https://cloud.debian.org/images/cloud/$RELEASE/daily/ | sed -n '/<a href=/ { s/^.*href="//; s_/".*$__; p }'| sort -nu | tail -n1)
exec $(dirname $0)/lib/debian.bootstrap "$1" "https://cloud.debian.org/images/cloud/buster/daily/$LATEST_DAILY/debian-${RELEASENUM}-genericcloud-amd64-daily-${LATEST_DAILY}.qcow2"

View File

@ -1,8 +0,0 @@
#! /bin/bash
set -e
/var/lib/testvm/debian.install "$@"
# HACK: https://bugs.debian.org/914694
sed -i '/IndividualCalls/ s/=no/=yes/' /etc/firewalld/firewalld.conf

View File

@ -1 +0,0 @@
debian.setup

View File

@ -1 +0,0 @@
debian-stable.bootstrap

View File

@ -1,8 +0,0 @@
#! /bin/bash
set -e
/var/lib/testvm/debian.install "$@"
# HACK: https://bugs.debian.org/914694
sed -i '/IndividualCalls/ s/=no/=yes/' /etc/firewalld/firewalld.conf

View File

@ -1 +0,0 @@
debian.setup

View File

@ -1,222 +0,0 @@
#! /bin/bash
# Shared .setup between all Debian/Ubuntu flavors
set -ex
# Enable a console on ttyS0 so that we can log-in via vm-run.
# and make the boot up more verbose
sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT/# GRUB_CMDLINE_LINUX_DEFAULT/' /etc/default/grub
# We install all dependencies of the cockpit packages since we want
# them to not spontaneously change from one test run to the next when
# the distribution repository is updated.
#
COCKPIT_DEPS="\
cryptsetup \
docker.io \
libblockdev-mdraid2 \
libjson-glib-1.0-0 \
libpcp3 \
libpolkit-agent-1-0 \
libpolkit-gobject-1-0 \
libpwquality-tools \
libssh-4 \
libteam-utils \
libvirt-daemon-system \
libvirt-dbus \
libosinfo-bin \
network-manager \
pcp \
policykit-1 \
python3-dbus \
qemu-block-extra \
realmd \
selinux-basics \
thin-provisioning-tools \
unattended-upgrades \
tuned \
xdg-utils \
udisks2 \
udisks2-lvm2 \
"
# We also install the packages necessary to join a FreeIPA domain so
# that we don't have to go to the network during a test run.
IPA_CLIENT_PACKAGES="\
freeipa-client \
sssd-tools \
sssd-dbus \
packagekit \
"
TEST_PACKAGES="\
acl \
curl \
firewalld \
gdb \
iproute2 \
mdadm \
nfs-server \
qemu-kvm \
socat \
systemd-coredump \
virtinst \
xfsprogs \
sosreport \
"
RELEASE=$(grep -m1 ^deb /etc/apt/sources.list | awk '{print $3}')
case "$RELEASE" in
bionic)
# these packages are not in Ubuntu 18.04
COCKPIT_DEPS="${COCKPIT_DEPS/libvirt-dbus /}"
;;
esac
if [ "${1#ubuntu}" != "$1" ]; then
# our tests need scsi_debug, which the cloud kernel does not have; install full kernel
TEST_PACKAGES="$TEST_PACKAGES linux-image-generic"
fi
# our cloud-init.iso does not set up the host name
echo "127.0.1.1 $(hostname)" >> /etc/hosts
if grep -q 'ID=ubuntu' /etc/os-release; then
PBUILDER_OPTS='COMPONENTS="main universe"'
# We want to use/test NetworkManager instead of netplan/networkd for ethernets
mkdir -p /etc/NetworkManager/conf.d
touch /etc/NetworkManager/conf.d/10-globally-managed-devices.conf
fi
# some cloud images have a pre-defined admin user or group, for them cloud-init admin creation fails
userdel -r admin || true
groupdel admin || true
useradd -m -U -c Administrator -G sudo -s /bin/bash admin
echo admin:foobar | chpasswd
cp -r ~root/.ssh ~admin/
chown -R admin:admin ~admin/.ssh
# avoid NM-wait-online hanging on disconnected interfaces
mkdir -p /etc/NetworkManager/conf.d/
printf '[main]\nno-auto-default=*\n' > /etc/NetworkManager/conf.d/noauto.conf
if [ "${1#debian}" != "$1" ]; then
# HACK: Debian's cloud-init generates a *.cfg file, but /etc/network/interfaces sources extension-less files
mv /etc/network/interfaces.d/50-cloud-init.cfg /etc/network/interfaces.d/50-cloud-init
fi
# debian-testing image gets bootstrapped from debian stable; upgrade
if [ "$1" = "debian-testing" ]; then
rm --verbose -f /etc/apt/sources.list.d/*
echo 'deb http://deb.debian.org/debian testing main' > /etc/apt/sources.list
fi
export DEBIAN_FRONTEND=noninteractive
apt-get -y update
# apt go-faster
echo 'Acquire::Languages "none";' > /etc/apt/apt.conf.d/90nolanguages
apt-get install -y eatmydata
# remove packages that we don't need
for p in lxd snapd landscape-common accountsservice open-vm-tools ufw cloud-init; do eatmydata apt-get purge -y --auto-remove $p || true; done
# HACK: work around fuse 2.9.9-1 install failure (https://bugs.debian.org/935496)
if [ "$1" = "debian-testing" ]; then
rm /dev/fuse
# this needs to happen right away, as upgrading other packages triggers udev events which recreate /dev/fuse
eatmydata apt-get install -y fuse
fi
# HACK: debian-stable image got /usr/bin/qemu-img removed, even though qemu-utils package is installed
if [ "$1" = "debian-stable" ]; then
eatmydata apt-get install --reinstall -y qemu-utils
fi
# install our dependencies
DEBIAN_FRONTEND=noninteractive eatmydata apt-get -y dist-upgrade
eatmydata apt-get -y install $TEST_PACKAGES $COCKPIT_DEPS $IPA_CLIENT_PACKAGES
# Prepare for building
#
# extract control files and adjust them for our release, so that we can parse the build deps
mkdir -p /tmp/out
curl -L https://github.com/cockpit-project/cockpit/archive/master.tar.gz | tar -C /tmp/out --strip-components=1 --wildcards -zxf - '*/debian/'
/tmp/out/tools/debian/adjust-for-release $(lsb_release -sc)
# Disable build-dep installation for the real builds
cat > ~/.pbuilderrc <<- EOF
DISTRIBUTION=$RELEASE
PBUILDERSATISFYDEPENDSCMD=true
$PBUILDER_OPTS
EOF
eatmydata apt-get -y install dpkg-dev pbuilder
pbuilder --create --extrapackages "fakeroot $PBUILDER_EXTRA"
/usr/lib/pbuilder/pbuilder-satisfydepends-classic --control /tmp/out/tools/debian/control --force-version --echo|grep apt-get | pbuilder --login --save-after-login
rm -rf /tmp/out
# Debian does not automatically start the default libvirt network
virsh net-autostart default
# Don't automatically update on boot or daily
systemctl disable apt-daily.service apt-daily.timer || true
# Enable coredumping via systemd
echo "kernel.core_pattern=|/lib/systemd/systemd-coredump %P %u %g %s %t %c %e" > /etc/sysctl.d/50-coredump.conf
printf 'DefaultLimitCORE=infinity\n' >> /etc/systemd/system.conf
# HACK: we need to restart it in case aufs-dkms was installed after docker.io
# and thus docker.io auto-switches its backend
systemctl restart docker || journalctl -u docker
I=$(docker info)
if ! echo "$I" | grep -Eq 'Storage.*(aufs|overlay)'; then
echo "ERROR! docker does not use aufs or overlayfs"
exit 1
fi
# docker images that we need for integration testing
/var/lib/testvm/docker-images.setup
rm -rf /var/lib/docker/devicemapper
# in case there are unnecessary packages
eatmydata apt-get -y autoremove || true
# disable udev network names, our tests expect the kernel schema
sed -i '/GRUB_CMDLINE_LINUX=/ s/"$/ net.ifnames=0 biosdevname=0"/' /etc/default/grub
rm -f /etc/udev/rules.d/70-persistent-net.rules /etc/udev/rules.d/75-cloud-ifupdown.rules
update-grub
sed -i 's/ens[^[:space:]:]*/eth0/' /etc/network/interfaces /etc/network/interfaces.d/* /etc/netplan/*.yaml || true
update-initramfs -u
# reduce image size
apt-get clean
pbuilder clean
rm -f /var/cache/apt/*cache.bin
/var/lib/testvm/zero-disk.setup
# Final tweaks
# Enable persistent journal
mkdir -p /var/log/journal
# Allow root login with password
sed -i 's/^[# ]*PermitRootLogin .*/PermitRootLogin yes/' /etc/ssh/sshd_config
# At least debian-9 virt-install image only has RSA key
[ -e /etc/ssh/ssh_host_ed25519_key ] || ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -N '' -t ed25519
[ -e /etc/ssh/ssh_host_ecdsa_key ] || ssh-keygen -f /etc/ssh/ssh_host_ecdsa_key -N '' -t ecdsa
# Prevent SSH from hanging for a long time when no external network access
echo 'UseDNS no' >> /etc/ssh/sshd_config
# HACK: https://bugzilla.mindrot.org/show_bug.cgi?id=2512
# Disable the restarting of sshd when networking changes
ln -snf /bin/true /etc/network/if-up.d/openssh-server
# Stop showing 'To run a command as administrator (user "root"), use "sudo <command>". See "man
# sudo_root" for details.` message in admins terminal.
touch /home/admin/.sudo_as_admin_successful

View File

@ -1,21 +0,0 @@
#!/bin/bash
#
# Copyright (C) 2015 Red Hat Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
# 02110-1301 USA.
BASE=$(dirname $0)
$BASE/virt-install-fedora "$1" x86_64 "https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/23/Server/x86_64/os/"

View File

@ -1,11 +0,0 @@
#! /bin/bash
useradd -c Administrator -G wheel admin
echo foobar | passwd --stdin admin
dnf -y update
dnf -y install fedora-release-server
firewall-cmd --permanent --add-service cockpit
# Phantom can't use TLS..
sed -i -e 's/ExecStart=.*/\0 --no-tls/' /usr/lib/systemd/system/cockpit.service

View File

@ -1,21 +0,0 @@
#!/bin/bash
#
# Copyright (C) 2018 Red Hat Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
# 02110-1301 USA.
BASE=$(dirname $0)
$BASE/virt-install-fedora "$1" x86_64 "http://dl.fedoraproject.org/pub/fedora/linux/releases/29/Server/x86_64/os/"

View File

@ -1,4 +0,0 @@
#! /bin/bash
set -e
/var/lib/testvm/fedora.install "$@"

View File

@ -1 +0,0 @@
fedora.setup

View File

@ -1,21 +0,0 @@
#!/bin/bash
#
# Copyright (C) 2019 Red Hat Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
# 02110-1301 USA.
BASE=$(dirname $0)
$BASE/virt-install-fedora "$1" x86_64 "http://dl.fedoraproject.org/pub/fedora/linux/releases/30/Server/x86_64/os/"

View File

@ -1,4 +0,0 @@
#! /bin/bash
set -e
/var/lib/testvm/fedora.install "$@"

View File

@ -1 +0,0 @@
fedora.setup

View File

@ -1,23 +0,0 @@
#!/bin/bash
#
# Copyright (C) 2019 Red Hat Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
# 02110-1301 USA.
BASE=$(dirname $0)
# once fedora 31 is released, replace url:
# http://dl.fedoraproject.org/pub/fedora/linux/releases/31/Server/x86_64/os/
$BASE/virt-install-fedora "$1" x86_64 "https://dl.fedoraproject.org/pub/fedora/linux/development/31/Everything/x86_64/os/"

View File

@ -1,4 +0,0 @@
#! /bin/bash
set -e
/var/lib/testvm/fedora.install "$@"

View File

@ -1 +0,0 @@
fedora.setup

View File

@ -1,14 +0,0 @@
#! /bin/bash
set -e
url="https://download.fedoraproject.org/pub/alt/atomic/stable/"
BASE=$(dirname $0)
# The Fedora URLs have the version twice in the name. for example:
# https://dl.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-28-20180425.0/AtomicHost/x86_64/images/Fedora-AtomicHost-28-20180425.0.x86_64.qcow2
$BASE/atomic.bootstrap "$1" "$url" \
sort 3 "Fedora(-atomic)?-[0-9][0-9](-updates)?-([-0-9\.]+)" \
"AtomicHost" "x86_64" "images" \
sort 1 "Fedora-AtomicHost-([-0-9\.]+).x86_64.qcow2"

View File

@ -1,9 +0,0 @@
#! /bin/bash
set -e
/var/lib/testvm/atomic.install --verbose --skip cockpit-kdump --extra "/root/rpms/libssh*" "$@"
# HACK: https://github.com/projectatomic/rpm-ostree/issues/1360
# rpm-ostree upgrade --check otherwise fails
mkdir -p /var/cache/rpm-ostree

View File

@ -1,18 +0,0 @@
#!/bin/bash
set -ex
# HACK: https://bugzilla.redhat.com/show_bug.cgi?id=1341829
# SELinux breaks coredumping on fedora-25
printf '(allow init_t domain (process (rlimitinh)))\n' > domain.cil
semodule -i domain.cil
# HACK: docker falls over regularly, print its log if it does
systemctl start docker || journalctl -u docker
os=$(ls /ostree/repo/refs/remotes/fedora-atomic/*/)
docker pull "registry.fedoraproject.org/f$os/cockpit"
docker tag "registry.fedoraproject.org/f$os/cockpit" cockpit/ws
/var/lib/testvm/atomic.setup

View File

@ -1,21 +0,0 @@
#!/bin/bash
#
# Copyright (C) 2019 Red Hat Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
# 02110-1301 USA.
BASE=$(dirname $0)
$BASE/virt-install-fedora "$1" i386 "https://dl.fedoraproject.org/pub/fedora-secondary/releases/30/Server/i386/os/"

View File

@ -1 +0,0 @@
fedora-30.install

View File

@ -1 +0,0 @@
fedora.setup

View File

@ -1,11 +0,0 @@
#! /bin/bash
useradd -c Administrator -G wheel admin
echo foobar | passwd --stdin admin
dnf -y update
dnf -y install fedora-release-server
firewall-cmd --permanent --add-service cockpit
# Phantom can't use TLS..
sed -i -e 's/ExecStart=.*/\0 --no-tls/' /usr/lib/systemd/system/cockpit.service

View File

@ -1 +0,0 @@
fedora-30.bootstrap

View File

@ -1 +0,0 @@
fedora-30.install

View File

@ -1 +0,0 @@
fedora.setup

View File

@ -1,201 +0,0 @@
#!/bin/bash
set -ex
IMAGE="$1"
# avoid failures when running image builds in a non-English locale (ssh transfers the host environment)
unset LANGUAGE
unset LANG
export LC_ALL=C.utf8
# keep this in sync with avocado/selenium image mapping in bots/tests-invoke
if [ "$IMAGE" = fedora-30 ]; then
AVOCADO=1
fi
# HACK - virt-resize might not be able to resize our xfs rootfs,
# depending on how it was compiled and which plugins are installed,
# and will just silently not do it. So we do it here.
#
if [ "$IMAGE" != fedora-31 ]; then
xfs_growfs /
fi
df -h /
echo foobar | passwd --stdin root
HAVE_KUBERNETES=
if [ $(uname -m) = x86_64 ]; then
HAVE_KUBERNETES=1
fi
# HACK docker not available on f31
# https://github.com/cockpit-project/cockpit/issues/12670
HAVE_DOCKER=
if [ "$1" != fedora-31 ]; then
HAVE_DOCKER=1
fi
# We install all dependencies of the cockpit packages since we want
# them to not spontaneously change from one test run to the next when
# the distribution repository is updated.
#
COCKPIT_DEPS="\
device-mapper-multipath \
etcd \
glibc-all-langpacks \
glib-networking \
grubby \
json-glib \
kexec-tools \
libssh \
libvirt-daemon-kvm \
libvirt-client \
libvirt-dbus \
NetworkManager-team \
openssl \
PackageKit \
pcp \
pcp-libs \
qemu \
realmd \
selinux-policy-targeted \
setroubleshoot-server \
sos \
sscg \
system-logos \
subscription-manager \
tuned \
virt-install \
"
[ -z "$HAVE_DOCKER" ] || COCKPIT_DEPS="$COCKPIT_DEPS atomic docker"
COCKPIT_DEPS="$COCKPIT_DEPS udisks2 udisks2-lvm2 udisks2-iscsi"
[ -z "$HAVE_KUBERNETES" ] || COCKPIT_DEPS="$COCKPIT_DEPS kubernetes"
# We also install the packages necessary to join a FreeIPA domain so
# that we don't have to go to the network during a test run.
#
IPA_CLIENT_PACKAGES="\
freeipa-client \
oddjob \
oddjob-mkhomedir \
sssd \
sssd-dbus \
libsss_sudo \
"
TEST_PACKAGES="\
systemtap-runtime-virtguest \
valgrind \
gdb \
targetcli \
dnf-automatic \
cryptsetup \
clevis-luks \
socat \
tang \
podman \
ntp \
libvirt-daemon-config-network \
"
# HACK - For correct work of ABRT in Fedora 26 Alpha release a following
# packages are necessary. In Fedora 26 Beta and later these packages should be
# installed by default. See https://bugzilla.redhat.com/show_bug.cgi?id=1436941
#
ABRT_PACKAGES="\
abrt-desktop \
libreport-plugin-systemd-journal \
"
rm -rf /etc/sysconfig/iptables
maybe() { if type "$1" >/dev/null 2>&1; then "$@"; fi; }
# For the D-Bus test server
maybe firewall-cmd --permanent --add-port 8765/tcp
echo 'NETWORKING=yes' > /etc/sysconfig/network
useradd -c Administrator -G wheel admin
echo foobar | passwd --stdin admin
if [ "${IMAGE%-i386}" != "$IMAGE" ]; then
TEST_PACKAGES="${TEST_PACKAGES/podman /}"
fi
if [ "${IMAGE%-testing}" != "$IMAGE" ]; then
dnf config-manager --set-enabled updates-testing
fi
dnf $DNF_OPTS -y upgrade
dnf $DNF_OPTS -y install $TEST_PACKAGES $COCKPIT_DEPS $IPA_CLIENT_PACKAGES $ABRT_PACKAGES
if [ -n "$AVOCADO" ]; then
# enable python3 avocado support repository
dnf module install -y avocado:69lts
dnf $DNF_OPTS -y install \
fontconfig \
npm \
chromium-headless \
python3-libvirt \
python3-avocado \
python3-avocado-plugins-output-html \
python3-selenium
npm -g install chrome-remote-interface
echo 'NODE_PATH=/usr/lib/node_modules' >> /etc/environment
fi
dnf $DNF_OPTS -y install mock dnf-plugins-core rpm-build
useradd -c Builder -G mock builder
if [ "${IMAGE%-testing}" != "$IMAGE" ]; then
# Enable updates-testing in mock
echo "config_opts['yum.conf'] += '[updates-testing]\nenabled=1'" >>/etc/mock/default.cfg
fi
# HACK - mock --installdeps is broken, it seems that it forgets to
# copy the source rpm to a location that dnf can actually access. A
# workaround is to pass "--no-bootstrap-chroot".
#
# When you remove this hack, also remove it in fedora-*.install.
#
# https://bugzilla.redhat.com/show_bug.cgi?id=1447627
opsys=$(cut -d '-' -f 1 <<< "$IMAGE")
version=$(cut -d '-' -f 2 <<< "$IMAGE")
# If version is not number (testing/i386) then use Fedora 30
if ! [ "$version" -eq "$version" ] 2>/dev/null; then version=30; fi
su builder -c "/usr/bin/mock --no-bootstrap-chroot --verbose -i $(/var/lib/testvm/build-deps.sh "$opsys $version")"
su builder -c "/usr/bin/mock --install --verbose rpmlint"
if [ -n "$HAVE_DOCKER" ]; then
# HACK: docker falls over regularly, print its log if it does
systemctl start docker || journalctl -u docker
# docker images that we need for integration testing
/var/lib/testvm/docker-images.setup
fi
# Configure kubernetes
[ -z "$HAVE_KUBERNETES" ] || /var/lib/testvm/kubernetes.setup
# reduce image size
dnf clean all
/var/lib/testvm/zero-disk.setup --keep-mock-cache
ln -sf ../selinux/config /etc/sysconfig/selinux
printf "SELINUX=enforcing\nSELINUXTYPE=targeted\n" > /etc/selinux/config
# Prevent SSH from hanging for a long time when no external network access
echo 'UseDNS no' >> /etc/ssh/sshd_config
# Audit events to the journal
rm -f '/etc/systemd/system/multi-user.target.wants/auditd.service'
rm -rf /var/log/audit/

View File

@ -1 +0,0 @@
fedora-29.bootstrap

View File

@ -1,49 +0,0 @@
#!/bin/bash
set -eufx
# ipa requires an UTF-8 locale
export LC_ALL=C.UTF-8
echo foobar | passwd --stdin root
dnf -y remove firewalld
dnf -y update
dnf -y install freeipa-server freeipa-server-dns bind bind-dyndb-ldap iptables
iptables -F
nmcli con add con-name "static-eth1" ifname eth1 type ethernet ip4 "10.111.112.100/20" ipv4.dns "10.111.112.100" gw4 "10.111.112.1"
nmcli con up "static-eth1"
hostnamectl set-hostname f0.cockpit.lan
# Let's make sure that ipa-server-install doesn't block on
# /dev/random.
#
rm -f /dev/random
ln -s /dev/urandom /dev/random
ipa-server-install -U -p foobarfoo -a foobarfoo -n cockpit.lan -r COCKPIT.LAN --setup-dns --no-forwarders
# Make sure any initial password change is overridden
printf 'foobarfoo\nfoobarfoo\nfoobarfoo\n' | kinit admin@COCKPIT.LAN
# Default password expiry of 90 days is impractical
ipa pwpolicy-mod --minlife=0 --maxlife=1000
# Change password to apply new password policy
printf 'foobarfoo\nfoobarfoo\n' | ipa user-mod --password admin
ipa user-show --all admin
# Allow "admins" IPA group members to run sudo
# This is an "unbreak my setup" step and ought to happen by default.
# See https://pagure.io/freeipa/issue/7538
ipa-advise enable-admins-sudo | sh -ex
ipa dnsconfig-mod --forwarder=8.8.8.8
ln -sf ../selinux/config /etc/sysconfig/selinux
echo 'SELINUX=permissive' > /etc/selinux/config
# reduce image size
dnf clean all
/var/lib/testvm/zero-disk.setup

View File

@ -1,303 +0,0 @@
#!/usr/bin/python2
# This file is part of Cockpit.
#
# Copyright (C) 2015 Red Hat, Inc.
#
# Cockpit is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Cockpit is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
import subprocess
import os
import sys
import shutil
try:
from urllib.request import URLopener
except ImportError:
from urllib import URLopener # Python 2
import argparse
import json
BASEDIR = os.path.dirname(__file__)
class AtomicCockpitInstaller:
branch = None
checkout_location = "/var/local-tree"
repo_location = "/var/local-repo"
rpm_location = "/usr/share/rpm"
key_id = "95A8BA1754D0E95E2B3A98A7EE15015654780CBD"
port = 12345
# Support installing random packages if needed.
external_packages = {}
# Temporarily force cockpit-system instead of cockpit-shell
packages_force_install = [ "cockpit-system",
"cockpit-docker",
"cockpit-kdump",
"cockpit-networkmanager",
"cockpit-sosreport" ]
def __init__(self, rpms=None, extra_rpms=None, verbose=False):
self.verbose = verbose
self.rpms = rpms
self.extra_rpms = extra_rpms
status = json.loads(subprocess.check_output(["rpm-ostree", "status", "--json"], universal_newlines=True))
origin = None
for deployment in status.get("deployments", []):
if deployment.get("booted"):
origin = deployment["origin"]
if not origin:
raise Exception("Couldn't find origin")
self.branch = origin.split(":", 1)[-1]
def setup_dirs(self):
if self.verbose:
print("setting up new ostree repo")
try:
shutil.rmtree(self.repo_location)
except:
pass
os.makedirs(self.repo_location)
subprocess.check_call(["ostree", "init", "--repo", self.repo_location,
"--mode", "archive-z2"])
if not os.path.exists(self.checkout_location):
if self.verbose:
print("cloning current branch")
subprocess.check_call(["ostree", "checkout", self.branch,
self.checkout_location])
# move /usr/etc to /etc, makes rpm installs easier
subprocess.check_call(["mv", os.path.join(self.checkout_location, "usr", "etc"),
os.path.join(self.checkout_location, "etc")])
def switch_to_local_tree(self):
if self.verbose:
print("install new ostree commit")
# Not an error if this fails
subprocess.call(["ostree", "remote", "delete", "local"])
subprocess.check_call(["ostree", "remote", "add", "local",
"file://{}".format(self.repo_location),
"--no-gpg-verify"])
# HACK: https://github.com/candlepin/subscription-manager/issues/1404
subprocess.call(["systemctl", "disable", "rhsmcertd"])
subprocess.call(["systemctl", "stop", "rhsmcertd"])
status = subprocess.check_output(["rpm-ostree", "status"])
if b"local:" in status:
subprocess.check_call(["rpm-ostree", "upgrade"])
else:
try:
subprocess.check_call(["setenforce", "0"])
subprocess.check_call(["rpm-ostree", "rebase",
"local:{0}".format(self.branch)])
except:
os.system("sysctl kernel.core_pattern")
os.system("coredumpctl || true")
raise
finally:
subprocess.check_call(["setenforce", "1"])
def commit_to_repo(self):
if self.verbose:
print("commit package changes to our repo")
# move etc back to /usr/etc
subprocess.check_call(["mv", os.path.join(self.checkout_location, "etc"),
os.path.join(self.checkout_location, "usr", "etc")])
subprocess.check_call(["ostree", "commit", "-s", "cockpit-tree",
"--repo", self.repo_location,
"-b", self.branch,
"--add-metadata-string", "version=cockpit-base.1",
"--tree=dir={0}".format(self.checkout_location),
"--gpg-sign={0}".format(self.key_id),
"--gpg-homedir={0}".format(BASEDIR)])
def install_packages(self, packages, deps=True, replace=False):
args = ["rpm", "-U", "--root", self.checkout_location,
"--dbpath", self.rpm_location]
if replace:
args.extend(["--replacepkgs", "--replacefiles"])
if not deps:
args.append("--nodeps")
for package in packages:
args.append(os.path.abspath(os.path.join(os.getcwd(), package)))
subprocess.check_call(args)
def remove_packages(self, packages):
args = ["rpm", "-e", "--root", self.checkout_location,
"--dbpath", self.rpm_location]
args.extend(packages)
subprocess.check_call(args)
def package_basename(self, package):
""" only accept package with the name 'cockpit-%s-*' and return 'cockpit-%s' or None"""
basename = "-".join(package.split("-")[:2])
if basename.startswith("cockpit-"):
return basename
else:
return None
def update_container(self):
""" Install the latest cockpit RPMs in our container"""
rpm_args = []
for package in self.rpms:
if 'cockpit-ws' in package or 'cockpit-dashboard' in package or 'cockpit-bridge' in package:
rpm_args.append("/host" + package)
extra_args = []
for package in self.extra_rpms:
extra_args.append("/host" + package)
if rpm_args:
subprocess.check_call(["docker", "run", "--name", "build-cockpit",
"-d", "--privileged", "-v", "/:/host",
"cockpit/ws", "sleep", "1d"])
if self.verbose:
print("updating cockpit-ws container")
if extra_args:
subprocess.check_call(["docker", "exec", "build-cockpit",
"rpm", "--install", "--verbose", "--force"] + extra_args)
subprocess.check_call(["docker", "exec", "build-cockpit",
"rpm", "--freshen", "--verbose", "--force"] + rpm_args)
# if we update the RPMs, also update the scripts, to keep them in sync
subprocess.check_call(["docker", "exec", "build-cockpit", "sh", "-exc",
"cp /host/var/tmp/containers/ws/atomic-* /container/"])
subprocess.check_call(["docker", "commit", "build-cockpit",
"cockpit/ws"])
subprocess.check_call(["docker", "kill", "build-cockpit"])
subprocess.check_call(["docker", "rm", "build-cockpit"])
def package_basenames(self, package_names):
""" convert a list of package names to a list of their basenames """
return list(filter(lambda s: s is not None, map(self.package_basename, package_names)))
def get_installed_cockpit_packages(self):
""" get list installed cockpit packages """
packages = subprocess.check_output("rpm -qa | grep cockpit", shell=True, universal_newlines=True)
if self.verbose:
print("installed packages: {0}".format(packages))
installed_packages = packages.strip().split("\n")
return installed_packages
def clean_network(self):
if self.verbose:
print("clean network configuration:")
subprocess.check_call(["rm", "-rf", "/var/lib/NetworkManager"])
subprocess.check_call(["rm", "-rf", "/var/lib/dhcp"])
def run(self):
# Delete previous deployment if it's present
output = subprocess.check_output(["ostree", "admin", "status"])
if output.count(b"origin refspec") != 1:
subprocess.check_call(["ostree", "admin", "undeploy", "1"])
self.setup_dirs()
installed_packages = self.get_installed_cockpit_packages()
self.remove_packages(installed_packages)
packages_to_install = self.package_basenames(installed_packages)
for p in self.packages_force_install:
if not p in packages_to_install:
if self.verbose:
print("adding package %s (forced)" % (p))
packages_to_install.append(p)
packages_to_install = list(filter(lambda p: any(os.path.split(p)[1].startswith(base) for base in packages_to_install), self.rpms))
if self.verbose:
print("packages to install:")
print(packages_to_install)
if self.external_packages:
names = self.external_packages.keys()
if self.verbose:
print("external packages to install:")
print(list(names))
downloader = URLopener()
for name, url in self.external_packages.items():
downloader.retrieve(url, name)
self.install_packages(names, replace=True)
for name in names:
os.remove(name)
self.install_packages(packages_to_install)
no_deps = [x for x in self.rpms \
if os.path.split(x)[-1].startswith("cockpit-tests") or
os.path.split(x)[-1].startswith("cockpit-machines")]
self.install_packages(no_deps, deps=False, replace=True)
# If firewalld is installed, we need to poke a hole for cockpit, so
# that we can run firewall tests on it (change firewall-cmd to
# --add-service=cockpit once all supported atomics ship with the
# service file)
if subprocess.call(["systemctl", "enable", "--now", "firewalld"]) == 0:
subprocess.call(["firewall-cmd", "--permanent", "--add-port=9090/tcp"])
self.commit_to_repo()
self.switch_to_local_tree()
self.update_container()
self.clean_network()
parser = argparse.ArgumentParser(description='Install Cockpit in Atomic')
parser.add_argument('-v', '--verbose', action='store_true', help='Display verbose progress details')
parser.add_argument('-q', '--quick', action='store_true', help='Build faster')
parser.add_argument('--build', action='store_true', help='Build')
parser.add_argument('--install', action='store_true', help='Install')
parser.add_argument('--extra', action='append', default=[], help='Extra packages to install inside the container')
parser.add_argument('--skip', action='append', default=[], help='Packes to skip during installation')
args = parser.parse_args()
if args.build:
sys.stderr.write("Can't build on Atomic\n")
sys.exit(1)
if args.install:
os.chdir("build-results")
# Force skip cockpit-dashboard
if args.skip:
skip = list(args.skip)
else:
skip = []
skip.append("cockpit-dashboard")
rpms = [os.path.abspath(f) for f in os.listdir(".")
if (f.endswith(".rpm") and not f.endswith(".src.rpm")
and not any(f.startswith(s) for s in args.skip))]
cockpit_installer = AtomicCockpitInstaller(rpms=rpms, extra_rpms=args.extra, verbose=args.verbose)
cockpit_installer.run()
# vim: ft=python

View File

@ -1,78 +0,0 @@
#!/bin/bash
# This file is part of Cockpit.
#
# Copyright (C) 2015 Red Hat, Inc.
#
# Cockpit is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Cockpit is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
set -ex
# The docker pool should grow automatically as needed, but we grow it
# explicitly here anyway. This is hopefully more reliable.
# Newer Fedora versions configure docker to use the root LV
# HACK: docker falls over regularly, print its log if it does
systemctl start docker || journalctl -u docker
lvresize atomicos/root -l+60%FREE -r
if lvs atomicos/docker-pool 2>/dev/null; then
lvresize atomicos/docker-pool -l+100%FREE
elif lvs atomicos/docker-root-lv; then
lvresize atomicos/docker-root-lv -l+100%FREE
fi
# docker images that we need for integration testing
/var/lib/testvm/docker-images.setup
# Download the libssh RPM plus dependencies which we'll use for
# package overlay. The only way to do this is via a container
. /etc/os-release
REPO="updates"
if [ "$ID" = "rhel" ]; then
subscription-manager repos --enable rhel-7-server-extras-rpms
REPO="rhel-7-server-extras-rpms"
ID="rhel7"
fi
docker run --rm --volume=/etc/yum.repos.d:/etc/yum.repos.d:z --volume=/root/rpms:/tmp/rpms:rw,z "$ID:$VERSION_ID" /bin/sh -cex "yum install -y findutils createrepo yum-utils && (cd /tmp/; yumdownloader --enablerepo=$REPO libssh) && find /tmp -name '*.$(uname -m).*rpm' | while read rpm; do mv -v \$rpm /tmp/rpms; done; createrepo /tmp/rpms"
rm -f /etc/yum.repos.d/*
cat >/etc/yum.repos.d/deps.repo <<EOF
[deps]
baseurl=file:///root/rpms
enabled=1
EOF
# fully upgrade host. Anything past this point can't touch /etc
# Upgrade host if there is a valid upgrade available (we might be on a RC)
if rpm-ostree upgrade --check; then
atomic host upgrade
# HACK - Find a better way to compute the ref.
# https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2016-July/msg00015.html
checkout=$(atomic host status --json | python -c 'import json; import sys; j = json.loads(sys.stdin.readline()); print j["deployments"][0]["origin"]')
else
checkout=$(atomic host status --json | python -c 'import json; import sys; j = json.loads(sys.stdin.readline()); print [x for x in j["deployments"] if x["booted"]][0]["checksum"]')
fi
# Checkout the just upgraded os branch since we'll use it every time
# we build a new tree.
ostree checkout "$checkout" /var/local-tree
# reduce image size
/var/lib/testvm/zero-disk.setup
# Prevent SSH from hanging for a long time when no external network access
echo 'UseDNS no' >> /etc/ssh/sshd_config
# Final tweaks
rm -rf /var/log/journal/*

View File

@ -1,5 +0,0 @@
FROM fedora:30
ADD setup.sh /setup.sh
RUN /setup.sh

View File

@ -1,5 +0,0 @@
Cockpit Base
===========================
Simple base container that installs cockpit-ws dependencies. Used in testing
and development to speed up container build times.

View File

@ -1,26 +0,0 @@
#! /bin/sh
upgrade() {
# https://bugzilla.redhat.com/show_bug.cgi?id=1483553
dnf -v -y update 2>err.txt
ecode=$?
if [ $ecode -ne 0 ] ; then
grep -q -F -e "BDB1539 Build signature doesn't match environment" err.txt
if [ $? -eq 0 ]; then
set -eu
rpm --rebuilddb
dnf -v -y update
else
cat err.txt
exit ${ecode}
fi
fi
}
upgrade
set -eu
dnf install -y sed findutils glib-networking json-glib libssh openssl python3
dnf clean all

View File

@ -1,16 +0,0 @@
#!/bin/bash
set -eu
# Download cockpit.spec, replace `npm-version` macro and then query all build requires
curl -s https://raw.githubusercontent.com/cockpit-project/cockpit/master/tools/cockpit.spec |
sed 's/%{npm-version:.*}/0/' |
sed '/Recommends:/d' |
rpmspec -D "$1" --buildrequires --query /dev/stdin |
sed 's/.*/"&"/' |
tr '\n' ' '
# support for backbranches
if [ "$1" = "rhel 7" ] || [ "$1" = "centos 7" ]; then
echo "golang-bin golang-src"
fi

View File

@ -1,35 +0,0 @@
#!/bin/bash
# This file is part of Cockpit.
#
# Copyright (C) 2016 Red Hat, Inc.
#
# Cockpit is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Cockpit is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
set -ex
# HACK: docker falls over regularly, print its log if it does
systemctl start docker || journalctl -u docker
for NAME in bastion
do
mkdir -p "/var/tmp/containers/$NAME/rpms"
cp -f /var/tmp/build-results/*.rpm "/var/tmp/containers/$NAME/rpms/"
cd "/var/tmp/containers/$NAME/"
sed -i -e "s#FROM .*#FROM cockpit/base#" Dockerfile
docker build --build-arg OFFLINE=1 -t "cockpit/$NAME" . 1>&2;
rm -r "/var/tmp/containers/$NAME/rpms"
done
journalctl --flush || true
journalctl --sync || killall systemd-journald || true
rm -rf /var/log/journal/* || true

View File

@ -1,13 +0,0 @@
#! /bin/bash
set -ex
out=$1
# download cloud image; re-use a previously downloaded image
image_url="$2"
image=tmp/$(basename $image_url)
mkdir -p $(dirname $image)
[ -f "$image" ] || curl -o "$image" "$image_url"
cp "$image" "$out"
qemu-img resize -f qcow2 "$out" +8G

View File

@ -1,97 +0,0 @@
#! /bin/sh
set -ex
export DEB_BUILD_OPTIONS=""
do_build=
do_install=
stdout_dest="/dev/null"
args=$(getopt -o "vqs:" -l "verbose,quick,skip:,build,install" -- "$@")
eval set -- "$args"
while [ $# -gt 0 ]; do
case $1 in
-v|--verbose)
stdout_dest="/dev/stdout"
;;
-q|--quick)
DEB_BUILD_OPTIONS="$DEB_BUILD_OPTIONS nocheck"
;;
--build)
do_build=t
;;
--install)
do_install=t
;;
--)
shift
break
;;
esac
shift
done
tar="$1"
# Build
if [ -n "$do_build" ]; then
rm -rf build-results
mkdir build-results
resultdir=$PWD/build-results
upstream_ver=$(ls cockpit-*.tar.gz | sed 's/^.*-//; s/.tar.gz//' | head -n1)
ln -sf cockpit-*.tar.gz cockpit_${upstream_ver}.orig.tar.gz
rm -rf cockpit-*/
tar -xzf cockpit-*.tar.gz
( cd cockpit-*/
cp -rp tools/debian debian
# put proper version into changelog, as we have versioned dependencies
sed -i "1 s/(.*)/($upstream_ver-1)/" debian/changelog
# Hack: Remove PCP build dependencies while pcp is not in testing
# (https://tracker.debian.org/pcp)
sed -i '/libpcp.*-dev/d' debian/control
dpkg-buildpackage -S -uc -us -nc
)
# Some unit tests want a real network interface
echo USENETWORK=yes >>~/.pbuilderrc
# pbuilder < 0.228.6 has broken /dev/pts/ptmx permissions; affects Ubuntu < 17.04
# see https://bugs.debian.org/841935
if ! grep -q ptmxmode /usr/lib/pbuilder/pbuilder-modules; then
echo "Fixing /dev/pts/ptmx mode in pbuilder"
sed -i '/mount -t devpts none/ s/$/,ptmxmode=666,newinstance/' /usr/lib/pbuilder/pbuilder-modules
fi
pbuilder build --buildresult "$resultdir" \
--logfile "$resultdir/build.log" \
cockpit_${upstream_ver}-1.dsc >$stdout_dest
lintian $resultdir/cockpit_*_$(dpkg --print-architecture).changes >&2
fi
# Install
if [ -n "$do_install" ]; then
packages=$(find build-results -name "*.deb")
dpkg --install $packages
# FIXME: our tests expect cockpit.socket to not be running after boot, only
# after start_cockpit().
systemctl disable cockpit.socket
# HACK: tuned breaks QEMU (https://launchpad.net/bugs/1774000)
systemctl disable tuned.service 2>/dev/null || true
# avoid random dpkg database locks, they break our package related tests
systemctl disable apt-daily-upgrade.timer
firewall-cmd --add-service=cockpit --permanent
# not managed by NM, so enable interface manually
firewall-cmd --zone=public --permanent --add-interface=eth1
journalctl --flush
journalctl --sync || killall systemd-journald
rm -rf /var/log/journal/*
fi

View File

@ -1,36 +0,0 @@
#!/bin/bash
set -ex
# This file is part of Cockpit.
#
# Copyright (C) 2016 Red Hat, Inc.
#
# Cockpit is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Cockpit is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
if [ $(uname -m) = x86_64 ]; then
docker pull busybox:latest
docker pull busybox:buildroot-2014.02
docker pull gcr.io/google_containers/pause:0.8.0
docker pull k8s.gcr.io/pause-amd64:3.1
# some aliases for different k8s variants
docker tag k8s.gcr.io/pause-amd64:3.1 gcr.io/google_containers/pause-amd64:3.0
docker tag k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1
fi
# Download the i386 image and rename it
if [ $(uname -m) = i686 ]; then
docker pull i386/busybox:latest
docker tag docker.io/i386/busybox busybox
docker rmi docker.io/i386/busybox
fi

View File

@ -1,120 +0,0 @@
#! /bin/bash
set -ex
# don't update already installed cockpit packages
installed=$(rpm --query --all --queryformat "%{NAME}-\[0-9\]\n" "cockpit*")
skip="cockpit-doc-[0-9]"
if [ -n "$installed" ]; then
skip="$skip
$installed"
fi
do_build=
do_install=
# we build RHEL 7.x in a CentOS mock, thus we can't parse os-release in the .spec
mock_opts="--define='os_version_id $(. /etc/os-release; echo $VERSION_ID)'"
args=$(getopt -o "vqs:" -l "verbose,quick,skip:,build,install,rhel,HACK-no-bootstrap-chroot" -- "$@")
eval set -- "$args"
while [ $# -gt 0 ]; do
case $1 in
-v|--verbose)
mock_opts="$mock_opts --verbose"
;;
-q|--quick)
mock_opts="$mock_opts --nocheck --define='selinux 0'"
;;
-s|--skip)
skip="$skip
$2"
shift
;;
--build)
do_build=t
;;
--install)
do_install=t
;;
--rhel)
# For RHEL we actually build in EPEL, which is based
# on CentOS. On CentOS, the spec file has both
# %centos and %rhel defined, but it gives precedence
# to %centos, as it must. To make it produce the RHEL
# packages, we explicitly undefine %centos here.
mock_opts="$mock_opts --define='centos 0'"
;;
--HACK-no-bootstrap-chroot)
mock_opts="$mock_opts --no-bootstrap-chroot"
;;
--)
shift
break
;;
esac
shift
done
tar=$1
# Build
if [ -n "$do_build" ]; then
# Some tests need a non-loopback internet address, so we allow
# networking during build. Note that we use "--offline" below, so
# we should still be protected against unexpected package
# installations.
echo "config_opts['rpmbuild_networking'] = True" >>/etc/mock/site-defaults.cfg
# don't destroy the mock after building, we want to run rpmlint
echo "config_opts['cleanup_on_success'] = False" >>/etc/mock/site-defaults.cfg
# HACK: don't fall over on unavailable repositories, as we are offline
# (https://bugzilla.redhat.com/show_bug.cgi?id=1549291)
sed --follow-symlinks -i '/skip_if_unavailable=False/d' /etc/mock/default.cfg
rm -rf build-results
srpm=$(/var/lib/testvm/make-srpm "$tar")
LC_ALL=C.UTF-8 su builder -c "/usr/bin/mock --offline --no-clean --resultdir build-results $mock_opts --rebuild $srpm"
cat <<EOF >/tmp/rpmlint
#! /bin/bash
rm -rf /builddir/build
if type rpmlint >/dev/null 2>&1; then
# blacklist "E: no-changelogname-tag" rpmlint error, expected due to our template cockpit.spec
mkdir -p ~/.config
echo 'addFilter("E: no-changelogname-tag")' > ~/.config/rpmlint
# we expect the srpm to be clean
echo
echo '====== rpmlint on srpm ====='
rpmlint /builddir/build/SRPMS/*.src.rpm
# this still has lots of errors, run it for information only
echo
echo '====== rpmlint binary rpms (advisory) ====='
rpmlint /builddir/build/RPMS/ || true
else
echo '====== skipping rpmlint check, not installed ====='
fi
EOF
chmod +x /tmp/rpmlint
su builder -c "/usr/bin/mock --offline --copyin /tmp/rpmlint /var/tmp/rpmlint"
su builder -c "/usr/bin/mock --offline --shell /var/tmp/rpmlint"
fi
# Install
if [ -n "$do_install" ]; then
packages=$(find build-results -name "*.rpm" -not -name "*.src.rpm" | grep -vG "$skip")
rpm -U --force $packages
if type firewall-cmd > /dev/null 2> /dev/null; then
systemctl start firewalld
firewall-cmd --add-service=cockpit --permanent
fi
# Make sure we clean out the journal
journalctl --flush
journalctl --sync || killall systemd-journald
rm -rf /var/log/journal/*
rm -rf /var/lib/NetworkManager/dhclient-*.lease
fi
if [ -n "$do_build" ]; then
su builder -c "/usr/bin/mock --clean"
fi

View File

@ -1,46 +0,0 @@
#!/bin/bash
# Kubernetes is delivered in a non-functional state on Fedora and similar operating systems
# The following commands are needed to get it running.
cd /etc/kubernetes/
cat <<EOF > openssl.conf
oid_section = new_oids
[new_oids]
[req]
encrypt_key = no
string_mask = nombstr
req_extensions = v3_req
distinguished_name = v3_name
[v3_name]
commonName = kubernetes
[v3_req]
basicConstraints = CA:FALSE
subjectAltName = @alt_names
[alt_names]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
DNS.3 = kubernetes.default.svc
DNS.4 = kubernetes.default.svc.cluster.local
IP.1 = 127.0.0.1
IP.2 = 10.254.0.1
EOF
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -days 3072 -out ca.crt -subj '/CN=kubernetes'
openssl genrsa -out server.key 2048
openssl req -config openssl.conf -new -key server.key -out server.csr -subj '/CN=kubernetes'
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 3072 -extensions v3_req -extfile openssl.conf
# make keys readable for "kube" group and thus for kube-apiserver.service on newer OSes
if getent group kube >/dev/null; then
chgrp kube ca.key server.key
chmod 640 ca.key server.key
fi
echo -e '{"user":"admin"}\n{"user":"scruffy","readonly": true}' > /etc/kubernetes/authorization
echo -e 'fubar,admin,10101\nscruffy,scruffy,10102' > /etc/kubernetes/passwd
echo 'KUBE_API_ARGS="--service-account-key-file=/etc/kubernetes/server.key --client-ca-file=/etc/kubernetes/ca.crt --tls-cert-file=/etc/kubernetes/server.crt --tls-private-key-file=/etc/kubernetes/server.key --basic-auth-file=/etc/kubernetes/passwd --authorization-mode=ABAC --authorization-policy-file=/etc/kubernetes/authorization"' >> apiserver
echo 'KUBE_CONTROLLER_MANAGER_ARGS="--root-ca-file=/etc/kubernetes/ca.crt --service-account-private-key-file=/etc/kubernetes/server.key"' >> controller-manager

View File

@ -1,33 +0,0 @@
#!/bin/bash
set -eu
tar=$1
version=$(echo "$1" | sed -n 's|.*cockpit-\([^ /-]\+\)\.tar\..*|\1|p')
if [ -z "$version" ]; then
echo "make-srpm: couldn't parse version from tarball: $1"
exit 2
fi
# We actually modify the spec so that the srpm is standalone buildable
modify_spec() {
sed -e "/^Version:.*/d" -e "1i\
%define wip wip\nVersion: $version\n"
}
tmpdir=$(mktemp -d $PWD/srpm-build.XXXXXX)
tar xaf "$1" -O cockpit-$version/tools/cockpit.spec | modify_spec > $tmpdir/cockpit.spec
rpmbuild -bs \
--quiet \
--define "_sourcedir $(dirname $1)" \
--define "_specdir $tmpdir" \
--define "_builddir $tmpdir" \
--define "_srcrpmdir `pwd`" \
--define "_rpmdir $tmpdir" \
--define "_buildrootdir $tmpdir/.build" \
$tmpdir/cockpit.spec
rpm --qf '%{Name}-%{Version}-%{Release}.src.rpm\n' -q --specfile $tmpdir/cockpit.spec | head -n1
rm -rf $tmpdir

Binary file not shown.

Binary file not shown.

View File

@ -1,50 +0,0 @@
#!/bin/bash
# This file is part of Cockpit.
#
# Copyright (C) 2016 Red Hat, Inc.
#
# Cockpit is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Cockpit is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
# We don't want to delete the pbuilder caches since we need them during build.
# Mock with --offline and dnf is sometimes happy without caches, and with yum it
# never is, so we provide an option to also leave the mock caches in place.
#
# We also want to keep cracklib since otherwise password quality
# checks break on Debian.
if [ -f /root/.skip-zero-disk ]; then
echo "Skipping zero-disk.setup as /root/.skip-zero-disk exists"
exit 0
fi
keep="! -path /var/cache/pbuilder ! -path /var/cache/cracklib ! -path /var/cache/tomcat"
while [ $# -gt 0 ]; do
case $1 in
--keep-mock-cache)
keep="$keep ! -path /var/cache/mock"
;;
esac
shift
done
if [ -d "/var/cache" ]; then
find /var/cache/* -maxdepth 0 -depth -name "*" $keep -exec rm -rf {} \;
fi
rm -rf /var/tmp/*
rm -rf /var/log/journal/*
dd if=/dev/zero of=/root/junk || true
sync
rm -f /root/junk

View File

@ -1,3 +0,0 @@
BOOTPROTO="dhcp"
DEVICE="eth0"
ONBOOT="yes"

View File

@ -1,3 +0,0 @@
BOOTPROTO="none"
DEVICE="eth1"
ONBOOT="no"

View File

@ -1,4 +0,0 @@
#! /bin/bash
BASE=$(dirname $0)
$BASE/virt-install-fedora "$1" x86_64 "http://mirror.centos.org/centos/7/os/x86_64/"

View File

@ -1,2 +0,0 @@
#!/bin/sh
# By default this does nothing

Some files were not shown because too many files have changed in this diff Show More