bots: Move bots and related tasks over to cockpit-project/bots
Closes #12872
This commit is contained in:
parent
2c8be2591a
commit
91d73df7df
|
@ -99,11 +99,7 @@ depcomp
|
|||
/src/selinux/cockpit.pp
|
||||
/pkg/*/test-*.log
|
||||
/pkg/*/test-*.trs
|
||||
/bots/images/*.qcow2
|
||||
/bots/images/*.partial
|
||||
/bots/images/*.xz
|
||||
/bots/images/*.img
|
||||
/bots/make-checkout-workdir
|
||||
/bots/
|
||||
/test/images/*
|
||||
/test/verify/naughty-*/*
|
||||
/test/container-probe*
|
||||
|
|
2
.tasks
2
.tasks
|
@ -23,8 +23,6 @@ fi
|
|||
# File issues for these tasks
|
||||
if [ $chance -gt 9 ]; then
|
||||
bots/po-trigger
|
||||
bots/image-trigger
|
||||
bots/npm-trigger
|
||||
bots/naughty-trigger
|
||||
bots/learn-trigger
|
||||
fi
|
||||
|
|
|
@ -1,6 +0,0 @@
|
|||
*.pyc
|
||||
*.qcow2
|
||||
*.partial
|
||||
*.xz
|
||||
/*.log
|
||||
/build-results/
|
|
@ -1,38 +0,0 @@
|
|||
# Hacking on the Cockpit Bots
|
||||
|
||||
These are automated bots and testing that works on the Cockpit project. This
|
||||
includes updating operating system images, bringing in changes from other
|
||||
projects, releasing Cockpit and more.
|
||||
|
||||
## Environment for the bots
|
||||
|
||||
The bots work in containers that are built in the [cockpituous](https://github.com/cockpit-project/cockpituous)
|
||||
repository. New dependencies should be added there in the `tests/Dockerfile`
|
||||
file in that repository.
|
||||
|
||||
## Invoking the bots
|
||||
|
||||
1. The containers in the `cockpitous` repository invoke the `.tasks` file
|
||||
at root of this repository.
|
||||
1. The ```.tasks``` file prints out a list of possible tasks on standard out.
|
||||
1. The printed tasks are sorted in alphabetical reverse order, and one of the
|
||||
first items in the list is executed.
|
||||
|
||||
## The bots themselves
|
||||
|
||||
Most bots are python scripts. They live in this `bots/` directory. Shared code
|
||||
is in the `bots/tasks` directory.
|
||||
|
||||
## Bots filing issues
|
||||
|
||||
Many bots file or work with issues in GitHub repository. We can use issues to tell
|
||||
bots what to do. Often certan bots will just file issues for tasks that are outstanding.
|
||||
And in many cases other bots will then perform those tasks.
|
||||
|
||||
These bots are listed in the `bots/issue-scan` file. They are written using the
|
||||
`bots/tasks/__init__.py` code, and you can see `bots/example-task` for an
|
||||
example of one.
|
||||
|
||||
## Bots printing output
|
||||
|
||||
The bot output is posted using the cockpitous [sink](https://github.com/cockpit-project/cockpituous/tree/master/sink) code. See that link for how it works.
|
122
bots/README.md
122
bots/README.md
|
@ -1,122 +0,0 @@
|
|||
# Cockpit Bots
|
||||
|
||||
These are automated bots and tools that work on Cockpit. This
|
||||
includes updating operating system images, testing changes,
|
||||
releasing Cockpit and more.
|
||||
|
||||
## Images
|
||||
|
||||
In order to test Cockpit-related projects, they are staged into an operating
|
||||
system image. These images are tracked in the ```bots/images``` directory.
|
||||
|
||||
These well known image names are expected to contain no ```.```
|
||||
characters and have no file name extension.
|
||||
|
||||
For managing these images:
|
||||
|
||||
* image-download: Download test images
|
||||
* image-upload: Upload test images
|
||||
* image-create: Create test machine images
|
||||
* image-customize: Generic tool to install packages, upload files, or run
|
||||
commands in a test machine image
|
||||
* image-prepare: Build and install Cockpit packages into a test machine image
|
||||
(specific to the cockpit project itself, thus it is in test/, not bots/)
|
||||
|
||||
For debugging the images:
|
||||
|
||||
* bots/vm-run: Run a test machine image
|
||||
* bots/vm-reset: Remove all overlays from image-customize, image-prepare, etc
|
||||
from test/images/
|
||||
|
||||
In case of `qemu-system-x86_64: -netdev bridge,br=cockpit1,id=bridge0: bridge helper failed`
|
||||
error, please [allow][1] `qemu-bridge-helper` to access the bridge settings.
|
||||
|
||||
To check when images will automatically be refreshed by the bots
|
||||
use the image-trigger tool:
|
||||
|
||||
$ bots/image-trigger -vd
|
||||
|
||||
## Tests
|
||||
|
||||
The bots automatically run the tests as needed on pull requests
|
||||
and branches. To check when and where tests will be run, use the
|
||||
tests-scan tool:
|
||||
|
||||
$ bots/tests-scan -vd
|
||||
|
||||
#### Note on eslintrc interaction
|
||||
|
||||
As eslint looks for additional configurations, eslintrc.(json|yaml) files, in
|
||||
parent directories, it is recommended to have `"root": true` in the eslint
|
||||
configuration of any project which is using eslint and is tested through
|
||||
cockpit-bots.
|
||||
|
||||
## Integration with GitHub
|
||||
|
||||
A number of machines are watching our GitHub repository and are
|
||||
executing tests for pull requests as well as making new images.
|
||||
|
||||
Most of this happens automatically, but you can influence their
|
||||
actions with the tests-trigger utility in this directory.
|
||||
|
||||
### Setup
|
||||
|
||||
You need a GitHub token in ~/.config/github-token. You can create one
|
||||
for your account at
|
||||
|
||||
https://github.com/settings/tokens
|
||||
|
||||
When generating a new personal access token, the scope only needs to
|
||||
encompass public_repo (or repo if you're accessing a private repo).
|
||||
|
||||
### Retrying a failed test
|
||||
|
||||
If you want to run the "verify/fedora-atomic" testsuite again for pull
|
||||
request #1234, run tests-trigger like so:
|
||||
|
||||
$ bots/tests-trigger 1234 verify/fedora-atomic
|
||||
|
||||
### Testing a pull request by a non-whitelisted user
|
||||
|
||||
If you want to run all tests on pull request #1234 that has been
|
||||
opened by someone who is not in our white-list, run tests-trigger
|
||||
like so:
|
||||
|
||||
$ bots/tests-trigger -f 1234
|
||||
|
||||
Of course, you should make sure that the pull request is proper and
|
||||
doesn't execute evil code during tests.
|
||||
|
||||
### Refreshing a test image
|
||||
|
||||
Test images are refreshed automatically once per week, and even if the
|
||||
last refresh has failed, the machines wait one week before trying again.
|
||||
|
||||
If you want the machines to refresh the fedora-atomic image immediately,
|
||||
run image-trigger like so:
|
||||
|
||||
$ bots/image-trigger fedora-atomic
|
||||
|
||||
### Creating new images for a pull request
|
||||
|
||||
If as part of some new feature you need to change the content of some
|
||||
or all images, you can ask the machines to create those images.
|
||||
|
||||
If you want to have a new fedora-atomic image for pull request #1234, add
|
||||
a bullet point to that pull request's description like so, and add the
|
||||
"bot" label to the pull request.
|
||||
|
||||
* [ ] image-refresh fedora-atomic
|
||||
|
||||
The machines will post comments to the pull request about their
|
||||
progress and at the end there will be links to commits with the new
|
||||
images. You can then include these commits into the pull request in
|
||||
any way you like.
|
||||
|
||||
If you are certain about the changes to the images, it is probably a
|
||||
good idea to make a dedicated pull request just for the images. That
|
||||
pull request can then hopefully be merged to master faster. If
|
||||
instead the images are created on the main feature pull request and
|
||||
sit there for a long time, they might cause annoying merge conflicts.
|
||||
|
||||
[1]: https://blog.christophersmart.com/2016/08/31/configuring-qemu-bridge-helper-after-access-denied-by-acl-file-error/
|
|
@ -1,49 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
|
||||
# This file is part of Cockpit.
|
||||
#
|
||||
# Copyright (C) 2016 Red Hat, Inc.
|
||||
#
|
||||
# Cockpit is free software; you can redistribute it and/or modify it
|
||||
# under the terms of the GNU Lesser General Public License as published by
|
||||
# the Free Software Foundation; either version 2.1 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# Cockpit is distributed in the hope that it will be useful, but
|
||||
# WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
# Lesser General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU Lesser General Public License
|
||||
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
# To use this example add a line to an issue with the "bot" label
|
||||
#
|
||||
# * [ ] example-bot 20
|
||||
#
|
||||
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
sys.dont_write_bytecode = True
|
||||
|
||||
import task
|
||||
|
||||
BOTS = os.path.abspath(os.path.dirname(__file__))
|
||||
BASE = os.path.normpath(os.path.join(BOTS, ".."))
|
||||
|
||||
def run(argument, verbose=False, **kwargs):
|
||||
try:
|
||||
int(argument)
|
||||
except (TypeError, ValueError):
|
||||
return "Failed to parse argument"
|
||||
|
||||
sys.stdout.write("Example message to log\n")
|
||||
|
||||
# Attach the package.json script as an example
|
||||
task.attach("./package.json")
|
||||
time.sleep(20)
|
||||
|
||||
if __name__ == '__main__':
|
||||
task.main(function=run, title="Example bot task")
|
|
@ -1,186 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
|
||||
# This file is part of Cockpit.
|
||||
#
|
||||
# Copyright (C) 2018 Red Hat, Inc.
|
||||
#
|
||||
# Cockpit is free software; you can redistribute it and/or modify it
|
||||
# under the terms of the GNU Lesser General Public License as published by
|
||||
# the Free Software Foundation; either version 2.1 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# Cockpit is distributed in the hope that it will be useful, but
|
||||
# WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
# Lesser General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU Lesser General Public License
|
||||
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
import sys
|
||||
import time
|
||||
import os
|
||||
import urllib
|
||||
import json
|
||||
import re
|
||||
import ssl
|
||||
|
||||
sys.dont_write_bytecode = True
|
||||
|
||||
import task
|
||||
|
||||
BOTS = os.path.dirname(os.path.realpath(__file__))
|
||||
|
||||
NUMBER_OPEN_ISSUES = 7 # How many issues do we want to have open at a given time?
|
||||
|
||||
# How far back does our data go? If a flake gets fixed but is still
|
||||
# flaky after this many days, the bots open another issue.
|
||||
|
||||
WINDOW_DAYS = 21
|
||||
|
||||
# This parses the output JSONL format discussed here, where various
|
||||
# values are grouped:
|
||||
#
|
||||
# https://github.com/cockpit-project/cockpituous/blob/master/learn/README.md
|
||||
|
||||
# Here we're looking for a field in a record that only has one value
|
||||
def value(record, field):
|
||||
values = record.get(field, [])
|
||||
if len(values) == 1:
|
||||
return values[0][0] or ""
|
||||
return None
|
||||
|
||||
# Here we're looking for the count of a specific field/value in the record
|
||||
def count(record, field, only):
|
||||
values = record.get(field, [])
|
||||
for value, count in values:
|
||||
if value != only:
|
||||
continue
|
||||
return count
|
||||
return 0
|
||||
|
||||
# For linking flakes to test logs
|
||||
|
||||
def slurp_one(url, n, logs):
|
||||
items_url = url + str(n) + "/items.jsonl"
|
||||
try:
|
||||
with urllib.request.urlopen(items_url) as f:
|
||||
for line in f.readlines():
|
||||
try:
|
||||
record = json.loads(line.decode('utf-8'))
|
||||
logs.setdefault(record["test"], [ ]).append(record["url"])
|
||||
except ValueError as ex:
|
||||
sys.stderr.write("{0}: {1}\n".format(url, ex))
|
||||
except urllib.error.URLError as ex:
|
||||
if ex.code == 404:
|
||||
return False
|
||||
raise
|
||||
return True
|
||||
|
||||
def slurp_failure_logs(url):
|
||||
logs = { }
|
||||
n = 0
|
||||
while slurp_one(url, n, logs):
|
||||
n = n + 1
|
||||
return logs
|
||||
|
||||
def get_failure_logs(failure_logs, name, context):
|
||||
match = context.replace("/", "-")
|
||||
return sorted(filter(lambda url: match in url, failure_logs[name]), reverse=True)[0:10]
|
||||
|
||||
# Main
|
||||
|
||||
def run(context, verbose=False, **kwargs):
|
||||
api = task.github.GitHub()
|
||||
|
||||
open_issues = api.issues(labels=[ "flake" ])
|
||||
create_count = NUMBER_OPEN_ISSUES - len(open_issues)
|
||||
|
||||
if create_count <= 0:
|
||||
return 0
|
||||
|
||||
if verbose:
|
||||
sys.stderr.write("Going to create %s new flake issue(s)\n" % create_count)
|
||||
|
||||
host = os.environ.get("COCKPIT_LEARN_SERVICE_HOST", "learn-cockpit.apps.ci.centos.org")
|
||||
port = os.environ.get("COCKPIT_LEARN_SERVICE_PORT", "443")
|
||||
url = "{0}://{1}:{2}/active/".format("https" if port == "443" else "http", host, port)
|
||||
cafile = os.path.join(BOTS, "images", "files", "ca.pem")
|
||||
context = ssl.create_default_context(cafile=cafile)
|
||||
|
||||
failure_logs = slurp_failure_logs(url)
|
||||
|
||||
# Retrieve the URL
|
||||
statistics = [ ]
|
||||
with urllib.request.urlopen(url + "statistics.jsonl", context=context) as f:
|
||||
for line in f.readlines():
|
||||
try:
|
||||
record = json.loads(line.decode('utf-8'))
|
||||
statistics.append(record)
|
||||
except ValueError as ex:
|
||||
sys.stderr.write("{0}: {1}\n".format(url, ex))
|
||||
|
||||
tests = { }
|
||||
|
||||
for record in statistics:
|
||||
test = value(record, "test")
|
||||
context = value(record, "context")
|
||||
status = value(record, "status")
|
||||
tracker = value(record, "tracker")
|
||||
|
||||
# Flaky tests only score on those that fail and are not tracked
|
||||
if test is not None and status == "failure" and not tracker:
|
||||
merged = count(record, "merged", True)
|
||||
not_merged = count(record, "merged", False)
|
||||
null_merged = count(record, "merged", None)
|
||||
total = merged + not_merged + null_merged
|
||||
|
||||
# And the key is that they were merged anyway
|
||||
if total > 10:
|
||||
tests.setdefault(test, [ ]).append((merged / total, context, record))
|
||||
|
||||
scores = [ ]
|
||||
|
||||
for n, t in tests.items():
|
||||
scores.append((sum(map(lambda f: f[0], t))/len(t), n, t))
|
||||
|
||||
closed_issues = api.issues(labels=["flake"], state="closed", since=(time.time() - (WINDOW_DAYS * 86400)))
|
||||
|
||||
def find_in_issues(issues, name):
|
||||
for issue in issues:
|
||||
if name in issue["title"]:
|
||||
return True
|
||||
return False
|
||||
|
||||
def url_desc(url):
|
||||
m = re.search("pull-[0-9]+", url)
|
||||
return m.group(0) if m else url
|
||||
|
||||
def failure_description(name, f, logs):
|
||||
return ("%s%% on %s\n" % (int(f[0]*100), f[1]) +
|
||||
"".join(map(lambda url: " - [%s](%s)\n" % (url_desc(url), url),
|
||||
get_failure_logs(logs, name, f[1]))))
|
||||
|
||||
scores.sort(reverse=True)
|
||||
for score, name, failures in scores:
|
||||
if find_in_issues(open_issues, name) or find_in_issues(closed_issues, name):
|
||||
continue
|
||||
|
||||
if verbose:
|
||||
sys.stderr.write("Opening issue for %s\n" % name)
|
||||
source = "<details><summary>Source material</summary>\n\n```json\n%s\n```\n</details>\n" % "\n".join(map(lambda f: json.dumps(f[2], indent=2), failures))
|
||||
data = {
|
||||
"title": "%s is flaky" % name,
|
||||
"body": ("\n".join(map(lambda f: failure_description(name, f, failure_logs), failures)) +
|
||||
"\n\n" + source),
|
||||
"labels": [ "flake" ]
|
||||
}
|
||||
api.post("issues", data)
|
||||
create_count -= 1
|
||||
if create_count == 0:
|
||||
break
|
||||
|
||||
return 0
|
||||
|
||||
if __name__ == '__main__':
|
||||
task.main(function=run, title="Create issues for test flakes")
|
|
@ -1,73 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
|
||||
# This file is part of Cockpit.
|
||||
#
|
||||
# Copyright (C) 2015 Red Hat, Inc.
|
||||
#
|
||||
# Cockpit is free software; you can redistribute it and/or modify it
|
||||
# under the terms of the GNU Lesser General Public License as published by
|
||||
# the Free Software Foundation; either version 2.1 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# Cockpit is distributed in the hope that it will be useful, but
|
||||
# WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
# Lesser General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU Lesser General Public License
|
||||
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
# Shared GitHub code. When run as a script, we print out info about
|
||||
# our GitHub interacition.
|
||||
|
||||
import argparse
|
||||
import datetime
|
||||
import sys
|
||||
|
||||
sys.dont_write_bytecode = True
|
||||
|
||||
from task import github
|
||||
|
||||
def httpdate(dt):
|
||||
"""Return a string representation of a date according to RFC 1123
|
||||
(HTTP/1.1).
|
||||
|
||||
The supplied date must be in UTC.
|
||||
|
||||
From: http://stackoverflow.com/a/225106
|
||||
|
||||
"""
|
||||
weekday = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"][dt.weekday()]
|
||||
month = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep",
|
||||
"Oct", "Nov", "Dec"][dt.month - 1]
|
||||
return "%s, %02d %s %04d %02d:%02d:%02d GMT" % (weekday, dt.day, month,
|
||||
dt.year, dt.hour, dt.minute, dt.second)
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description='Test GitHub rate limits')
|
||||
parser.parse_args()
|
||||
|
||||
# in order for the limit not to be affected by the call itself,
|
||||
# use a conditional request with a timestamp in the future
|
||||
|
||||
future_timestamp = datetime.datetime.utcnow() + datetime.timedelta(seconds=3600)
|
||||
|
||||
api = github.GitHub()
|
||||
headers = { 'If-Modified-Since': httpdate(future_timestamp) }
|
||||
response = api.request("GET", "git/refs/heads/master", "", headers)
|
||||
sys.stdout.write("Rate limits:\n")
|
||||
for entry in ["X-RateLimit-Limit", "X-RateLimit-Remaining", "X-RateLimit-Reset"]:
|
||||
entries = [t for t in response['headers'].items() if t[0].lower() == entry.lower()]
|
||||
if entries:
|
||||
if entry == "X-RateLimit-Reset":
|
||||
try:
|
||||
readable = datetime.datetime.utcfromtimestamp(float(entries[0][1])).isoformat()
|
||||
except:
|
||||
readable = "parse error"
|
||||
pass
|
||||
sys.stdout.write("{0}: {1} ({2})\n".format(entry, entries[0][1], readable))
|
||||
else:
|
||||
sys.stdout.write("{0}: {1}\n".format(entry, entries[0][1]))
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
|
@ -1,210 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# This file is part of Cockpit.
|
||||
#
|
||||
# Copyright (C) 2015 Red Hat, Inc.
|
||||
#
|
||||
# Cockpit is free software; you can redistribute it and/or modify it
|
||||
# under the terms of the GNU Lesser General Public License as published by
|
||||
# the Free Software Foundation; either version 2.1 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# Cockpit is distributed in the hope that it will be useful, but
|
||||
# WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
# Lesser General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU Lesser General Public License
|
||||
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
# image-create -- Make a root image suitable for use with vm-run.
|
||||
#
|
||||
# Installs the OS indicated by TEST_OS into the image
|
||||
# for test machine and tweaks it to be useable with
|
||||
# vm-run and testlib.py.
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import tempfile
|
||||
|
||||
BOTS = os.path.abspath(os.path.dirname(__file__))
|
||||
BASE = os.path.normpath(os.path.join(BOTS, ".."))
|
||||
|
||||
from machine import testvm
|
||||
|
||||
parser = argparse.ArgumentParser(description='Create a virtual machine image')
|
||||
parser.add_argument('-v', '--verbose', action='store_true', help='Display verbose progress details')
|
||||
parser.add_argument('-s', '--sit', action='store_true', help='Sit and wait if setup script fails')
|
||||
parser.add_argument('-n', '--no-save', action='store_true', help='Don\'t save the new image')
|
||||
parser.add_argument('-u', '--upload', action='store_true', help='Upload the image after creation')
|
||||
parser.add_argument('--no-build', action='store_true', dest='no_build',
|
||||
help='Don''t build packages and create the vm without build capabilities')
|
||||
parser.add_argument("--store", default=None, help="Where to send images")
|
||||
parser.add_argument('image', help='The image to create')
|
||||
args = parser.parse_args()
|
||||
|
||||
# default to --no-build for some images
|
||||
if args.image in ["candlepin", "continuous-atomic", "fedora-atomic", "ipa", "rhel-atomic", "selenium", "openshift"]:
|
||||
if not args.no_build:
|
||||
if args.verbose:
|
||||
print("Creating machine without build capabilities based on the image type")
|
||||
args.no_build = True
|
||||
|
||||
class MachineBuilder:
|
||||
def __init__(self, machine):
|
||||
tempdir = testvm.get_temp_dir()
|
||||
self.machine = machine
|
||||
|
||||
os.makedirs(tempdir, 0o750, exist_ok=True)
|
||||
|
||||
# Use a tmp filename
|
||||
self.target_file = self.machine.image_file
|
||||
fp, self.machine.image_file = tempfile.mkstemp(dir=tempdir, prefix=self.machine.image, suffix=".qcow2")
|
||||
os.close(fp)
|
||||
|
||||
def bootstrap_system(self):
|
||||
assert not self.machine._domain
|
||||
|
||||
os.makedirs(self.machine.run_dir, 0o750, exist_ok=True)
|
||||
|
||||
bootstrap_script = os.path.join(testvm.SCRIPTS_DIR, "%s.bootstrap" % (self.machine.image, ))
|
||||
|
||||
|
||||
if os.path.isfile(bootstrap_script):
|
||||
subprocess.check_call([ bootstrap_script, self.machine.image_file ])
|
||||
else:
|
||||
raise testvm.Failure("Unsupported OS %s: %s not found." % (self.machine.image, bootstrap_script))
|
||||
|
||||
def run_setup_script(self, script):
|
||||
"""Prepare a test image further by running some commands in it."""
|
||||
self.machine.start()
|
||||
try:
|
||||
self.machine.wait_boot(timeout_sec=120)
|
||||
self.machine.upload([ os.path.join(testvm.SCRIPTS_DIR, "lib") ], "/var/lib/testvm")
|
||||
self.machine.upload([script], "/var/tmp/SETUP")
|
||||
self.machine.upload([ os.path.join(testvm.SCRIPTS_DIR, "lib", "base") ],
|
||||
"/var/tmp/cockpit-base")
|
||||
|
||||
if "rhel" in self.machine.image:
|
||||
self.machine.upload([ os.path.expanduser("~/.rhel") ], "/root/")
|
||||
|
||||
env = {
|
||||
"TEST_OS": self.machine.image,
|
||||
"DO_BUILD": "0" if args.no_build else "1",
|
||||
}
|
||||
self.machine.message("run setup script on guest")
|
||||
|
||||
try:
|
||||
self.machine.execute(script="/var/tmp/SETUP " + self.machine.image,
|
||||
environment=env, quiet=not self.machine.verbose, timeout=7200)
|
||||
self.machine.execute(command="rm -f /var/tmp/SETUP")
|
||||
self.machine.execute(command="rm -rf /root/.rhel")
|
||||
|
||||
if self.machine.image == 'openshift':
|
||||
# update our local openshift kube config file to match the new image
|
||||
self.machine.download("/root/.kube/config", os.path.join(BOTS, "images/files/openshift.kubeconfig"))
|
||||
|
||||
except subprocess.CalledProcessError as ex:
|
||||
if args.sit:
|
||||
sys.stderr.write(self.machine.diagnose())
|
||||
input ("Press RET to continue... ")
|
||||
raise testvm.Failure("setup failed with code {0}\n".format(ex.returncode))
|
||||
|
||||
finally:
|
||||
self.machine.stop(timeout_sec=500)
|
||||
|
||||
def boot_system(self):
|
||||
"""Start the system to make sure it can boot, then shutdown cleanly
|
||||
This also takes care of any selinux relabeling setup triggered
|
||||
Don't wait for an ip address during start, since the system might reboot"""
|
||||
self.machine.start()
|
||||
try:
|
||||
self.machine.wait_boot(timeout_sec=120)
|
||||
finally:
|
||||
self.machine.stop(timeout_sec=120)
|
||||
|
||||
def build(self):
|
||||
self.bootstrap_system()
|
||||
|
||||
# gather the scripts, separated by reboots
|
||||
script = os.path.join(testvm.SCRIPTS_DIR, "%s.setup" % (self.machine.image, ))
|
||||
|
||||
if not os.path.exists(script):
|
||||
return
|
||||
|
||||
self.machine.message("Running setup script %s" % (script))
|
||||
self.run_setup_script(script)
|
||||
|
||||
tries_left = 3
|
||||
successfully_booted = False
|
||||
while tries_left > 0:
|
||||
try:
|
||||
# make sure we can boot the system
|
||||
self.boot_system()
|
||||
successfully_booted = True
|
||||
break
|
||||
except:
|
||||
# we might need to wait for the image to become available again
|
||||
# accessing it in maintain=True mode successively can trigger qemu errors
|
||||
time.sleep(3)
|
||||
tries_left -= 1
|
||||
if not successfully_booted:
|
||||
raise testvm.Failure("Unable to verify that machine boot works.")
|
||||
|
||||
def save(self):
|
||||
data_dir = testvm.get_images_data_dir()
|
||||
|
||||
os.makedirs(data_dir, 0o750, exist_ok=True)
|
||||
|
||||
if not os.path.exists(self.machine.image_file):
|
||||
raise testvm.Failure("Nothing to save.")
|
||||
|
||||
partial = os.path.join(data_dir, self.machine.image + ".partial")
|
||||
|
||||
# Copy image via convert, to make it sparse again
|
||||
subprocess.check_call([ "qemu-img", "convert", "-c", "-O", "qcow2", self.machine.image_file, partial ])
|
||||
|
||||
# Hash the image here
|
||||
(sha, x1, x2) = subprocess.check_output([ "sha256sum", partial ], universal_newlines=True).partition(" ")
|
||||
if not sha:
|
||||
raise testvm.Failure("sha256sum returned invalid output")
|
||||
|
||||
name = self.machine.image + "-" + sha + ".qcow2"
|
||||
data_file = os.path.join(data_dir, name)
|
||||
shutil.move(partial, data_file)
|
||||
|
||||
# Remove temp image file
|
||||
os.unlink(self.machine.image_file)
|
||||
|
||||
# Update the images symlink
|
||||
if os.path.islink(self.target_file):
|
||||
os.unlink(self.target_file)
|
||||
os.symlink(name, self.target_file)
|
||||
|
||||
# Handle alternate images data directory
|
||||
image_file = os.path.join(testvm.IMAGES_DIR, name)
|
||||
if not os.path.exists(image_file):
|
||||
os.symlink(os.path.abspath(data_file), image_file)
|
||||
|
||||
try:
|
||||
testvm.VirtMachine.memory_mb = 2048
|
||||
machine = testvm.VirtMachine(verbose=args.verbose, image=args.image, maintain=True)
|
||||
builder = MachineBuilder(machine)
|
||||
builder.build()
|
||||
if not args.no_save:
|
||||
print("Saving...")
|
||||
builder.save()
|
||||
if args.upload:
|
||||
print("Uploading...")
|
||||
cmd = [ os.path.join(BOTS, "image-upload") ]
|
||||
if args.store:
|
||||
cmd += [ "--store", args.store ]
|
||||
cmd += [ args.image ]
|
||||
subprocess.check_call(cmd)
|
||||
|
||||
except testvm.Failure as ex:
|
||||
sys.stderr.write("image-create: %s\n" % ex)
|
||||
sys.exit(1)
|
|
@ -1,147 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
# This file is part of Cockpit.
|
||||
#
|
||||
# Copyright (C) 2015 Red Hat, Inc.
|
||||
#
|
||||
# Cockpit is free software; you can redistribute it and/or modify it
|
||||
# under the terms of the GNU Lesser General Public License as published by
|
||||
# the Free Software Foundation; either version 2.1 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# Cockpit is distributed in the hope that it will be useful, but
|
||||
# WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
# Lesser General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU Lesser General Public License
|
||||
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import subprocess
|
||||
|
||||
BOTS = os.path.abspath(os.path.dirname(__file__))
|
||||
BASE = os.path.normpath(os.path.join(BOTS, ".."))
|
||||
TEST = os.path.join(BASE, "test")
|
||||
os.environ["PATH"] = "{0}:{1}".format(os.environ.get("PATH"), BOTS)
|
||||
|
||||
from machine import testvm
|
||||
|
||||
parser = argparse.ArgumentParser(
|
||||
description='Run command inside or install packages into a Cockpit virtual machine',
|
||||
formatter_class=argparse.ArgumentDefaultsHelpFormatter
|
||||
)
|
||||
parser.add_argument('-v', '--verbose', action='store_true', help='Display verbose progress details')
|
||||
parser.add_argument('-i', '--install', action='append', dest="packagelist", default=[], help='Install packages')
|
||||
parser.add_argument('-I', '--install-command', action='store', dest="installcommand",
|
||||
default="yum --setopt=skip_missing_names_on_install=False -y install",
|
||||
help="Command used to install packages in machine")
|
||||
parser.add_argument('-r', '--run-command', action='append', dest="commandlist",
|
||||
default=[], help='Run command inside virtual machine')
|
||||
parser.add_argument('-s', '--script', action='append', dest="scriptlist",
|
||||
default=[], help='Run selected script inside virtual machine')
|
||||
parser.add_argument('-u', '--upload', action='append', dest="uploadlist",
|
||||
default=[], help='Upload file/dir to destination file/dir separated by ":" example: -u file.txt:/var/lib')
|
||||
parser.add_argument('--base-image', help='Base image name, if "image" does not match a standard Cockpit VM image name')
|
||||
parser.add_argument('--resize', help="Resize the image. Size in bytes with using K, M, or G suffix.")
|
||||
parser.add_argument('image', help='The image to use (destination name when using --base-image)')
|
||||
args = parser.parse_args()
|
||||
|
||||
if not args.base_image:
|
||||
args.base_image = os.path.basename(args.image)
|
||||
|
||||
args.base_image = testvm.get_test_image(args.base_image)
|
||||
|
||||
# Create the necessary layered image for the build/install
|
||||
def prepare_install_image(base_image, install_image):
|
||||
if "/" not in base_image:
|
||||
base_image = os.path.join(testvm.IMAGES_DIR, base_image)
|
||||
if "/" not in install_image:
|
||||
install_image = os.path.join(os.path.join(TEST, "images"), os.path.basename(install_image))
|
||||
|
||||
# In vm-customize we don't force recreate images
|
||||
if not os.path.exists(install_image):
|
||||
install_image_dir = os.path.dirname(install_image)
|
||||
os.makedirs(install_image_dir, exist_ok=True)
|
||||
base_image = os.path.realpath(base_image)
|
||||
qcow2_image = "{0}.qcow2".format(install_image)
|
||||
subprocess.check_call([ "qemu-img", "create", "-q", "-f", "qcow2",
|
||||
"-o", "backing_file={0},backing_fmt=qcow2".format(base_image), qcow2_image ])
|
||||
if os.path.lexists(install_image):
|
||||
os.unlink(install_image)
|
||||
os.symlink(os.path.basename(qcow2_image), install_image)
|
||||
|
||||
if args.resize:
|
||||
subprocess.check_call(["qemu-img", "resize", install_image, args.resize])
|
||||
|
||||
return install_image
|
||||
|
||||
def run_command(machine_instance, commandlist):
|
||||
"""Run command inside image"""
|
||||
for foo in commandlist:
|
||||
try:
|
||||
machine_instance.execute(foo, timeout=1800)
|
||||
except subprocess.CalledProcessError as e:
|
||||
sys.stderr.write("%s\n" % e)
|
||||
sys.exit(e.returncode)
|
||||
|
||||
def run_script(machine_instance, scriptlist):
|
||||
"""Run script inside image"""
|
||||
for foo in scriptlist:
|
||||
if os.path.isfile(foo):
|
||||
pname = os.path.basename(foo)
|
||||
uploadpath = "/var/tmp/" + pname
|
||||
machine_instance.upload([os.path.abspath(foo)], uploadpath)
|
||||
machine_instance.execute("chmod a+x %s" % uploadpath)
|
||||
try:
|
||||
machine_instance.execute(uploadpath, timeout=1800)
|
||||
except subprocess.CalledProcessError as e:
|
||||
sys.stderr.write("%s\n" % e)
|
||||
sys.exit(e.returncode)
|
||||
else:
|
||||
sys.stderr.write("Bad path to script: %s\n" % foo)
|
||||
|
||||
def upload_files(machine_instance, uploadfiles):
|
||||
"""Upload files/directories inside image"""
|
||||
for foo in uploadfiles:
|
||||
srcfile, dest = foo.split(":")
|
||||
src_absolute = os.path.join(os.getcwd(), srcfile)
|
||||
machine_instance.upload([src_absolute], dest)
|
||||
|
||||
def install_packages(machine_instance, packagelist, install_command):
|
||||
"""Install packages into a test image
|
||||
It could be done via local rpms or normal package installation
|
||||
"""
|
||||
allpackages = []
|
||||
for foo in packagelist:
|
||||
if os.path.isfile(foo):
|
||||
pname = os.path.basename(foo)
|
||||
machine_instance.upload([foo], "/var/tmp/" + pname)
|
||||
allpackages.append("/var/tmp/" + pname)
|
||||
elif not re.search("/", foo):
|
||||
allpackages.append(foo)
|
||||
else:
|
||||
sys.stderr.write("Bad package name or path: %s\n" % foo)
|
||||
if allpackages:
|
||||
machine_instance.execute(install_command + " " + ' '.join(allpackages), timeout=1800)
|
||||
|
||||
if args.commandlist or args.packagelist or args.scriptlist or args.uploadlist or args.resize:
|
||||
if '/' not in args.base_image:
|
||||
subprocess.check_call(["image-download", args.base_image])
|
||||
machine = testvm.VirtMachine(maintain=True,
|
||||
verbose=args.verbose, image=prepare_install_image(args.base_image, args.image))
|
||||
machine.start()
|
||||
machine.wait_boot()
|
||||
try:
|
||||
if args.uploadlist:
|
||||
upload_files(machine, args.uploadlist)
|
||||
if args.commandlist:
|
||||
run_command(machine, args.commandlist)
|
||||
if args.packagelist:
|
||||
install_packages(machine, args.packagelist, args.installcommand)
|
||||
if args.scriptlist:
|
||||
run_script(machine, args.scriptlist)
|
||||
finally:
|
||||
machine.stop()
|
|
@ -1,305 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
|
||||
# This file is part of Cockpit.
|
||||
#
|
||||
# Copyright (C) 2013 Red Hat, Inc.
|
||||
#
|
||||
# Cockpit is free software; you can redistribute it and/or modify it
|
||||
# under the terms of the GNU Lesser General Public License as published by
|
||||
# the Free Software Foundation; either version 2.1 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# Cockpit is distributed in the hope that it will be useful, but
|
||||
# WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
# Lesser General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU Lesser General Public License
|
||||
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
#
|
||||
# Download images or other state
|
||||
#
|
||||
# Images usually have a name specific link committed to git. These
|
||||
# are referred to as 'committed'
|
||||
#
|
||||
# Other state is simply referenced by name without a link in git
|
||||
# This is referred to as 'state'
|
||||
#
|
||||
# The stores are places to look for images or other state
|
||||
#
|
||||
|
||||
import argparse
|
||||
import email
|
||||
import io
|
||||
import os
|
||||
import shutil
|
||||
import socket
|
||||
import stat
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import fcntl
|
||||
import urllib.parse
|
||||
|
||||
from machine import testvm
|
||||
from task import REDHAT_STORE
|
||||
|
||||
CONFIG = "~/.config/image-stores"
|
||||
DEFAULT = [
|
||||
"http://cockpit-images.verify.svc.cluster.local",
|
||||
"https://images-cockpit.apps.ci.centos.org/",
|
||||
"https://209.132.184.41:8493/",
|
||||
REDHAT_STORE
|
||||
]
|
||||
|
||||
BOTS = os.path.dirname(os.path.realpath(__file__))
|
||||
|
||||
DEVNULL = open("/dev/null", "r+")
|
||||
EPOCH = "Thu, 1 Jan 1970 00:00:00 GMT"
|
||||
|
||||
def find(name, stores, latest, quiet):
|
||||
found = [ ]
|
||||
ca = os.path.join(testvm.IMAGES_DIR, "files", "ca.pem")
|
||||
|
||||
for store in stores:
|
||||
url = urllib.parse.urlparse(store)
|
||||
|
||||
defport = url.scheme == 'http' and 80 or 443
|
||||
|
||||
try:
|
||||
ai = socket.getaddrinfo(url.hostname, url.port or defport, socket.AF_INET, 0, socket.IPPROTO_TCP)
|
||||
except socket.gaierror:
|
||||
ai = [ ]
|
||||
message = store
|
||||
|
||||
for (family, socktype, proto, canonname, sockaddr) in ai:
|
||||
message = "{scheme}://{0}:{1}{path}".format(*sockaddr, scheme=url.scheme, path=url.path)
|
||||
|
||||
def curl(*args):
|
||||
try:
|
||||
cmd = ["curl"] + list(args) + ["--head", "--silent", "--fail", "--cacert", ca, source]
|
||||
start = time.time()
|
||||
output = subprocess.check_output(cmd, universal_newlines=True)
|
||||
found.append((cmd, output, message, time.time() - start))
|
||||
if not quiet:
|
||||
sys.stderr.write(" > {0}\n".format(message))
|
||||
return True
|
||||
except subprocess.CalledProcessError:
|
||||
return False
|
||||
|
||||
# first try with stores that accept the "cockpit-tests" host name
|
||||
resolve = "cockpit-tests:{1}:{0}".format(*sockaddr)
|
||||
source = urllib.parse.urljoin("{0}://cockpit-tests:{1}{2}".format(url.scheme, sockaddr[1], url.path), name)
|
||||
if curl("--resolve", resolve):
|
||||
continue
|
||||
|
||||
# fall back for OpenShift proxied stores which send their own SSL cert initially; host name has to match that
|
||||
source = urllib.parse.urljoin(store, name)
|
||||
if curl():
|
||||
continue
|
||||
|
||||
if not quiet:
|
||||
sys.stderr.write(" x {0}\n".format(message))
|
||||
|
||||
# If we couldn't find the file, but it exists, we're good
|
||||
if not found:
|
||||
return None, None
|
||||
|
||||
# Find the most recent version of this file
|
||||
def header_date(args):
|
||||
cmd, output, message, latency = args
|
||||
try:
|
||||
reply_line, headers_alone = output.split('\n', 1)
|
||||
last_modified = email.message_from_file(io.StringIO(headers_alone)).get("Last-Modified", "")
|
||||
return time.mktime(time.strptime(last_modified, '%a, %d %b %Y %H:%M:%S %Z'))
|
||||
except ValueError:
|
||||
return ""
|
||||
|
||||
if latest:
|
||||
found.sort(reverse=True, key=header_date)
|
||||
else:
|
||||
found.sort(reverse=False, key=lambda x: x[3])
|
||||
|
||||
# Return the command and message
|
||||
return found[0][0], found[0][2]
|
||||
|
||||
def download(dest, force, state, quiet, stores):
|
||||
if not stores:
|
||||
config = os.path.expanduser(CONFIG)
|
||||
if os.path.exists(config):
|
||||
with open(config, 'r') as fp:
|
||||
stores = fp.read().strip().split("\n")
|
||||
else:
|
||||
stores = []
|
||||
stores += DEFAULT
|
||||
|
||||
# The time condition for If-Modified-Since
|
||||
exists = not force and os.path.exists(dest)
|
||||
if exists:
|
||||
since = dest
|
||||
else:
|
||||
since = EPOCH
|
||||
|
||||
name = os.path.basename(dest)
|
||||
cmd, message = find(name, stores, latest=state, quiet=quiet)
|
||||
|
||||
# If we couldn't find the file, but it exists, we're good
|
||||
if not cmd:
|
||||
if exists:
|
||||
return
|
||||
raise RuntimeError("image-download: couldn't find file anywhere: {0}".format(name))
|
||||
|
||||
# Choose the first found item after sorting by date
|
||||
if not quiet:
|
||||
sys.stderr.write(" > {0}\n".format(urllib.parse.urljoin(message, name)))
|
||||
|
||||
temp = dest + ".partial"
|
||||
|
||||
# Adjust the command above that worked to make it visible and download real stuff
|
||||
cmd.remove("--head")
|
||||
cmd.append("--show-error")
|
||||
if not quiet and os.isatty(sys.stdout.fileno()):
|
||||
cmd.remove("--silent")
|
||||
cmd.insert(1, "--progress-bar")
|
||||
cmd.append("--remote-time")
|
||||
cmd.append("--time-cond")
|
||||
cmd.append(since)
|
||||
cmd.append("--output")
|
||||
cmd.append(temp)
|
||||
if os.path.exists(temp):
|
||||
if force:
|
||||
os.remove(temp)
|
||||
else:
|
||||
cmd.append("-C")
|
||||
cmd.append("-")
|
||||
|
||||
# Always create the destination file (because --state)
|
||||
else:
|
||||
open(temp, 'a').close()
|
||||
|
||||
curl = subprocess.Popen(cmd)
|
||||
ret = curl.wait()
|
||||
if ret != 0:
|
||||
raise RuntimeError("curl: unable to download %s (returned: %s)" % (message, ret))
|
||||
|
||||
os.chmod(temp, stat.S_IRUSR | stat.S_IRGRP | stat.S_IROTH)
|
||||
|
||||
# Due to time-cond the file size may be zero
|
||||
# A new file downloaded, put it in place
|
||||
if not exists or os.path.getsize(temp) > 0:
|
||||
shutil.move(temp, dest)
|
||||
|
||||
# Calculate a place to put images where links are not committed in git
|
||||
def state_target(path):
|
||||
data_dir = testvm.get_images_data_dir()
|
||||
os.makedirs(data_dir, mode=0o775, exist_ok=True)
|
||||
return os.path.join(data_dir, path)
|
||||
|
||||
# Calculate a place to put images where links are committed in git
|
||||
def committed_target(image):
|
||||
link = os.path.join(testvm.IMAGES_DIR, image)
|
||||
if not os.path.islink(link):
|
||||
raise RuntimeError("image link does not exist: " + image)
|
||||
|
||||
dest = os.readlink(link)
|
||||
relative_dir = os.path.dirname(os.path.abspath(link))
|
||||
full_dest = os.path.join(relative_dir, dest)
|
||||
while os.path.islink(full_dest):
|
||||
link = full_dest
|
||||
dest = os.readlink(link)
|
||||
relative_dir = os.path.dirname(os.path.abspath(link))
|
||||
full_dest = os.path.join(relative_dir, dest)
|
||||
|
||||
dest = os.path.join(testvm.get_images_data_dir(), dest)
|
||||
|
||||
# We have the file but there is not valid link
|
||||
if os.path.exists(dest):
|
||||
try:
|
||||
os.symlink(dest, os.path.join(testvm.IMAGES_DIR, os.readlink(link)))
|
||||
except FileExistsError:
|
||||
pass
|
||||
|
||||
# The image file in the images directory, may be same as dest
|
||||
image_file = os.path.join(testvm.IMAGES_DIR, os.readlink(link))
|
||||
|
||||
# Double check that symlink in place but never make a cycle.
|
||||
if os.path.abspath(dest) != os.path.abspath(image_file):
|
||||
try:
|
||||
os.symlink(os.path.abspath(dest), image_file)
|
||||
except FileExistsError:
|
||||
pass
|
||||
|
||||
return dest
|
||||
|
||||
def wait_lock(target):
|
||||
lockfile = os.path.join(tempfile.gettempdir(), ".cockpit-test-resources", os.path.basename(target) + ".lock")
|
||||
os.makedirs(os.path.dirname(lockfile), exist_ok=True)
|
||||
|
||||
# we need to keep the lock fd open throughout the entire runtime, so remember it in a global-scoped variable
|
||||
wait_lock.f = open(lockfile, "w")
|
||||
for retry in range(360):
|
||||
try:
|
||||
fcntl.flock(wait_lock.f, fcntl.LOCK_NB | fcntl.LOCK_EX)
|
||||
return
|
||||
except BlockingIOError:
|
||||
if retry == 0:
|
||||
print("Waiting for concurrent image-download of %s..." % os.path.basename(target))
|
||||
time.sleep(10)
|
||||
else:
|
||||
raise TimeoutError("timed out waiting for concurrent downloads of %s\n" % target)
|
||||
|
||||
def download_images(image_list, force, quiet, state, store):
|
||||
data_dir = testvm.get_images_data_dir()
|
||||
os.makedirs(data_dir, exist_ok=True)
|
||||
|
||||
# A default set of images are all links in git. These links have
|
||||
# no directory part. Other links might exist, such as the
|
||||
# auxiliary links created by committed_target above, and we ignore
|
||||
# them.
|
||||
if not image_list:
|
||||
image_list = []
|
||||
if not state:
|
||||
for filename in os.listdir(testvm.IMAGES_DIR):
|
||||
link = os.path.join(testvm.IMAGES_DIR, filename)
|
||||
if os.path.islink(link) and os.path.dirname(os.readlink(link)) == "":
|
||||
image_list.append(filename)
|
||||
|
||||
success = True
|
||||
|
||||
for image in image_list:
|
||||
image = testvm.get_test_image(image)
|
||||
try:
|
||||
if state:
|
||||
target = state_target(image)
|
||||
else:
|
||||
target = committed_target(image)
|
||||
|
||||
# don't download the same thing multiple times in parallel
|
||||
wait_lock(target)
|
||||
|
||||
if force or state or not os.path.exists(target):
|
||||
download(target, force, state, quiet, store)
|
||||
except Exception as ex:
|
||||
success = False
|
||||
sys.stderr.write("image-download: {0}\n".format(str(ex)))
|
||||
|
||||
return success
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description='Download a bot state or images')
|
||||
parser.add_argument("--force", action="store_true", help="Force unnecessary downloads")
|
||||
parser.add_argument("--store", action="append", help="Where to find state or images")
|
||||
parser.add_argument("--quiet", action="store_true", help="Make downloading quieter")
|
||||
parser.add_argument("--state", action="store_true", help="Images or state not recorded in git")
|
||||
parser.add_argument('image', nargs='*')
|
||||
args = parser.parse_args()
|
||||
|
||||
if not download_images(args.image, args.force, args.quiet, args.state, args.store):
|
||||
return 1
|
||||
|
||||
return 0
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
218
bots/image-prune
218
bots/image-prune
|
@ -1,218 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
|
||||
# This file is part of Cockpit.
|
||||
#
|
||||
# Copyright (C) 2013 Red Hat, Inc.
|
||||
#
|
||||
# Cockpit is free software; you can redistribute it and/or modify it
|
||||
# under the terms of the GNU Lesser General Public License as published by
|
||||
# the Free Software Foundation; either version 2.1 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# Cockpit is distributed in the hope that it will be useful, but
|
||||
# WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
# Lesser General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU Lesser General Public License
|
||||
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
# Days after which images expire if not in use
|
||||
IMAGE_EXPIRE = 14
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
from contextlib import contextmanager
|
||||
|
||||
from task import github
|
||||
|
||||
from machine import testvm
|
||||
|
||||
BOTS = os.path.dirname(os.path.realpath(__file__))
|
||||
|
||||
# threshold in G below which unreferenced qcow2 images will be pruned, even if they aren't old
|
||||
PRUNE_THRESHOLD_G = float(os.environ.get("PRUNE_THRESHOLD_G", 15))
|
||||
|
||||
def enough_disk_space():
|
||||
"""Check if available disk space in our data store is sufficient
|
||||
"""
|
||||
st = os.statvfs(testvm.get_images_data_dir())
|
||||
free = st.f_bavail * st.f_frsize / (1024*1024*1024)
|
||||
return free >= PRUNE_THRESHOLD_G;
|
||||
|
||||
def get_refs(open_pull_requests=True, offline=False):
|
||||
"""Return dictionary for available refs of the format {'rhel-7.4': 'ad50328990e44c22501bd5e454746d4b5e561b7c'}
|
||||
|
||||
Expects to be called from the top level of the git checkout
|
||||
If offline is true, git show-ref is used instead of listing the remote
|
||||
"""
|
||||
# get all remote heads and filter empty lines
|
||||
# output of ls-remote has the format
|
||||
#
|
||||
# d864d3792db442e3de3d1811fa4bc371793a8f4f refs/heads/master
|
||||
# ad50328990e44c22501bd5e454746d4b5e561b7c refs/heads/rhel-7.4
|
||||
|
||||
refs = { }
|
||||
|
||||
considerable = {}
|
||||
g = github.GitHub()
|
||||
if open_pull_requests:
|
||||
if offline:
|
||||
raise Exception("Unable to consider open pull requests when in offline mode")
|
||||
for p in g.pulls():
|
||||
files = g.get("pulls/{0}/files".format(p["number"]))
|
||||
images = []
|
||||
for fl in files:
|
||||
fl_name = fl['filename']
|
||||
if fl_name.startswith("bots/images/"):
|
||||
fl_name_split = fl_name.split("/", 2)
|
||||
if "/" not in fl_name_split[2]:
|
||||
images.append(fl_name_split[2])
|
||||
if images:
|
||||
sha = p["head"]["sha"]
|
||||
considerable[sha] = images
|
||||
subprocess.call(["git", "fetch", "origin", "pull/{0}/head".format(p["number"])])
|
||||
refs["pull request #{} ({})".format(p["number"], p["title"])] = sha
|
||||
|
||||
git_cmd = "show-ref" if offline else "ls-remote"
|
||||
ref_output = subprocess.check_output(["git", git_cmd], universal_newlines=True).splitlines()
|
||||
# filter out the "refs/heads/" prefix and generate a dictionary
|
||||
prefix = "refs/heads"
|
||||
for ln in ref_output:
|
||||
[ref, name] = ln.split()
|
||||
if name.startswith(prefix):
|
||||
refs[name[len(prefix):]] = ref
|
||||
|
||||
return (refs, considerable)
|
||||
|
||||
def get_image_links(ref, git_path):
|
||||
"""Return all image links for the given git ref
|
||||
|
||||
Expects to be called from the top level of the git checkout
|
||||
"""
|
||||
# get all the links we have first
|
||||
# trailing slash on path is important
|
||||
if not git_path.endswith("/"):
|
||||
git_path = "{0}/".format(git_path)
|
||||
|
||||
try:
|
||||
entries = subprocess.check_output(["git", "ls-tree", "--name-only", ref, git_path], universal_newlines=True).splitlines()
|
||||
except subprocess.CalledProcessError as e:
|
||||
if e.returncode == 128:
|
||||
sys.stderr.write("Skipping {0} due to tree error.\n".format(ref))
|
||||
return []
|
||||
raise
|
||||
links = [subprocess.check_output(["git", "show", "{0}:{1}".format(ref, entry)], universal_newlines=True) for entry in entries]
|
||||
return [link for link in links if link.endswith(".qcow2")]
|
||||
|
||||
@contextmanager
|
||||
def remember_cwd():
|
||||
curdir = os.getcwd()
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
os.chdir(curdir)
|
||||
|
||||
def get_image_names(quiet=False, open_pull_requests=True, offline=False):
|
||||
"""Return all image names used by all branches and optionally in open pull requests
|
||||
"""
|
||||
images = set()
|
||||
# iterate over visible refs (mostly branches)
|
||||
# this hinges on being in the top level directory of the the git checkout
|
||||
with remember_cwd():
|
||||
os.chdir(os.path.join(BOTS, ".."))
|
||||
(refs, considerable) = get_refs(open_pull_requests, offline)
|
||||
# list images present in each branch / pull request
|
||||
for name, ref in refs.items():
|
||||
if not quiet:
|
||||
sys.stderr.write("Considering images from {0} ({1})\n".format(name, ref))
|
||||
for link in get_image_links(ref, "bots/images"):
|
||||
if ref in considerable:
|
||||
for consider in considerable[ref]:
|
||||
if link.startswith(consider):
|
||||
images.add(link)
|
||||
else:
|
||||
images.add(link)
|
||||
|
||||
return images
|
||||
|
||||
def prune_images(force, dryrun, quiet=False, open_pull_requests=True, offline=False, checkout_only=False):
|
||||
"""Prune images
|
||||
"""
|
||||
now = time.time()
|
||||
|
||||
# everything we want to keep
|
||||
if checkout_only:
|
||||
targets = set()
|
||||
else:
|
||||
targets = get_image_names(quiet, open_pull_requests, offline)
|
||||
|
||||
# what we have in the current checkout might already have been added by its branch, but check anyway
|
||||
for filename in os.listdir(testvm.IMAGES_DIR):
|
||||
path = os.path.join(testvm.IMAGES_DIR, filename)
|
||||
|
||||
# only consider original image entries as trustworthy sources and ignore non-links
|
||||
if path.endswith(".qcow2") or path.endswith(".partial") or not os.path.islink(path):
|
||||
continue
|
||||
|
||||
target = os.readlink(path)
|
||||
targets.add(target)
|
||||
|
||||
expiry_threshold = now - IMAGE_EXPIRE * 86400
|
||||
for filename in os.listdir(testvm.get_images_data_dir()):
|
||||
path = os.path.join(testvm.get_images_data_dir(), filename)
|
||||
if not force and (enough_disk_space() and os.lstat(path).st_mtime > expiry_threshold):
|
||||
continue
|
||||
if os.path.isfile(path) and (path.endswith(".xz") or path.endswith(".qcow2") or path.endswith(".partial")) and filename not in targets:
|
||||
if not quiet or dryrun:
|
||||
sys.stderr.write("Pruning {0}\n".format(filename))
|
||||
if not dryrun:
|
||||
os.unlink(path)
|
||||
|
||||
# now prune broken links
|
||||
for filename in os.listdir(testvm.IMAGES_DIR):
|
||||
path = os.path.join(testvm.IMAGES_DIR, filename)
|
||||
|
||||
# don't prune original image entries and ignore non-links
|
||||
if not path.endswith(".qcow2") or not os.path.islink(path):
|
||||
continue
|
||||
|
||||
# if the link isn't valid, prune
|
||||
if not os.path.isfile(path):
|
||||
if not quiet or dryrun:
|
||||
sys.stderr.write("Pruning link {0}\n".format(path))
|
||||
if not dryrun:
|
||||
os.unlink(path)
|
||||
|
||||
def every_image():
|
||||
result = []
|
||||
for filename in os.listdir(testvm.IMAGES_DIR):
|
||||
link = os.path.join(testvm.IMAGES_DIR, filename)
|
||||
if os.path.islink(link):
|
||||
result.append(filename)
|
||||
return result
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description='Prune downloaded images')
|
||||
parser.add_argument("--force", action="store_true", help="Delete images even if they aren't old")
|
||||
parser.add_argument("--quiet", action="store_true", help="Make downloading quieter")
|
||||
parser.add_argument("-d", "--dry-run-prune", dest="dryrun", action="store_true", help="Don't actually delete images and links")
|
||||
parser.add_argument("-b", "--branches-only", dest="branches_only", action="store_true", help="Don't consider pull requests on GitHub, only look at branches")
|
||||
parser.add_argument("-c", "--checkout-only", dest="checkout_only", action="store_true", help="Consider neither pull requests on GitHub nor branches, only look at the current checkout")
|
||||
parser.add_argument("-o", "--offline", dest="offline", action="store_true", help="Don't access external sources such as GitHub")
|
||||
args = parser.parse_args()
|
||||
|
||||
try:
|
||||
prune_images(args.force, args.dryrun, quiet=args.quiet, open_pull_requests=(not args.branches_only), offline=args.offline, checkout_only=args.checkout_only)
|
||||
except RuntimeError as ex:
|
||||
sys.stderr.write("image-prune: {0}\n".format(str(ex)))
|
||||
return 1
|
||||
|
||||
return 0
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
|
@ -1,169 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
|
||||
# This file is part of Cockpit.
|
||||
#
|
||||
# Copyright (C) 2016 Red Hat, Inc.
|
||||
#
|
||||
# Cockpit is free software; you can redistribute it and/or modify it
|
||||
# under the terms of the GNU Lesser General Public License as published by
|
||||
# the Free Software Foundation; either version 2.1 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# Cockpit is distributed in the hope that it will be useful, but
|
||||
# WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
# Lesser General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU Lesser General Public License
|
||||
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
import task
|
||||
from task import github, REDHAT_STORE
|
||||
|
||||
TRIGGERS = {
|
||||
"centos-7": [
|
||||
"centos-7@cockpit-project/starter-kit",
|
||||
],
|
||||
"continuous-atomic": [
|
||||
"continuous-atomic@cockpit-project/cockpit-ostree",
|
||||
"continuous-atomic@cockpit-project/cockpit/rhel-7.7",
|
||||
],
|
||||
"debian-testing": [
|
||||
"debian-testing"
|
||||
],
|
||||
"debian-stable": [
|
||||
"debian-stable"
|
||||
],
|
||||
"fedora-29": [
|
||||
"fedora-atomic",
|
||||
"fedora-29@cockpit-project/cockpit-podman",
|
||||
],
|
||||
"fedora-30": [
|
||||
"fedora-30",
|
||||
"fedora-30/selenium-chrome",
|
||||
"fedora-30/selenium-firefox",
|
||||
"fedora-30/selenium-edge",
|
||||
"fedora-30/container-bastion",
|
||||
"fedora-30@cockpit-project/starter-kit",
|
||||
"fedora-30@cockpit-project/cockpit-podman",
|
||||
"fedora-30@weldr/lorax",
|
||||
"fedora-30/live-iso@weldr/lorax",
|
||||
"fedora-30/qcow2@weldr/lorax",
|
||||
"fedora-30/chrome@weldr/cockpit-composer",
|
||||
"fedora-30/firefox@weldr/cockpit-composer",
|
||||
"fedora-30/edge@weldr/cockpit-composer",
|
||||
],
|
||||
"fedora-31": [
|
||||
"fedora-31",
|
||||
"fedora-31@cockpit-project/cockpit-podman",
|
||||
],
|
||||
"fedora-atomic": [
|
||||
"fedora-atomic",
|
||||
"fedora-atomic@cockpit-project/cockpit-ostree",
|
||||
],
|
||||
"fedora-testing": [
|
||||
"fedora-testing"
|
||||
],
|
||||
"fedora-i386": [
|
||||
"fedora-i386"
|
||||
],
|
||||
"ubuntu-1804": [
|
||||
"ubuntu-1804"
|
||||
],
|
||||
"ubuntu-stable": [
|
||||
"ubuntu-stable"
|
||||
],
|
||||
"openshift": [
|
||||
"rhel-7-7@cockpit-project/cockpit/rhel-7.7",
|
||||
"rhel-7-8@cockpit-project/cockpit/rhel-7.8",
|
||||
],
|
||||
"ipa": [
|
||||
"fedora-30",
|
||||
"ubuntu-1804",
|
||||
"debian-stable"
|
||||
],
|
||||
"selenium": [
|
||||
"fedora-30/selenium-chrome",
|
||||
"fedora-30/selenium-firefox",
|
||||
"fedora-30/chrome@weldr/cockpit-composer",
|
||||
"fedora-30/firefox@weldr/cockpit-composer",
|
||||
"rhel-7-7/firefox@weldr/cockpit-composer",
|
||||
"rhel-8-1/chrome@weldr/cockpit-composer",
|
||||
],
|
||||
"rhel-7-7": [
|
||||
"rhel-7-7/firefox@weldr/cockpit-composer",
|
||||
"rhel-7-7@cockpit-project/cockpit/rhel-7.7",
|
||||
"rhel-atomic@cockpit-project/cockpit/rhel-7.7",
|
||||
"continuous-atomic@cockpit-project/cockpit/rhel-7.7",
|
||||
],
|
||||
"rhel-7-8": [
|
||||
"rhel-7-8@cockpit-project/cockpit/rhel-7.8",
|
||||
],
|
||||
"rhel-8-0": [
|
||||
"rhel-8-0@cockpit-project/cockpit/rhel-8.0",
|
||||
],
|
||||
"rhel-8-1": [
|
||||
"rhel-8-1",
|
||||
"rhel-8-1-distropkg",
|
||||
"rhel-8-1@cockpit-project/cockpit/rhel-8.1",
|
||||
"rhel-8-1@cockpit-project/cockpit/rhel-8-appstream",
|
||||
"rhel-8-1/chrome@weldr/cockpit-composer",
|
||||
"rhel-8-1@cockpit-project/cockpit-podman",
|
||||
],
|
||||
"rhel-atomic": [
|
||||
"rhel-atomic@cockpit-project/cockpit-ostree",
|
||||
"rhel-atomic@cockpit-project/cockpit/rhel-7.7",
|
||||
]
|
||||
}
|
||||
|
||||
STORES = {
|
||||
"rhel-7-7": REDHAT_STORE,
|
||||
"rhel-7-8": REDHAT_STORE,
|
||||
"rhel-8-0": REDHAT_STORE,
|
||||
"rhel-8-1": REDHAT_STORE,
|
||||
"rhel-atomic": REDHAT_STORE,
|
||||
"windows-10": REDHAT_STORE,
|
||||
}
|
||||
|
||||
BOTS = os.path.abspath(os.path.dirname(__file__))
|
||||
BASE = os.path.normpath(os.path.join(BOTS, ".."))
|
||||
|
||||
sys.dont_write_bytecode = True
|
||||
|
||||
def run(image, verbose=False, **kwargs):
|
||||
if not image:
|
||||
raise RuntimeError("no image specified")
|
||||
|
||||
triggers = TRIGGERS.get(image, [ ])
|
||||
store = STORES.get(image, None)
|
||||
|
||||
# Cleanup any extraneous disk usage elsewhere
|
||||
subprocess.check_call([ os.path.join(BOTS, "vm-reset") ])
|
||||
|
||||
cmd = [ os.path.join(BOTS, "image-create"), "--verbose", "--upload" ]
|
||||
if store:
|
||||
cmd += [ "--store", store ]
|
||||
cmd += [ image ]
|
||||
|
||||
os.environ['VIRT_BUILDER_NO_CACHE'] = "yes"
|
||||
ret = subprocess.call(cmd)
|
||||
if ret:
|
||||
return ret
|
||||
|
||||
branch = task.branch(image, "images: Update {0} image".format(image), pathspec="bots/images", **kwargs)
|
||||
if branch:
|
||||
pull = task.pull(branch, labels=['bot', 'no-test'], run_tests=False, **kwargs)
|
||||
|
||||
# Trigger this pull request
|
||||
api = github.GitHub()
|
||||
head = pull["head"]["sha"]
|
||||
for trigger in triggers:
|
||||
api.post("statuses/{0}".format(head), { "state": "pending", "context": trigger,
|
||||
"description": github.NOT_TESTED_DIRECT })
|
||||
|
||||
if __name__ == '__main__':
|
||||
task.main(function=run, title="Refresh image")
|
|
@ -1,111 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
|
||||
# This file is part of Cockpit.
|
||||
#
|
||||
# Copyright (C) 2015 Red Hat, Inc.
|
||||
#
|
||||
# Cockpit is free software; you can redistribute it and/or modify it
|
||||
# under the terms of the GNU Lesser General Public License as published by
|
||||
# the Free Software Foundation; either version 2.1 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# Cockpit is distributed in the hope that it will be useful, but
|
||||
# WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
# Lesser General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU Lesser General Public License
|
||||
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
DAYS = 7
|
||||
|
||||
REFRESH = {
|
||||
"candlepin": { "refresh-days": 120 },
|
||||
"centos-7": { },
|
||||
"continuous-atomic": { },
|
||||
"debian-testing": { },
|
||||
"debian-stable": { },
|
||||
"fedora-29": { },
|
||||
"fedora-30": { },
|
||||
"fedora-31": { },
|
||||
"fedora-atomic": { },
|
||||
"fedora-testing": { },
|
||||
"fedora-i386": { },
|
||||
"ipa": { "refresh-days": 120 },
|
||||
"ubuntu-1804": { },
|
||||
"ubuntu-stable": { },
|
||||
"openshift": { "refresh-days": 30 },
|
||||
'rhel-7-7': { },
|
||||
'rhel-8-0': { },
|
||||
'rhel-8-1': { },
|
||||
'rhel-atomic': { },
|
||||
"selenium": { "refresh-days": 30 },
|
||||
}
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import subprocess
|
||||
|
||||
sys.dont_write_bytecode = True
|
||||
|
||||
import task
|
||||
from task import github
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description='Ensure necessary issue exists for image refresh')
|
||||
parser.add_argument('-v', '--verbose', action="store_true", default=False,
|
||||
help="Print verbose information")
|
||||
parser.add_argument("image", nargs="?")
|
||||
opts = parser.parse_args()
|
||||
api = github.GitHub()
|
||||
|
||||
try:
|
||||
results = scan(api, opts.image, opts.verbose)
|
||||
except RuntimeError as ex:
|
||||
sys.stderr.write("image-trigger: " + str(ex) + "\n")
|
||||
return 1
|
||||
|
||||
for result in results:
|
||||
if result:
|
||||
sys.stdout.write(result + "\n")
|
||||
|
||||
return 0
|
||||
|
||||
# Prepare an image prune command
|
||||
def scan_for_prune():
|
||||
tasks = [ ]
|
||||
stamp = os.path.join(tempfile.gettempdir(), "cockpit-image-prune.stamp")
|
||||
|
||||
# Don't prune more than once per hour
|
||||
try:
|
||||
mtime = os.stat(stamp).st_mtime
|
||||
except OSError:
|
||||
mtime = 0
|
||||
if mtime < time.time() - 3600:
|
||||
tasks.append("PRIORITY=0000 touch {0} && bots/image-prune".format(stamp))
|
||||
|
||||
return tasks
|
||||
|
||||
def scan(api, force, verbose):
|
||||
subprocess.check_call([ "git", "fetch", "origin", "master" ])
|
||||
for (image, options) in REFRESH.items():
|
||||
perform = False
|
||||
|
||||
if force:
|
||||
perform = image == force
|
||||
else:
|
||||
days = options.get("refresh-days", DAYS)
|
||||
perform = task.stale(days, os.path.join("bots", "images", image), "origin/master")
|
||||
|
||||
if perform:
|
||||
text = "Image refresh for {0}".format(image)
|
||||
issue = task.issue(text, text, "image-refresh", image)
|
||||
sys.stderr.write("#{0}: image-refresh {1}\n".format(issue["number"], image))
|
||||
|
||||
return scan_for_prune()
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
|
@ -1,120 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
|
||||
# This file is part of Cockpit.
|
||||
#
|
||||
# Copyright (C) 2013 Red Hat, Inc.
|
||||
#
|
||||
# Cockpit is free software; you can redistribute it and/or modify it
|
||||
# under the terms of the GNU Lesser General Public License as published by
|
||||
# the Free Software Foundation; either version 2.1 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# Cockpit is distributed in the hope that it will be useful, but
|
||||
# WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
# Lesser General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU Lesser General Public License
|
||||
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
|
||||
# The default settings here should match one of the default download stores
|
||||
DEFAULT_UPLOAD = [
|
||||
"https://images-cockpit.apps.ci.centos.org/",
|
||||
"https://209.132.184.41:8493/",
|
||||
]
|
||||
|
||||
TOKEN = "~/.config/github-token"
|
||||
|
||||
import argparse
|
||||
import getpass
|
||||
import errno
|
||||
import os
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import urllib.parse
|
||||
|
||||
from machine import testvm
|
||||
|
||||
BOTS = os.path.dirname(__file__)
|
||||
|
||||
def upload(store, source):
|
||||
ca = os.path.join(BOTS, "images", "files", "ca.pem")
|
||||
url = urllib.parse.urlparse(store)
|
||||
|
||||
# Start building the command
|
||||
cmd = ["curl", "--progress-bar", "--cacert", ca, "--fail", "--upload-file", source ]
|
||||
|
||||
def try_curl(cmd):
|
||||
print("Uploading to", cmd[-1])
|
||||
# Passing through a non terminal stdout is necessary to make progress work
|
||||
curl = subprocess.Popen(cmd, stdout=subprocess.PIPE)
|
||||
cat = subprocess.Popen(["cat"], stdin=curl.stdout)
|
||||
curl.stdout.close()
|
||||
ret = curl.wait()
|
||||
cat.wait()
|
||||
if ret != 0:
|
||||
sys.stderr.write("image-upload: unable to upload image: {0}\n".format(cmd[-1]))
|
||||
return ret
|
||||
|
||||
# Parse the user name and token, if present
|
||||
user = url.username or getpass.getuser()
|
||||
try:
|
||||
with open(os.path.expanduser(TOKEN), "r") as gt:
|
||||
token = gt.read().strip()
|
||||
cmd += [ "--user", user + ":" + token ]
|
||||
except IOError as exc:
|
||||
if exc.errno == errno.ENOENT:
|
||||
pass
|
||||
|
||||
# First try to use the original store URL, for stores with valid SSL cert on an OpenShift proxy
|
||||
if try_curl(cmd + [store]) == 0:
|
||||
return 0
|
||||
|
||||
# Fall back for stores that use our self-signed cockpit certificate
|
||||
# Parse out the actual address to connect to and override certificate info
|
||||
defport = url.scheme == 'http' and 80 or 443
|
||||
ai = socket.getaddrinfo(url.hostname, url.port or defport, socket.AF_INET, 0, socket.IPPROTO_TCP)
|
||||
for (family, socktype, proto, canonname, sockaddr) in ai:
|
||||
resolve = "cockpit-tests:{1}:{0}".format(*sockaddr)
|
||||
curl_url = "https://cockpit-tests:{0}{1}".format(url.port or defport, url.path)
|
||||
ret = try_curl(cmd + ["--resolve", resolve, curl_url])
|
||||
if ret == 0:
|
||||
return 0
|
||||
|
||||
return 1
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description='Upload bot state or images')
|
||||
parser.add_argument("--store", action="append", default=[], help="Where to send state or images")
|
||||
parser.add_argument("--state", action="store_true", help="Images or state not recorded in git")
|
||||
parser.add_argument('image', nargs='*')
|
||||
args = parser.parse_args()
|
||||
|
||||
data_dir = testvm.get_images_data_dir()
|
||||
sources = []
|
||||
for image in args.image:
|
||||
if args.state:
|
||||
source = os.path.join(data_dir, image)
|
||||
else:
|
||||
link = os.path.join(testvm.IMAGES_DIR, image)
|
||||
if not os.path.islink(link):
|
||||
parser.error("image link does not exist: " + image)
|
||||
source = os.path.join(data_dir, os.readlink(link))
|
||||
if not os.path.isfile(source):
|
||||
parser.error("image does not exist: " + image)
|
||||
sources.append(source)
|
||||
|
||||
for source in sources:
|
||||
for store in (args.store or DEFAULT_UPLOAD):
|
||||
ret = upload(store, source)
|
||||
if ret == 0:
|
||||
return ret
|
||||
else:
|
||||
# all stores failed, so return last exit code
|
||||
return ret
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
|
@ -1 +0,0 @@
|
|||
candlepin-3a39cecb7d2fea2e75b0093a891b3c476141406e20f332cb2a12f2dfb6e9d275.qcow2
|
|
@ -1 +0,0 @@
|
|||
centos-7-b12881afba5b51520073d9633295a89121e4f52b2e8aee4e2c422a95064a902f.qcow2
|
|
@ -1 +0,0 @@
|
|||
cirros-d5fcb44e05f2dafc7eaab6bce906ba9cc06af51f84f1e7a527fe12102e34bbcf.qcow2
|
|
@ -1 +0,0 @@
|
|||
continuous-atomic-a4c3407c689e53e6864f7ba92b95a0d7eea42b04545a7aeb77271b6f5521bd08.qcow2
|
|
@ -1 +0,0 @@
|
|||
debian-stable-980197e60a14278239e4c06b39421e9bd16d9ad97e1d4bb7d405b8fb2a8b15a0.qcow2
|
|
@ -1 +0,0 @@
|
|||
debian-testing-da77a67318002005f72e5f988978198fe6a53bea4bd60f2727a4123e98d6dd26.qcow2
|
|
@ -1 +0,0 @@
|
|||
fedora-23-stock-1a7ce615dcf1772ff6514148513fc88e420b9179f32c5395e3a27dab3b107dcc.qcow2
|
|
@ -1 +0,0 @@
|
|||
fedora-29-7191db4290794ba7bcf8eb30739d07a6cf59b072513b813b97755adb69162a95.qcow2
|
|
@ -1 +0,0 @@
|
|||
fedora-30-219724b87c59cbb21378c3f1e5fe80e8963072dc0fa97c15c809cf19118f8433.qcow2
|
|
@ -1 +0,0 @@
|
|||
fedora-31-632704fccac608d265572c0500eeecae3d61e0d932683adb5b5092980032b3ef.qcow2
|
|
@ -1 +0,0 @@
|
|||
fedora-atomic-6a63990b2443f568bb3321efe55fe8bad8891d128ec4c5b818303cd52a34e1e0.qcow2
|
|
@ -1 +0,0 @@
|
|||
fedora-i386-f5c6c9730facd6b7d00d5c07f59cf7bf3a9ce3de1270f174cf5d9aefcd86a297.qcow2
|
|
@ -1 +0,0 @@
|
|||
stock-fedora-22-x86_64-2.qcow2
|
|
@ -1 +0,0 @@
|
|||
fedora-testing-fedf1d06768b7cb69efbb2ef27ae665161c938d94f844d63de3c3fb20f509b8f.qcow2
|
|
@ -1,21 +0,0 @@
|
|||
# This is the CA for cockpit-tests images and data
|
||||
|
||||
-----BEGIN CERTIFICATE-----
|
||||
MIIDDDCCAfSgAwIBAgIJANdoyGJiUz+8MA0GCSqGSIb3DQEBCwUAMDUxEDAOBgNV
|
||||
BAoMB0NvY2twaXQxFDASBgNVBAsMC0NvY2twaXR1b3VzMQswCQYDVQQDDAJDQTAg
|
||||
Fw0xOTAyMDcxMDE4NDNaGA8zMDE4MDYxMDEwMTg0M1owNTEQMA4GA1UECgwHQ29j
|
||||
a3BpdDEUMBIGA1UECwwLQ29ja3BpdHVvdXMxCzAJBgNVBAMMAkNBMIIBIjANBgkq
|
||||
hkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAnvIZetd5yEhdE0c/9lYp1mC4M6qiu6E2
|
||||
wVMbJLwsOuCyCSaZs5eDap1kremHz7ms+Fq07TUsN/o5U7PBnNgM3z6Zbv78QN6R
|
||||
wn6ovLHfCyVqpg0nPMh3Hzpd0HDZQ+3eBayL2xfmBhU8p1+/vWVBOe49SDO15YDM
|
||||
/Ian7I/HRsnprz5PH3atquSf+B8/Q+lgbO0dHKhXlbnTsSy/Esee82HhYrDlxD3p
|
||||
Ow7EcZ7HACh/2dvF70BQpjnxTEc//4LNgP7hiqk4phsGzM/9QSFHW8ol4XlBDUi0
|
||||
F5nNXZTs3jKITTOeda5mppuKoZoC+7iFk8dLvV0Y187xD38X2XgGnwIDAQABox0w
|
||||
GzAMBgNVHRMEBTADAQH/MAsGA1UdDwQEAwIBBjANBgkqhkiG9w0BAQsFAAOCAQEA
|
||||
PHaVKb97ZN2m/sEVU+TGepVhCZ15frIaCJRuBPEs5rwcJjIctyRF4H6R6ec2b2lB
|
||||
6ni9eqU6pPgS+rVJPsxqCpelQiCZALR7FYoA6+FtfpLkB5+zwJUfexr7Q6I7llWI
|
||||
8OBOmtEADRv//2D+Iu6mM6nkzUK1K/wCcFS//roLjK/nKH2xd2lWbYk2Ro+nTPIm
|
||||
slwgk6fAUXQcd5v/XqrySZ5jny73jMqo7SRVC5suNuAfiT0/YGvE5N99+I5AkD5I
|
||||
R/R80/w1bDExfcqtx5UPBitMG2bx/gA07k4XbAGsEH5zvIdgsV9S5uYQEDjIRZys
|
||||
ScLMpNOd3JyD7ncvr6Ga6g==
|
||||
-----END CERTIFICATE-----
|
|
@ -1,37 +0,0 @@
|
|||
apiVersion: v1
|
||||
clusters:
|
||||
- cluster:
|
||||
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM2akNDQWRLZ0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREFtTVNRd0lnWURWUVFEREJ0dmNHVnUKYzJocFpuUXRjMmxuYm1WeVFERTFOamMxTWpjMk56UXdIaGNOTVRrd09UQXpNVFl5TVRFeldoY05NalF3T1RBeApNVFl5TVRFMFdqQW1NU1F3SWdZRFZRUUREQnR2Y0dWdWMyaHBablF0YzJsbmJtVnlRREUxTmpjMU1qYzJOelF3CmdnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUURQOUVKZ05NdEk0TG1KSk9wRGNiUlQKbWNDazF1amJiQVVydTU3bjFFY0ZnTEhDNzZCQmxsME9sbzNQcnRUVWVTR0d5c3R3eW5HT0J3MytGOFBzL1hjZApwWmwrdGkySkMweGVVSnFNVDMyclRwTHVwWG1tczZGMi90d0dTc0lSTWx5NjgyZ01NaHlmV0wrT2FRZERaM3NWCjd6aDBPd2E3cE5wWWUwSE1VbUM5QzFPaEltc254YVF2Mzh4TGU5SjgvQXZxSXZMV21Wc1J3cnEveWhVakphOWkKMkZYV20wa2Z4WjBNckpRQWI2cGlTZzVEbE5pc2htbURPbU1QR21mWlI5ZkY2SHA5MWRuV3lId2NCVmpnVTJHagpRMmlqWWh5WGRMYS9RaDEzTG9BQUFiQ29aUjVKcVNUL3luQU1SWlFXTE1CRFYvSC95Wk53RFI1WUJ1YXI4LzlsCkFnTUJBQUdqSXpBaE1BNEdBMVVkRHdFQi93UUVBd0lDcERBUEJnTlZIUk1CQWY4RUJUQURBUUgvTUEwR0NTcUcKU0liM0RRRUJDd1VBQTRJQkFRQXdWSDk2SGtTQ1NWTlRXbmJOK0R1TE1Kb0l0M3ZRTW5aQ2hwcnAzWUxiZ0MvZgpxeGMxUmtXaXBTemNhYUJQSHU4RkR6aTlMOUpFcVIwVWhyeHN5Sk9iSGxJOVB5cnN2WnhpR05pc2UvOE1IKzRtCjFlMUVDZGNlT3pHOVJlK09SOGV4b25GaitJSk9ZNG9xanVtRFM2ZmdRS01Ja08vN29SZmhxRGZJREMveVUvTnUKc09xYmZnS1dMeWxxOWJKTEtQVUkwemw3YnUrSmNyK3g1anhQaTFLY05yY1BXaXNoVFpXNExrakh0Wkl5QkNyLwp5MTQ2eHZLc1hHMXY2aEJ4ZTBvRnFrcVNqMzhUYTRXV2NNUTVXd1lxbU5xMVhkRFN0T0UzMm1iZmNMZWJXSXRuCitDVGVOcldOSnQxQXE4Q3p2UDdOMEwwVkxrT1NPSmpoTVMzNVdKUnYKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
|
||||
server: https://10.111.112.101:8443
|
||||
name: 10-111-112-101:8443
|
||||
contexts:
|
||||
- context:
|
||||
cluster: 10-111-112-101:8443
|
||||
user: scruffy/10-111-112-101:8443
|
||||
name: /10-111-112-101:8443/scruffy
|
||||
- context:
|
||||
cluster: 10-111-112-101:8443
|
||||
namespace: default
|
||||
user: system:admin/10-111-112-101:8443
|
||||
name: default/10-111-112-101:8443/system:admin
|
||||
- context:
|
||||
cluster: 10-111-112-101:8443
|
||||
namespace: marmalade
|
||||
user: scruffy/10-111-112-101:8443
|
||||
name: marmalade/10-111-112-101:8443/scruffy
|
||||
- context:
|
||||
cluster: 10-111-112-101:8443
|
||||
namespace: pizzazz
|
||||
user: scruffy/10-111-112-101:8443
|
||||
name: pizzazz/10-111-112-101:8443/scruffy
|
||||
current-context: default/10-111-112-101:8443/system:admin
|
||||
kind: Config
|
||||
preferences: {}
|
||||
users:
|
||||
- name: scruffy/10-111-112-101:8443
|
||||
user:
|
||||
token: S6xMoSAI-Rs4shOP0aU4Y4hMEqQJB2KlBAHU1q86M5Y
|
||||
- name: system:admin/10-111-112-101:8443
|
||||
user:
|
||||
client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURKRENDQWd5Z0F3SUJBZ0lCQ0RBTkJna3Foa2lHOXcwQkFRc0ZBREFtTVNRd0lnWURWUVFEREJ0dmNHVnUKYzJocFpuUXRjMmxuYm1WeVFERTFOamMxTWpjMk56UXdIaGNOTVRrd09UQXpNVFl5TVRFNFdoY05NakV3T1RBeQpNVFl5TVRFNVdqQk9NVFV3RlFZRFZRUUtFdzV6ZVhOMFpXMDZiV0Z6ZEdWeWN6QWNCZ05WQkFvVEZYTjVjM1JsCmJUcGpiSFZ6ZEdWeUxXRmtiV2x1Y3pFVk1CTUdBMVVFQXhNTWMzbHpkR1Z0T21Ga2JXbHVNSUlCSWpBTkJna3EKaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF0UTJvT1NhaHlBSnpVdlYxMmtKdXc5ZFBjaDFiT29CNgpRNW1vdEM3b0k0V1F3SnhISnBWTWIxRkhWTm96dnNJQWZEMTZiaXZGd1VLWk5iQmRQNlVCZy90MlRST1JJTHZRCjBvYUg5OEd5NGZXUEE3TE9kekZ3THcxeW1BTk9UNjdYZFpUdzB0cEhOQUN6akgyd0t1cS9MRTFoVDNpTGFKSFEKQVRxdzRuOVNFSFQwbVJTYTBDcU5HRWUrZkxZQXNrd3FPOHE0UW5NOWY4QTdHUVdxY29lMFdlVDRha1VYOUVlbwpONUg1Vmt2V0VuMHhmTXBYWHQ3VGM0YUlPbk9Ba3lJbnkwQzU4TVhDNlJibHo0KzN1cHNwRlB3ZTJtS1o5bXliClZrWFcrZjJQcjJaSEZJdS9ORmx1MFJnOEdmWVVMRHgvTG9TSGFxbzBzRndNTFpwZ1RjTzdzd0lEQVFBQm96VXcKTXpBT0JnTlZIUThCQWY4RUJBTUNCYUF3RXdZRFZSMGxCQXd3Q2dZSUt3WUJCUVVIQXdJd0RBWURWUjBUQVFILwpCQUl3QURBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQUVGRzQyS3lyNkFBNCsrdjU4M1VablRlVE4xdFdLYWlLCjRURjdlWFNQNG1TRmtrYjF1RGpuekphZWUwejJUMTBpVkUvNGdsOXhKcXVUZ3k4b0RQSmFSUFJuc0kxbGVLTzkKMi90bkFJQ3kzbE5wL2U4MksybFZGVDcvdlEwL3Nqd1lVaVAzQnRoQTZkdHpMUWpFTE1abytNZm1ucnJCM0k2TgpCelk1a1pTN2hENTdZQ2FCZmRjZ0hlNWQzY0p4ZEQ4RGhMNVNBemFUTUsrcW54ZXEvU2U2TU42alRYOVBnamxIClJjK2liSnhYVkh2TmUwNC9sN0I3S2pGVW9qRG1aTE8rK290RFdldmx4SHRNRzNuV21POWo1Z0V4NVlLaXJPcjkKSTU0NHVEUGlINkdJQUdENytwaFVpMzFMVTFuYmIreWN2akpvWWVlTG1WRERzWGZ6Ri9xUzN3PT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
|
||||
client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcEFJQkFBS0NBUUVBdFEyb09TYWh5QUp6VXZWMTJrSnV3OWRQY2gxYk9vQjZRNW1vdEM3b0k0V1F3SnhICkpwVk1iMUZIVk5venZzSUFmRDE2Yml2RndVS1pOYkJkUDZVQmcvdDJUUk9SSUx2UTBvYUg5OEd5NGZXUEE3TE8KZHpGd0x3MXltQU5PVDY3WGRaVHcwdHBITkFDempIMndLdXEvTEUxaFQzaUxhSkhRQVRxdzRuOVNFSFQwbVJTYQowQ3FOR0VlK2ZMWUFza3dxTzhxNFFuTTlmOEE3R1FXcWNvZTBXZVQ0YWtVWDlFZW9ONUg1Vmt2V0VuMHhmTXBYClh0N1RjNGFJT25PQWt5SW55MEM1OE1YQzZSYmx6NCszdXBzcEZQd2UybUtaOW15YlZrWFcrZjJQcjJaSEZJdS8KTkZsdTBSZzhHZllVTER4L0xvU0hhcW8wc0Z3TUxacGdUY083c3dJREFRQUJBb0lCQVFDR3ZQY29NUHZNNFNYNQo0dm9seDdLdXhCazNqMmxKRER2dyt2VjF3a0szekxxQTNNeUdoaTB2Mm9qL09MT3hqcWJWenRyQ0NvbE0zY2N2CkVXVVQ3RFJJaUdidHpWWC95a1lKcGx5aG9PRURENyt5dk9xeUFYUy9UMzZzYWlscFczQzA3SGFjTkIweE1pUnMKdFV6Wlk0R0o4cndzYkVVek9QQlhPZHBSZFBjWmp1OEF3ZDBkS2oxQmwvMGVCL2RxZCsrR3o0ZEFNR1h0b1V6WApSVGtBSWtLWGJEUHVON1oyck5BMzVmWnhBOE44b0NzalYyODRlUnhiUURQejVUQXB1RExEdHZqYk8xTVZvdVpECjZyZUtmSStnbDlIUnZ6YlMxekhXWjVSZ1F2RUFIV0pKcHBBMGF2NHdEU0NKZjhLNnl2R3NvbjJ0UzAveUErY3YKVVMxWEJIckJBb0dCQU1nNW5IM2twclBUUFRkcTRVQmhRWndLWG50Nm5kL2huRmQ2d1h1VVY5UmRrU1ZXbmxkcQpMOVcwVDU5ODFMeXdEb1ZXL2swUTkvelFMRG9ONTZDOGpTTnBkZHZtQlBHY2tkNU1JN1RtTEwvTVRtNWRLOGZHClZGNnorMVRvMm9BcW83czFFSWhseXlmckZPTVI0NThsSENWVEdPNlI3Y2MzQy8rVU01MVNQY0tMQW9HQkFPZDgKNFNwWFg0TjJHZmRZQTRyKzdqNHFDWXdLYVpyWjlmckxZQXI2cnBja2xScDRZL0M3b0Nja2VTM3BNMDB6SVlQQgpMQis0YVRJeXdBaVNxY3I1YVhnM0tZTkVKNFFpaEpiRjNqV1I2cUdtc3kxMzBCWVpHQVg1VDRMdHVJc0ZIV2NvCkdjNDJQSXhXREtxdzJzYUZBZTBFa2lkb2dsTit5eGt6WWJMQWVWaDVBb0dCQUxyaTQwbXl5Vktod2NyZkxQNTkKaU5MUDd0Nk1WWjJwcE5jV1VsQTU1enptVk5zb0hVVjBiTStvcklVdDdCZHVzUzhPUXZERi9PSnhvRVpUd2phSwpwNlk5QW5CTkk2SXRSUTNidlp4VkY4R3lQaWJQT2xVT3JxTnlsUTN0QmoySkR5aG00RmFmeE44dWttRmJ5ajA2ClV5b1hoUGJ4S0tMQW82ZGJ1azJHZlBUL0FvR0FZYlFPb1QxaGZlNENCYWlyVGlaTlRnV1dJL3BkR2xPMmc1VUYKUTMwTTVaUTJMb2J6djY2aGFRUDI5WTdBN1d1UVVMamVzOEMwL2MvM3gyYUhyYmpaY0Rqd0Y0eFRsV3l3UTZiZQpKQVFqWVBrb0ZSL0Z2eDMyU3Njd2JSV0MxNEpnSjZNQVNVNFEvalp2Z0RmSER4VWllL1I2NzVFbnVMQUNidStGCjQ5bGpIaGtDZ1lBUFA2RGM1SCtyMC8vVWUrSlNVTzY1N2pIdW5WYU8rZE81bVJZTzZqR1NHc2JpWnZ3RmZjZEIKZXZBTFJQQ0NOS3F4U1dGU2RXc3p0WUNtZ3BHSVFpaXowMncvSk54bERHTVd1T002WnlsODl2WmdPYk1SWnBWWQpzZ0pMNGZJMFFLZmdhMkZoYUdML1Z4L2c0RE91Vy8zWjZ6NEI3YzYySXE1OVFhaEZURERERlE9PQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
|
|
@ -1 +0,0 @@
|
|||
ipa-94b1c71c3c7d0eb739170278f537225fb347530ecc2a3282fb9224dbe334b84a.qcow2
|
|
@ -1 +0,0 @@
|
|||
openshift-c65dc46c4885d5b7cd1198d310cb0a62a515e0cf67aba4a0ba55caf2b7fddd39.qcow2
|
|
@ -1 +0,0 @@
|
|||
ovirt-f033c4457fecb1e9078eb16d7ac5239fe79455ca6b533f2a37de4f965cf174e7.qcow2
|
|
@ -1 +0,0 @@
|
|||
rhel-7-7-9d4481970308e14f9af75151118b6ffc9bf0fb2c1c9bda2f6c19d70b3cc612d4.qcow2
|
|
@ -1 +0,0 @@
|
|||
rhel-7-8-90e79cf4377714b69929e8613375987c0d7b6a8e5deff38bbfa9662ff583d5e0.qcow2
|
|
@ -1 +0,0 @@
|
|||
rhel-8-0-a7af23080d6f6d5595b5fbc5331a2f7148069223659701c1870c844babc56833.qcow2
|
|
@ -1 +0,0 @@
|
|||
rhel-8-1-25b42538c7798dda35e910167a9de2af924ff28d81e8612bb5318469c2962c06.qcow2
|
|
@ -1 +0,0 @@
|
|||
rhel-atomic-a1388f58b093cb83a15f4260f2fc3bbb42b85aae102e5f4e467f2635faeca4c9.qcow2
|
|
@ -1,78 +0,0 @@
|
|||
#! /bin/bash
|
||||
|
||||
# This file is part of Cockpit.
|
||||
#
|
||||
# Copyright (C) 2015 Red Hat, Inc.
|
||||
#
|
||||
# Cockpit is free software; you can redistribute it and/or modify it
|
||||
# under the terms of the GNU Lesser General Public License as published by
|
||||
# the Free Software Foundation; either version 2.1 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# Cockpit is distributed in the hope that it will be useful, but
|
||||
# WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
# Lesser General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU Lesser General Public License
|
||||
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
set -ex
|
||||
|
||||
out="$1"
|
||||
base="$2"
|
||||
|
||||
redirect_base=$(curl -s -w "%{redirect_url}" "$base" -o /dev/null)
|
||||
if [ -n "$redirect_base" ]; then
|
||||
base="$redirect_base"
|
||||
fi
|
||||
|
||||
# Lookup the newest base image recursively
|
||||
url="$base"
|
||||
while [ $# -gt 2 ]; do
|
||||
fragment="$3"
|
||||
|
||||
if [ "$fragment" = "sort" ]; then
|
||||
backref="$4"
|
||||
pattern="$5"
|
||||
|
||||
result="`wget -q -O- $url | grep -oE "$pattern" | sed -E "s/${pattern}/\\\\${backref} \\0/" | sort -V -k1 | tail -1`"
|
||||
fragment="`echo $result | cut -f2 -d' '`"
|
||||
|
||||
|
||||
if [ -z "$fragment" ]; then
|
||||
echo "Could not find '$pattern' at: $url" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
shift; shift
|
||||
fi
|
||||
|
||||
base="$url"
|
||||
url="$base/$fragment"
|
||||
|
||||
shift
|
||||
done
|
||||
|
||||
# we link to the file so wget can properly detect if we have already downloaded it
|
||||
# note that due to mirroring, timestamp comparison can result in unnecessary downloading
|
||||
out_base="`dirname $out`"
|
||||
intermediate="$out_base/$fragment"
|
||||
|
||||
if [ "$intermediate" != "$out" ]; then
|
||||
wget --no-clobber --directory-prefix="$out_base" "$base/$fragment"
|
||||
cp "$intermediate" "$out"
|
||||
else
|
||||
rm -f "$out"
|
||||
wget --directory-prefix="$out_base" "$base/$fragment"
|
||||
fi
|
||||
|
||||
# Make the image be at least 12 Gig. During boot, docker-storage-setup
|
||||
# will grow the partitions etc as appropriate, and atomic.setup will
|
||||
# explicitly grow the docker pool.
|
||||
|
||||
vsize=$(qemu-img info "$out" --output=json | python3 -c 'import json, sys; print(json.load(sys.stdin)["virtual-size"])')
|
||||
|
||||
if [ "$vsize" -lt 12884901888 ]; then
|
||||
qemu-img resize "$out" 12884901888
|
||||
fi
|
|
@ -1 +0,0 @@
|
|||
centos-7.bootstrap
|
|
@ -1,65 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
set -ex
|
||||
|
||||
YUM_INSTALL="yum --setopt=skip_missing_names_on_install=False -y install"
|
||||
|
||||
# We deploy candlepin via ansible
|
||||
$YUM_INSTALL epel-release
|
||||
|
||||
# Install dependencies
|
||||
CANDLEPIN_DEPS="\
|
||||
ansible \
|
||||
git \
|
||||
openssl \
|
||||
"
|
||||
|
||||
$YUM_INSTALL $CANDLEPIN_DEPS
|
||||
|
||||
mkdir -p playbookdir; cd playbookdir;
|
||||
|
||||
mkdir -p roles
|
||||
git clone https://github.com/candlepin/ansible-role-candlepin.git roles/candlepin
|
||||
|
||||
# Run the playbook
|
||||
cat > inventory <<- EOF
|
||||
[dev]
|
||||
localhost
|
||||
EOF
|
||||
|
||||
useradd -m admin
|
||||
echo admin:foobar | chpasswd
|
||||
echo 'admin ALL=(ALL) NOPASSWD: ALL' > /etc/sudoers.d/admin
|
||||
|
||||
cat > playbook.yml <<- EOF
|
||||
- hosts: dev
|
||||
|
||||
environment:
|
||||
JAVA_HOME: /usr/lib/jvm/java-1.8.0/
|
||||
|
||||
roles:
|
||||
- role: candlepin
|
||||
candlepin_git_pull: True
|
||||
candlepin_deploy_args: "-g -a -f -t"
|
||||
candlepin_user: admin
|
||||
candlepin_user_home: /home/admin
|
||||
candlepin_checkout: /home/admin/candlepin
|
||||
EOF
|
||||
|
||||
ansible-playbook -i inventory -c local -v --skip-tags 'system_update' playbook.yml
|
||||
|
||||
rm -rf playbookdir
|
||||
|
||||
# reduce image size
|
||||
yum clean all
|
||||
/var/lib/testvm/zero-disk.setup
|
||||
|
||||
# Final tweaks
|
||||
|
||||
rm -rf /var/log/journal/*
|
||||
echo "kernel.core_pattern=|/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e" > /etc/sysctl.d/50-coredump.conf
|
||||
|
||||
# Audit events to the journal
|
||||
rm -f '/etc/systemd/system/multi-user.target.wants/auditd.service'
|
||||
rm -rf /var/log/audit/
|
||||
|
|
@ -1,4 +0,0 @@
|
|||
#! /bin/bash
|
||||
|
||||
BASE=$(dirname $0)
|
||||
$BASE/virt-install-fedora "$1" x86_64 "http://mirror.centos.org/centos/7/os/x86_64/"
|
|
@ -1,8 +0,0 @@
|
|||
#! /bin/bash
|
||||
|
||||
set -e
|
||||
|
||||
# remove cockpit distro packages, testing with upstream master
|
||||
rpm --erase --verbose cockpit cockpit-ws cockpit-bridge cockpit-system
|
||||
|
||||
/var/lib/testvm/fedora.install "$@"
|
|
@ -1 +0,0 @@
|
|||
rhel.setup
|
|
@ -1,28 +0,0 @@
|
|||
#!/bin/sh
|
||||
set -eux
|
||||
|
||||
OUTPUT="$1"
|
||||
|
||||
curl https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-i386-disk.img > "$OUTPUT"
|
||||
|
||||
# prepare a cloud-init iso for disabling network source, to avoid a 90s timeout at boot
|
||||
WORKDIR=$(mktemp -d)
|
||||
trap "rm -rf '$WORKDIR'" EXIT INT QUIT PIPE
|
||||
cd "$WORKDIR"
|
||||
|
||||
cat > meta-data <<EOF
|
||||
{ "instance-id": "nocloud" }
|
||||
EOF
|
||||
|
||||
cat > user-data <<EOF
|
||||
#!/bin/sh
|
||||
set -ex
|
||||
sed -i 's/configdrive *//; s/ec2 *//' /etc/cirros-init/config
|
||||
(sleep 1; poweroff) &
|
||||
EOF
|
||||
|
||||
genisoimage -input-charset utf-8 -output cloud-init.iso -volid cidata -joliet -rock user-data meta-data
|
||||
|
||||
# boot it once with the cloud-init ISO
|
||||
qemu-system-x86_64 -enable-kvm -nographic -net none \
|
||||
-drive file="$OUTPUT",if=virtio -cdrom cloud-init.iso
|
|
@ -1,9 +0,0 @@
|
|||
#! /bin/bash
|
||||
|
||||
set -e
|
||||
|
||||
url="https://cloud.centos.org/centos/7/atomic/images"
|
||||
prefix="CentOS-Atomic-Host-GenericCloud.qcow2"
|
||||
|
||||
BASE=$(dirname $0)
|
||||
$BASE/atomic.bootstrap "$1" "$url" "$prefix"
|
|
@ -1,5 +0,0 @@
|
|||
#! /bin/bash
|
||||
|
||||
set -e
|
||||
|
||||
/var/lib/testvm/atomic.install --skip cockpit-sosreport --extra "/root/rpms/libssh*" --extra "/var/tmp/build-results/cockpit-dashboard*" "$@"
|
|
@ -1,72 +0,0 @@
|
|||
#!/bin/bash
|
||||
# This file is part of Cockpit.
|
||||
#
|
||||
# Copyright (C) 2016 Red Hat, Inc.
|
||||
#
|
||||
# Cockpit is free software; you can redistribute it and/or modify it
|
||||
# under the terms of the GNU Lesser General Public License as published by
|
||||
# the Free Software Foundation; either version 2.1 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# Cockpit is distributed in the hope that it will be useful, but
|
||||
# WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
# Lesser General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU Lesser General Public License
|
||||
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
set -ex
|
||||
|
||||
# The docker pool should grow automatically as needed, but we grow it
|
||||
# explicitly here anyway. This is hopefully more reliable.
|
||||
# HACK: docker falls over regularly, print its log if it does
|
||||
systemctl start docker || journalctl -u docker
|
||||
lvresize atomicos/root -l+50%FREE -r
|
||||
if lvs atomicos/docker-pool 2>/dev/null; then
|
||||
lvresize atomicos/docker-pool -l+100%FREE
|
||||
elif lvs atomicos/docker-root-lv; then
|
||||
lvresize atomicos/docker-root-lv -l+100%FREE
|
||||
fi
|
||||
|
||||
# Get the centos cockpit/ws image
|
||||
docker pull registry.centos.org/cockpit/ws:latest
|
||||
docker tag registry.centos.org/cockpit/ws cockpit/ws
|
||||
|
||||
# docker images that we need for integration testing
|
||||
/var/lib/testvm/docker-images.setup
|
||||
|
||||
# Configure core dumps
|
||||
echo "kernel.core_pattern=|/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e" > /etc/sysctl.d/50-coredump.conf
|
||||
|
||||
# Download the libssh RPM plus dependencies which we'll use for
|
||||
# package overlay. The only way to do this is via a container
|
||||
. /etc/os-release
|
||||
REPO="updates"
|
||||
if [ "$ID" = "rhel" ]; then
|
||||
subscription-manager repos --enable rhel-7-server-extras-rpms
|
||||
REPO="rhel-7-server-extras-rpms"
|
||||
ID="rhel7"
|
||||
fi
|
||||
docker run --rm --volume=/etc/yum.repos.d:/etc/yum.repos.d:z --volume=/root/rpms:/tmp/rpms:rw,z "$ID:$VERSION_ID" /bin/sh -cex "yum install -y findutils createrepo_c && yum install -y --downloadonly --enablerepo=$REPO libssh && find /var -name '*.rpm' | while read rpm; do mv -v \$rpm /tmp/rpms; done; createrepo_c /tmp/rpms"
|
||||
rm -f /etc/yum.repos.d/*
|
||||
cat >/etc/yum.repos.d/deps.repo <<EOF
|
||||
[deps]
|
||||
baseurl=file:///root/rpms
|
||||
enabled=1
|
||||
EOF
|
||||
|
||||
# Switch to continuous stream
|
||||
ostree remote add --set=gpg-verify=false centos-atomic-continuous https://ci.centos.org/artifacts/sig-atomic/rdgo/centos-continuous/ostree/repo/
|
||||
rpm-ostree rebase centos-atomic-continuous:centos-atomic-host/7/x86_64/devel/continuous
|
||||
|
||||
ostree checkout centos-atomic-continuous:centos-atomic-host/7/x86_64/devel/continuous /var/local-tree
|
||||
|
||||
# reduce image size
|
||||
/var/lib/testvm/zero-disk.setup
|
||||
|
||||
# Prevent SSH from hanging for a long time when no external network access
|
||||
echo 'UseDNS no' >> /etc/ssh/sshd_config
|
||||
|
||||
# Final tweaks
|
||||
rm -rf /var/log/journal/*
|
|
@ -1,7 +0,0 @@
|
|||
#! /bin/sh -ex
|
||||
|
||||
RELEASE=buster
|
||||
RELEASENUM=10
|
||||
LATEST_DAILY=$(curl -s https://cloud.debian.org/images/cloud/$RELEASE/daily/ | sed -n '/<a href=/ { s/^.*href="//; s_/".*$__; p }'| sort -nu | tail -n1)
|
||||
|
||||
exec $(dirname $0)/lib/debian.bootstrap "$1" "https://cloud.debian.org/images/cloud/buster/daily/$LATEST_DAILY/debian-${RELEASENUM}-genericcloud-amd64-daily-${LATEST_DAILY}.qcow2"
|
|
@ -1,8 +0,0 @@
|
|||
#! /bin/bash
|
||||
|
||||
set -e
|
||||
|
||||
/var/lib/testvm/debian.install "$@"
|
||||
|
||||
# HACK: https://bugs.debian.org/914694
|
||||
sed -i '/IndividualCalls/ s/=no/=yes/' /etc/firewalld/firewalld.conf
|
|
@ -1 +0,0 @@
|
|||
debian.setup
|
|
@ -1 +0,0 @@
|
|||
debian-stable.bootstrap
|
|
@ -1,8 +0,0 @@
|
|||
#! /bin/bash
|
||||
|
||||
set -e
|
||||
|
||||
/var/lib/testvm/debian.install "$@"
|
||||
|
||||
# HACK: https://bugs.debian.org/914694
|
||||
sed -i '/IndividualCalls/ s/=no/=yes/' /etc/firewalld/firewalld.conf
|
|
@ -1 +0,0 @@
|
|||
debian.setup
|
|
@ -1,222 +0,0 @@
|
|||
#! /bin/bash
|
||||
# Shared .setup between all Debian/Ubuntu flavors
|
||||
|
||||
set -ex
|
||||
|
||||
# Enable a console on ttyS0 so that we can log-in via vm-run.
|
||||
# and make the boot up more verbose
|
||||
sed -i 's/GRUB_CMDLINE_LINUX_DEFAULT/# GRUB_CMDLINE_LINUX_DEFAULT/' /etc/default/grub
|
||||
|
||||
# We install all dependencies of the cockpit packages since we want
|
||||
# them to not spontaneously change from one test run to the next when
|
||||
# the distribution repository is updated.
|
||||
#
|
||||
COCKPIT_DEPS="\
|
||||
cryptsetup \
|
||||
docker.io \
|
||||
libblockdev-mdraid2 \
|
||||
libjson-glib-1.0-0 \
|
||||
libpcp3 \
|
||||
libpolkit-agent-1-0 \
|
||||
libpolkit-gobject-1-0 \
|
||||
libpwquality-tools \
|
||||
libssh-4 \
|
||||
libteam-utils \
|
||||
libvirt-daemon-system \
|
||||
libvirt-dbus \
|
||||
libosinfo-bin \
|
||||
network-manager \
|
||||
pcp \
|
||||
policykit-1 \
|
||||
python3-dbus \
|
||||
qemu-block-extra \
|
||||
realmd \
|
||||
selinux-basics \
|
||||
thin-provisioning-tools \
|
||||
unattended-upgrades \
|
||||
tuned \
|
||||
xdg-utils \
|
||||
udisks2 \
|
||||
udisks2-lvm2 \
|
||||
"
|
||||
|
||||
# We also install the packages necessary to join a FreeIPA domain so
|
||||
# that we don't have to go to the network during a test run.
|
||||
IPA_CLIENT_PACKAGES="\
|
||||
freeipa-client \
|
||||
sssd-tools \
|
||||
sssd-dbus \
|
||||
packagekit \
|
||||
"
|
||||
|
||||
TEST_PACKAGES="\
|
||||
acl \
|
||||
curl \
|
||||
firewalld \
|
||||
gdb \
|
||||
iproute2 \
|
||||
mdadm \
|
||||
nfs-server \
|
||||
qemu-kvm \
|
||||
socat \
|
||||
systemd-coredump \
|
||||
virtinst \
|
||||
xfsprogs \
|
||||
sosreport \
|
||||
"
|
||||
|
||||
RELEASE=$(grep -m1 ^deb /etc/apt/sources.list | awk '{print $3}')
|
||||
case "$RELEASE" in
|
||||
bionic)
|
||||
# these packages are not in Ubuntu 18.04
|
||||
COCKPIT_DEPS="${COCKPIT_DEPS/libvirt-dbus /}"
|
||||
;;
|
||||
esac
|
||||
|
||||
if [ "${1#ubuntu}" != "$1" ]; then
|
||||
# our tests need scsi_debug, which the cloud kernel does not have; install full kernel
|
||||
TEST_PACKAGES="$TEST_PACKAGES linux-image-generic"
|
||||
fi
|
||||
|
||||
# our cloud-init.iso does not set up the host name
|
||||
echo "127.0.1.1 $(hostname)" >> /etc/hosts
|
||||
|
||||
if grep -q 'ID=ubuntu' /etc/os-release; then
|
||||
PBUILDER_OPTS='COMPONENTS="main universe"'
|
||||
|
||||
# We want to use/test NetworkManager instead of netplan/networkd for ethernets
|
||||
mkdir -p /etc/NetworkManager/conf.d
|
||||
touch /etc/NetworkManager/conf.d/10-globally-managed-devices.conf
|
||||
fi
|
||||
|
||||
# some cloud images have a pre-defined admin user or group, for them cloud-init admin creation fails
|
||||
userdel -r admin || true
|
||||
groupdel admin || true
|
||||
useradd -m -U -c Administrator -G sudo -s /bin/bash admin
|
||||
echo admin:foobar | chpasswd
|
||||
cp -r ~root/.ssh ~admin/
|
||||
chown -R admin:admin ~admin/.ssh
|
||||
|
||||
# avoid NM-wait-online hanging on disconnected interfaces
|
||||
mkdir -p /etc/NetworkManager/conf.d/
|
||||
printf '[main]\nno-auto-default=*\n' > /etc/NetworkManager/conf.d/noauto.conf
|
||||
|
||||
if [ "${1#debian}" != "$1" ]; then
|
||||
# HACK: Debian's cloud-init generates a *.cfg file, but /etc/network/interfaces sources extension-less files
|
||||
mv /etc/network/interfaces.d/50-cloud-init.cfg /etc/network/interfaces.d/50-cloud-init
|
||||
fi
|
||||
|
||||
# debian-testing image gets bootstrapped from debian stable; upgrade
|
||||
if [ "$1" = "debian-testing" ]; then
|
||||
rm --verbose -f /etc/apt/sources.list.d/*
|
||||
echo 'deb http://deb.debian.org/debian testing main' > /etc/apt/sources.list
|
||||
fi
|
||||
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
apt-get -y update
|
||||
# apt go-faster
|
||||
echo 'Acquire::Languages "none";' > /etc/apt/apt.conf.d/90nolanguages
|
||||
apt-get install -y eatmydata
|
||||
# remove packages that we don't need
|
||||
for p in lxd snapd landscape-common accountsservice open-vm-tools ufw cloud-init; do eatmydata apt-get purge -y --auto-remove $p || true; done
|
||||
|
||||
# HACK: work around fuse 2.9.9-1 install failure (https://bugs.debian.org/935496)
|
||||
if [ "$1" = "debian-testing" ]; then
|
||||
rm /dev/fuse
|
||||
# this needs to happen right away, as upgrading other packages triggers udev events which recreate /dev/fuse
|
||||
eatmydata apt-get install -y fuse
|
||||
fi
|
||||
|
||||
# HACK: debian-stable image got /usr/bin/qemu-img removed, even though qemu-utils package is installed
|
||||
if [ "$1" = "debian-stable" ]; then
|
||||
eatmydata apt-get install --reinstall -y qemu-utils
|
||||
fi
|
||||
|
||||
# install our dependencies
|
||||
DEBIAN_FRONTEND=noninteractive eatmydata apt-get -y dist-upgrade
|
||||
eatmydata apt-get -y install $TEST_PACKAGES $COCKPIT_DEPS $IPA_CLIENT_PACKAGES
|
||||
|
||||
# Prepare for building
|
||||
#
|
||||
|
||||
# extract control files and adjust them for our release, so that we can parse the build deps
|
||||
mkdir -p /tmp/out
|
||||
curl -L https://github.com/cockpit-project/cockpit/archive/master.tar.gz | tar -C /tmp/out --strip-components=1 --wildcards -zxf - '*/debian/'
|
||||
/tmp/out/tools/debian/adjust-for-release $(lsb_release -sc)
|
||||
|
||||
# Disable build-dep installation for the real builds
|
||||
cat > ~/.pbuilderrc <<- EOF
|
||||
DISTRIBUTION=$RELEASE
|
||||
PBUILDERSATISFYDEPENDSCMD=true
|
||||
$PBUILDER_OPTS
|
||||
EOF
|
||||
|
||||
eatmydata apt-get -y install dpkg-dev pbuilder
|
||||
|
||||
pbuilder --create --extrapackages "fakeroot $PBUILDER_EXTRA"
|
||||
/usr/lib/pbuilder/pbuilder-satisfydepends-classic --control /tmp/out/tools/debian/control --force-version --echo|grep apt-get | pbuilder --login --save-after-login
|
||||
rm -rf /tmp/out
|
||||
|
||||
# Debian does not automatically start the default libvirt network
|
||||
virsh net-autostart default
|
||||
|
||||
# Don't automatically update on boot or daily
|
||||
systemctl disable apt-daily.service apt-daily.timer || true
|
||||
|
||||
# Enable coredumping via systemd
|
||||
echo "kernel.core_pattern=|/lib/systemd/systemd-coredump %P %u %g %s %t %c %e" > /etc/sysctl.d/50-coredump.conf
|
||||
printf 'DefaultLimitCORE=infinity\n' >> /etc/systemd/system.conf
|
||||
|
||||
# HACK: we need to restart it in case aufs-dkms was installed after docker.io
|
||||
# and thus docker.io auto-switches its backend
|
||||
systemctl restart docker || journalctl -u docker
|
||||
I=$(docker info)
|
||||
if ! echo "$I" | grep -Eq 'Storage.*(aufs|overlay)'; then
|
||||
echo "ERROR! docker does not use aufs or overlayfs"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# docker images that we need for integration testing
|
||||
/var/lib/testvm/docker-images.setup
|
||||
|
||||
rm -rf /var/lib/docker/devicemapper
|
||||
|
||||
# in case there are unnecessary packages
|
||||
eatmydata apt-get -y autoremove || true
|
||||
|
||||
# disable udev network names, our tests expect the kernel schema
|
||||
sed -i '/GRUB_CMDLINE_LINUX=/ s/"$/ net.ifnames=0 biosdevname=0"/' /etc/default/grub
|
||||
rm -f /etc/udev/rules.d/70-persistent-net.rules /etc/udev/rules.d/75-cloud-ifupdown.rules
|
||||
update-grub
|
||||
sed -i 's/ens[^[:space:]:]*/eth0/' /etc/network/interfaces /etc/network/interfaces.d/* /etc/netplan/*.yaml || true
|
||||
update-initramfs -u
|
||||
|
||||
|
||||
# reduce image size
|
||||
apt-get clean
|
||||
pbuilder clean
|
||||
rm -f /var/cache/apt/*cache.bin
|
||||
/var/lib/testvm/zero-disk.setup
|
||||
|
||||
# Final tweaks
|
||||
|
||||
# Enable persistent journal
|
||||
mkdir -p /var/log/journal
|
||||
|
||||
# Allow root login with password
|
||||
sed -i 's/^[# ]*PermitRootLogin .*/PermitRootLogin yes/' /etc/ssh/sshd_config
|
||||
|
||||
# At least debian-9 virt-install image only has RSA key
|
||||
[ -e /etc/ssh/ssh_host_ed25519_key ] || ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -N '' -t ed25519
|
||||
[ -e /etc/ssh/ssh_host_ecdsa_key ] || ssh-keygen -f /etc/ssh/ssh_host_ecdsa_key -N '' -t ecdsa
|
||||
|
||||
# Prevent SSH from hanging for a long time when no external network access
|
||||
echo 'UseDNS no' >> /etc/ssh/sshd_config
|
||||
|
||||
# HACK: https://bugzilla.mindrot.org/show_bug.cgi?id=2512
|
||||
# Disable the restarting of sshd when networking changes
|
||||
ln -snf /bin/true /etc/network/if-up.d/openssh-server
|
||||
|
||||
# Stop showing 'To run a command as administrator (user "root"), use "sudo <command>". See "man
|
||||
# sudo_root" for details.` message in admins terminal.
|
||||
touch /home/admin/.sudo_as_admin_successful
|
|
@ -1,21 +0,0 @@
|
|||
#!/bin/bash
|
||||
#
|
||||
# Copyright (C) 2015 Red Hat Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful, but
|
||||
# WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
# General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
|
||||
# 02110-1301 USA.
|
||||
|
||||
BASE=$(dirname $0)
|
||||
$BASE/virt-install-fedora "$1" x86_64 "https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/23/Server/x86_64/os/"
|
|
@ -1,11 +0,0 @@
|
|||
#! /bin/bash
|
||||
|
||||
useradd -c Administrator -G wheel admin
|
||||
echo foobar | passwd --stdin admin
|
||||
|
||||
dnf -y update
|
||||
dnf -y install fedora-release-server
|
||||
firewall-cmd --permanent --add-service cockpit
|
||||
|
||||
# Phantom can't use TLS..
|
||||
sed -i -e 's/ExecStart=.*/\0 --no-tls/' /usr/lib/systemd/system/cockpit.service
|
|
@ -1,21 +0,0 @@
|
|||
#!/bin/bash
|
||||
#
|
||||
# Copyright (C) 2018 Red Hat Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful, but
|
||||
# WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
# General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
|
||||
# 02110-1301 USA.
|
||||
|
||||
BASE=$(dirname $0)
|
||||
$BASE/virt-install-fedora "$1" x86_64 "http://dl.fedoraproject.org/pub/fedora/linux/releases/29/Server/x86_64/os/"
|
|
@ -1,4 +0,0 @@
|
|||
#! /bin/bash
|
||||
|
||||
set -e
|
||||
/var/lib/testvm/fedora.install "$@"
|
|
@ -1 +0,0 @@
|
|||
fedora.setup
|
|
@ -1,21 +0,0 @@
|
|||
#!/bin/bash
|
||||
#
|
||||
# Copyright (C) 2019 Red Hat Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful, but
|
||||
# WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
# General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
|
||||
# 02110-1301 USA.
|
||||
|
||||
BASE=$(dirname $0)
|
||||
$BASE/virt-install-fedora "$1" x86_64 "http://dl.fedoraproject.org/pub/fedora/linux/releases/30/Server/x86_64/os/"
|
|
@ -1,4 +0,0 @@
|
|||
#! /bin/bash
|
||||
|
||||
set -e
|
||||
/var/lib/testvm/fedora.install "$@"
|
|
@ -1 +0,0 @@
|
|||
fedora.setup
|
|
@ -1,23 +0,0 @@
|
|||
#!/bin/bash
|
||||
#
|
||||
# Copyright (C) 2019 Red Hat Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful, but
|
||||
# WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
# General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
|
||||
# 02110-1301 USA.
|
||||
|
||||
BASE=$(dirname $0)
|
||||
# once fedora 31 is released, replace url:
|
||||
# http://dl.fedoraproject.org/pub/fedora/linux/releases/31/Server/x86_64/os/
|
||||
$BASE/virt-install-fedora "$1" x86_64 "https://dl.fedoraproject.org/pub/fedora/linux/development/31/Everything/x86_64/os/"
|
|
@ -1,4 +0,0 @@
|
|||
#! /bin/bash
|
||||
|
||||
set -e
|
||||
/var/lib/testvm/fedora.install "$@"
|
|
@ -1 +0,0 @@
|
|||
fedora.setup
|
|
@ -1,14 +0,0 @@
|
|||
#! /bin/bash
|
||||
|
||||
set -e
|
||||
|
||||
url="https://download.fedoraproject.org/pub/alt/atomic/stable/"
|
||||
|
||||
BASE=$(dirname $0)
|
||||
|
||||
# The Fedora URLs have the version twice in the name. for example:
|
||||
# https://dl.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-28-20180425.0/AtomicHost/x86_64/images/Fedora-AtomicHost-28-20180425.0.x86_64.qcow2
|
||||
$BASE/atomic.bootstrap "$1" "$url" \
|
||||
sort 3 "Fedora(-atomic)?-[0-9][0-9](-updates)?-([-0-9\.]+)" \
|
||||
"AtomicHost" "x86_64" "images" \
|
||||
sort 1 "Fedora-AtomicHost-([-0-9\.]+).x86_64.qcow2"
|
|
@ -1,9 +0,0 @@
|
|||
#! /bin/bash
|
||||
|
||||
set -e
|
||||
|
||||
/var/lib/testvm/atomic.install --verbose --skip cockpit-kdump --extra "/root/rpms/libssh*" "$@"
|
||||
|
||||
# HACK: https://github.com/projectatomic/rpm-ostree/issues/1360
|
||||
# rpm-ostree upgrade --check otherwise fails
|
||||
mkdir -p /var/cache/rpm-ostree
|
|
@ -1,18 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
set -ex
|
||||
|
||||
# HACK: https://bugzilla.redhat.com/show_bug.cgi?id=1341829
|
||||
# SELinux breaks coredumping on fedora-25
|
||||
printf '(allow init_t domain (process (rlimitinh)))\n' > domain.cil
|
||||
semodule -i domain.cil
|
||||
|
||||
# HACK: docker falls over regularly, print its log if it does
|
||||
systemctl start docker || journalctl -u docker
|
||||
|
||||
os=$(ls /ostree/repo/refs/remotes/fedora-atomic/*/)
|
||||
docker pull "registry.fedoraproject.org/f$os/cockpit"
|
||||
docker tag "registry.fedoraproject.org/f$os/cockpit" cockpit/ws
|
||||
|
||||
|
||||
/var/lib/testvm/atomic.setup
|
|
@ -1,21 +0,0 @@
|
|||
#!/bin/bash
|
||||
#
|
||||
# Copyright (C) 2019 Red Hat Inc.
|
||||
#
|
||||
# This program is free software; you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation; either version 2 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful, but
|
||||
# WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
# General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program; if not, write to the Free Software
|
||||
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
|
||||
# 02110-1301 USA.
|
||||
|
||||
BASE=$(dirname $0)
|
||||
$BASE/virt-install-fedora "$1" i386 "https://dl.fedoraproject.org/pub/fedora-secondary/releases/30/Server/i386/os/"
|
|
@ -1 +0,0 @@
|
|||
fedora-30.install
|
|
@ -1 +0,0 @@
|
|||
fedora.setup
|
|
@ -1,11 +0,0 @@
|
|||
#! /bin/bash
|
||||
|
||||
useradd -c Administrator -G wheel admin
|
||||
echo foobar | passwd --stdin admin
|
||||
|
||||
dnf -y update
|
||||
dnf -y install fedora-release-server
|
||||
firewall-cmd --permanent --add-service cockpit
|
||||
|
||||
# Phantom can't use TLS..
|
||||
sed -i -e 's/ExecStart=.*/\0 --no-tls/' /usr/lib/systemd/system/cockpit.service
|
|
@ -1 +0,0 @@
|
|||
fedora-30.bootstrap
|
|
@ -1 +0,0 @@
|
|||
fedora-30.install
|
|
@ -1 +0,0 @@
|
|||
fedora.setup
|
|
@ -1,201 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
set -ex
|
||||
IMAGE="$1"
|
||||
|
||||
# avoid failures when running image builds in a non-English locale (ssh transfers the host environment)
|
||||
unset LANGUAGE
|
||||
unset LANG
|
||||
export LC_ALL=C.utf8
|
||||
|
||||
# keep this in sync with avocado/selenium image mapping in bots/tests-invoke
|
||||
if [ "$IMAGE" = fedora-30 ]; then
|
||||
AVOCADO=1
|
||||
fi
|
||||
|
||||
# HACK - virt-resize might not be able to resize our xfs rootfs,
|
||||
# depending on how it was compiled and which plugins are installed,
|
||||
# and will just silently not do it. So we do it here.
|
||||
#
|
||||
if [ "$IMAGE" != fedora-31 ]; then
|
||||
xfs_growfs /
|
||||
fi
|
||||
df -h /
|
||||
|
||||
echo foobar | passwd --stdin root
|
||||
|
||||
HAVE_KUBERNETES=
|
||||
if [ $(uname -m) = x86_64 ]; then
|
||||
HAVE_KUBERNETES=1
|
||||
fi
|
||||
|
||||
# HACK docker not available on f31
|
||||
# https://github.com/cockpit-project/cockpit/issues/12670
|
||||
HAVE_DOCKER=
|
||||
if [ "$1" != fedora-31 ]; then
|
||||
HAVE_DOCKER=1
|
||||
fi
|
||||
|
||||
# We install all dependencies of the cockpit packages since we want
|
||||
# them to not spontaneously change from one test run to the next when
|
||||
# the distribution repository is updated.
|
||||
#
|
||||
COCKPIT_DEPS="\
|
||||
device-mapper-multipath \
|
||||
etcd \
|
||||
glibc-all-langpacks \
|
||||
glib-networking \
|
||||
grubby \
|
||||
json-glib \
|
||||
kexec-tools \
|
||||
libssh \
|
||||
libvirt-daemon-kvm \
|
||||
libvirt-client \
|
||||
libvirt-dbus \
|
||||
NetworkManager-team \
|
||||
openssl \
|
||||
PackageKit \
|
||||
pcp \
|
||||
pcp-libs \
|
||||
qemu \
|
||||
realmd \
|
||||
selinux-policy-targeted \
|
||||
setroubleshoot-server \
|
||||
sos \
|
||||
sscg \
|
||||
system-logos \
|
||||
subscription-manager \
|
||||
tuned \
|
||||
virt-install \
|
||||
"
|
||||
|
||||
[ -z "$HAVE_DOCKER" ] || COCKPIT_DEPS="$COCKPIT_DEPS atomic docker"
|
||||
|
||||
COCKPIT_DEPS="$COCKPIT_DEPS udisks2 udisks2-lvm2 udisks2-iscsi"
|
||||
|
||||
[ -z "$HAVE_KUBERNETES" ] || COCKPIT_DEPS="$COCKPIT_DEPS kubernetes"
|
||||
|
||||
# We also install the packages necessary to join a FreeIPA domain so
|
||||
# that we don't have to go to the network during a test run.
|
||||
#
|
||||
IPA_CLIENT_PACKAGES="\
|
||||
freeipa-client \
|
||||
oddjob \
|
||||
oddjob-mkhomedir \
|
||||
sssd \
|
||||
sssd-dbus \
|
||||
libsss_sudo \
|
||||
"
|
||||
|
||||
TEST_PACKAGES="\
|
||||
systemtap-runtime-virtguest \
|
||||
valgrind \
|
||||
gdb \
|
||||
targetcli \
|
||||
dnf-automatic \
|
||||
cryptsetup \
|
||||
clevis-luks \
|
||||
socat \
|
||||
tang \
|
||||
podman \
|
||||
ntp \
|
||||
libvirt-daemon-config-network \
|
||||
"
|
||||
|
||||
# HACK - For correct work of ABRT in Fedora 26 Alpha release a following
|
||||
# packages are necessary. In Fedora 26 Beta and later these packages should be
|
||||
# installed by default. See https://bugzilla.redhat.com/show_bug.cgi?id=1436941
|
||||
#
|
||||
ABRT_PACKAGES="\
|
||||
abrt-desktop \
|
||||
libreport-plugin-systemd-journal \
|
||||
"
|
||||
|
||||
rm -rf /etc/sysconfig/iptables
|
||||
|
||||
maybe() { if type "$1" >/dev/null 2>&1; then "$@"; fi; }
|
||||
|
||||
# For the D-Bus test server
|
||||
maybe firewall-cmd --permanent --add-port 8765/tcp
|
||||
|
||||
echo 'NETWORKING=yes' > /etc/sysconfig/network
|
||||
|
||||
useradd -c Administrator -G wheel admin
|
||||
echo foobar | passwd --stdin admin
|
||||
|
||||
if [ "${IMAGE%-i386}" != "$IMAGE" ]; then
|
||||
TEST_PACKAGES="${TEST_PACKAGES/podman /}"
|
||||
fi
|
||||
|
||||
if [ "${IMAGE%-testing}" != "$IMAGE" ]; then
|
||||
dnf config-manager --set-enabled updates-testing
|
||||
fi
|
||||
|
||||
dnf $DNF_OPTS -y upgrade
|
||||
dnf $DNF_OPTS -y install $TEST_PACKAGES $COCKPIT_DEPS $IPA_CLIENT_PACKAGES $ABRT_PACKAGES
|
||||
|
||||
if [ -n "$AVOCADO" ]; then
|
||||
|
||||
# enable python3 avocado support repository
|
||||
dnf module install -y avocado:69lts
|
||||
|
||||
dnf $DNF_OPTS -y install \
|
||||
fontconfig \
|
||||
npm \
|
||||
chromium-headless \
|
||||
python3-libvirt \
|
||||
python3-avocado \
|
||||
python3-avocado-plugins-output-html \
|
||||
python3-selenium
|
||||
|
||||
npm -g install chrome-remote-interface
|
||||
echo 'NODE_PATH=/usr/lib/node_modules' >> /etc/environment
|
||||
fi
|
||||
|
||||
dnf $DNF_OPTS -y install mock dnf-plugins-core rpm-build
|
||||
useradd -c Builder -G mock builder
|
||||
|
||||
if [ "${IMAGE%-testing}" != "$IMAGE" ]; then
|
||||
# Enable updates-testing in mock
|
||||
echo "config_opts['yum.conf'] += '[updates-testing]\nenabled=1'" >>/etc/mock/default.cfg
|
||||
fi
|
||||
|
||||
# HACK - mock --installdeps is broken, it seems that it forgets to
|
||||
# copy the source rpm to a location that dnf can actually access. A
|
||||
# workaround is to pass "--no-bootstrap-chroot".
|
||||
#
|
||||
# When you remove this hack, also remove it in fedora-*.install.
|
||||
#
|
||||
# https://bugzilla.redhat.com/show_bug.cgi?id=1447627
|
||||
|
||||
opsys=$(cut -d '-' -f 1 <<< "$IMAGE")
|
||||
version=$(cut -d '-' -f 2 <<< "$IMAGE")
|
||||
# If version is not number (testing/i386) then use Fedora 30
|
||||
if ! [ "$version" -eq "$version" ] 2>/dev/null; then version=30; fi
|
||||
|
||||
su builder -c "/usr/bin/mock --no-bootstrap-chroot --verbose -i $(/var/lib/testvm/build-deps.sh "$opsys $version")"
|
||||
su builder -c "/usr/bin/mock --install --verbose rpmlint"
|
||||
|
||||
if [ -n "$HAVE_DOCKER" ]; then
|
||||
# HACK: docker falls over regularly, print its log if it does
|
||||
systemctl start docker || journalctl -u docker
|
||||
# docker images that we need for integration testing
|
||||
/var/lib/testvm/docker-images.setup
|
||||
fi
|
||||
|
||||
# Configure kubernetes
|
||||
[ -z "$HAVE_KUBERNETES" ] || /var/lib/testvm/kubernetes.setup
|
||||
|
||||
# reduce image size
|
||||
dnf clean all
|
||||
/var/lib/testvm/zero-disk.setup --keep-mock-cache
|
||||
|
||||
ln -sf ../selinux/config /etc/sysconfig/selinux
|
||||
printf "SELINUX=enforcing\nSELINUXTYPE=targeted\n" > /etc/selinux/config
|
||||
|
||||
# Prevent SSH from hanging for a long time when no external network access
|
||||
echo 'UseDNS no' >> /etc/ssh/sshd_config
|
||||
|
||||
# Audit events to the journal
|
||||
rm -f '/etc/systemd/system/multi-user.target.wants/auditd.service'
|
||||
rm -rf /var/log/audit/
|
|
@ -1 +0,0 @@
|
|||
fedora-29.bootstrap
|
|
@ -1,49 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
set -eufx
|
||||
|
||||
# ipa requires an UTF-8 locale
|
||||
export LC_ALL=C.UTF-8
|
||||
|
||||
echo foobar | passwd --stdin root
|
||||
|
||||
dnf -y remove firewalld
|
||||
dnf -y update
|
||||
dnf -y install freeipa-server freeipa-server-dns bind bind-dyndb-ldap iptables
|
||||
|
||||
iptables -F
|
||||
|
||||
nmcli con add con-name "static-eth1" ifname eth1 type ethernet ip4 "10.111.112.100/20" ipv4.dns "10.111.112.100" gw4 "10.111.112.1"
|
||||
nmcli con up "static-eth1"
|
||||
hostnamectl set-hostname f0.cockpit.lan
|
||||
|
||||
# Let's make sure that ipa-server-install doesn't block on
|
||||
# /dev/random.
|
||||
#
|
||||
rm -f /dev/random
|
||||
ln -s /dev/urandom /dev/random
|
||||
|
||||
ipa-server-install -U -p foobarfoo -a foobarfoo -n cockpit.lan -r COCKPIT.LAN --setup-dns --no-forwarders
|
||||
|
||||
# Make sure any initial password change is overridden
|
||||
printf 'foobarfoo\nfoobarfoo\nfoobarfoo\n' | kinit admin@COCKPIT.LAN
|
||||
|
||||
# Default password expiry of 90 days is impractical
|
||||
ipa pwpolicy-mod --minlife=0 --maxlife=1000
|
||||
# Change password to apply new password policy
|
||||
printf 'foobarfoo\nfoobarfoo\n' | ipa user-mod --password admin
|
||||
ipa user-show --all admin
|
||||
|
||||
# Allow "admins" IPA group members to run sudo
|
||||
# This is an "unbreak my setup" step and ought to happen by default.
|
||||
# See https://pagure.io/freeipa/issue/7538
|
||||
ipa-advise enable-admins-sudo | sh -ex
|
||||
|
||||
ipa dnsconfig-mod --forwarder=8.8.8.8
|
||||
|
||||
ln -sf ../selinux/config /etc/sysconfig/selinux
|
||||
echo 'SELINUX=permissive' > /etc/selinux/config
|
||||
|
||||
# reduce image size
|
||||
dnf clean all
|
||||
/var/lib/testvm/zero-disk.setup
|
|
@ -1,303 +0,0 @@
|
|||
#!/usr/bin/python2
|
||||
|
||||
# This file is part of Cockpit.
|
||||
#
|
||||
# Copyright (C) 2015 Red Hat, Inc.
|
||||
#
|
||||
# Cockpit is free software; you can redistribute it and/or modify it
|
||||
# under the terms of the GNU Lesser General Public License as published by
|
||||
# the Free Software Foundation; either version 2.1 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# Cockpit is distributed in the hope that it will be useful, but
|
||||
# WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
# Lesser General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU Lesser General Public License
|
||||
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
import subprocess
|
||||
import os
|
||||
import sys
|
||||
import shutil
|
||||
try:
|
||||
from urllib.request import URLopener
|
||||
except ImportError:
|
||||
from urllib import URLopener # Python 2
|
||||
import argparse
|
||||
import json
|
||||
|
||||
BASEDIR = os.path.dirname(__file__)
|
||||
|
||||
class AtomicCockpitInstaller:
|
||||
branch = None
|
||||
checkout_location = "/var/local-tree"
|
||||
repo_location = "/var/local-repo"
|
||||
rpm_location = "/usr/share/rpm"
|
||||
key_id = "95A8BA1754D0E95E2B3A98A7EE15015654780CBD"
|
||||
port = 12345
|
||||
|
||||
# Support installing random packages if needed.
|
||||
external_packages = {}
|
||||
|
||||
# Temporarily force cockpit-system instead of cockpit-shell
|
||||
packages_force_install = [ "cockpit-system",
|
||||
"cockpit-docker",
|
||||
"cockpit-kdump",
|
||||
"cockpit-networkmanager",
|
||||
"cockpit-sosreport" ]
|
||||
|
||||
def __init__(self, rpms=None, extra_rpms=None, verbose=False):
|
||||
self.verbose = verbose
|
||||
self.rpms = rpms
|
||||
self.extra_rpms = extra_rpms
|
||||
status = json.loads(subprocess.check_output(["rpm-ostree", "status", "--json"], universal_newlines=True))
|
||||
origin = None
|
||||
for deployment in status.get("deployments", []):
|
||||
if deployment.get("booted"):
|
||||
origin = deployment["origin"]
|
||||
|
||||
if not origin:
|
||||
raise Exception("Couldn't find origin")
|
||||
|
||||
self.branch = origin.split(":", 1)[-1]
|
||||
|
||||
def setup_dirs(self):
|
||||
if self.verbose:
|
||||
print("setting up new ostree repo")
|
||||
|
||||
try:
|
||||
shutil.rmtree(self.repo_location)
|
||||
except:
|
||||
pass
|
||||
|
||||
os.makedirs(self.repo_location)
|
||||
subprocess.check_call(["ostree", "init", "--repo", self.repo_location,
|
||||
"--mode", "archive-z2"])
|
||||
|
||||
if not os.path.exists(self.checkout_location):
|
||||
if self.verbose:
|
||||
print("cloning current branch")
|
||||
|
||||
subprocess.check_call(["ostree", "checkout", self.branch,
|
||||
self.checkout_location])
|
||||
|
||||
# move /usr/etc to /etc, makes rpm installs easier
|
||||
subprocess.check_call(["mv", os.path.join(self.checkout_location, "usr", "etc"),
|
||||
os.path.join(self.checkout_location, "etc")])
|
||||
|
||||
def switch_to_local_tree(self):
|
||||
if self.verbose:
|
||||
print("install new ostree commit")
|
||||
|
||||
# Not an error if this fails
|
||||
subprocess.call(["ostree", "remote", "delete", "local"])
|
||||
|
||||
subprocess.check_call(["ostree", "remote", "add", "local",
|
||||
"file://{}".format(self.repo_location),
|
||||
"--no-gpg-verify"])
|
||||
|
||||
# HACK: https://github.com/candlepin/subscription-manager/issues/1404
|
||||
subprocess.call(["systemctl", "disable", "rhsmcertd"])
|
||||
subprocess.call(["systemctl", "stop", "rhsmcertd"])
|
||||
|
||||
status = subprocess.check_output(["rpm-ostree", "status"])
|
||||
if b"local:" in status:
|
||||
subprocess.check_call(["rpm-ostree", "upgrade"])
|
||||
else:
|
||||
try:
|
||||
subprocess.check_call(["setenforce", "0"])
|
||||
subprocess.check_call(["rpm-ostree", "rebase",
|
||||
"local:{0}".format(self.branch)])
|
||||
except:
|
||||
os.system("sysctl kernel.core_pattern")
|
||||
os.system("coredumpctl || true")
|
||||
raise
|
||||
finally:
|
||||
subprocess.check_call(["setenforce", "1"])
|
||||
|
||||
def commit_to_repo(self):
|
||||
if self.verbose:
|
||||
print("commit package changes to our repo")
|
||||
|
||||
# move etc back to /usr/etc
|
||||
subprocess.check_call(["mv", os.path.join(self.checkout_location, "etc"),
|
||||
os.path.join(self.checkout_location, "usr", "etc")])
|
||||
|
||||
subprocess.check_call(["ostree", "commit", "-s", "cockpit-tree",
|
||||
"--repo", self.repo_location,
|
||||
"-b", self.branch,
|
||||
"--add-metadata-string", "version=cockpit-base.1",
|
||||
"--tree=dir={0}".format(self.checkout_location),
|
||||
"--gpg-sign={0}".format(self.key_id),
|
||||
"--gpg-homedir={0}".format(BASEDIR)])
|
||||
|
||||
def install_packages(self, packages, deps=True, replace=False):
|
||||
args = ["rpm", "-U", "--root", self.checkout_location,
|
||||
"--dbpath", self.rpm_location]
|
||||
|
||||
if replace:
|
||||
args.extend(["--replacepkgs", "--replacefiles"])
|
||||
|
||||
if not deps:
|
||||
args.append("--nodeps")
|
||||
|
||||
for package in packages:
|
||||
args.append(os.path.abspath(os.path.join(os.getcwd(), package)))
|
||||
|
||||
subprocess.check_call(args)
|
||||
|
||||
def remove_packages(self, packages):
|
||||
args = ["rpm", "-e", "--root", self.checkout_location,
|
||||
"--dbpath", self.rpm_location]
|
||||
args.extend(packages)
|
||||
subprocess.check_call(args)
|
||||
|
||||
def package_basename(self, package):
|
||||
""" only accept package with the name 'cockpit-%s-*' and return 'cockpit-%s' or None"""
|
||||
basename = "-".join(package.split("-")[:2])
|
||||
if basename.startswith("cockpit-"):
|
||||
return basename
|
||||
else:
|
||||
return None
|
||||
|
||||
def update_container(self):
|
||||
""" Install the latest cockpit RPMs in our container"""
|
||||
rpm_args = []
|
||||
for package in self.rpms:
|
||||
if 'cockpit-ws' in package or 'cockpit-dashboard' in package or 'cockpit-bridge' in package:
|
||||
rpm_args.append("/host" + package)
|
||||
extra_args = []
|
||||
for package in self.extra_rpms:
|
||||
extra_args.append("/host" + package)
|
||||
|
||||
if rpm_args:
|
||||
subprocess.check_call(["docker", "run", "--name", "build-cockpit",
|
||||
"-d", "--privileged", "-v", "/:/host",
|
||||
"cockpit/ws", "sleep", "1d"])
|
||||
if self.verbose:
|
||||
print("updating cockpit-ws container")
|
||||
|
||||
if extra_args:
|
||||
subprocess.check_call(["docker", "exec", "build-cockpit",
|
||||
"rpm", "--install", "--verbose", "--force"] + extra_args)
|
||||
|
||||
subprocess.check_call(["docker", "exec", "build-cockpit",
|
||||
"rpm", "--freshen", "--verbose", "--force"] + rpm_args)
|
||||
|
||||
# if we update the RPMs, also update the scripts, to keep them in sync
|
||||
subprocess.check_call(["docker", "exec", "build-cockpit", "sh", "-exc",
|
||||
"cp /host/var/tmp/containers/ws/atomic-* /container/"])
|
||||
|
||||
subprocess.check_call(["docker", "commit", "build-cockpit",
|
||||
"cockpit/ws"])
|
||||
subprocess.check_call(["docker", "kill", "build-cockpit"])
|
||||
subprocess.check_call(["docker", "rm", "build-cockpit"])
|
||||
|
||||
def package_basenames(self, package_names):
|
||||
""" convert a list of package names to a list of their basenames """
|
||||
return list(filter(lambda s: s is not None, map(self.package_basename, package_names)))
|
||||
|
||||
def get_installed_cockpit_packages(self):
|
||||
""" get list installed cockpit packages """
|
||||
packages = subprocess.check_output("rpm -qa | grep cockpit", shell=True, universal_newlines=True)
|
||||
|
||||
if self.verbose:
|
||||
print("installed packages: {0}".format(packages))
|
||||
|
||||
installed_packages = packages.strip().split("\n")
|
||||
return installed_packages
|
||||
|
||||
def clean_network(self):
|
||||
if self.verbose:
|
||||
print("clean network configuration:")
|
||||
subprocess.check_call(["rm", "-rf", "/var/lib/NetworkManager"])
|
||||
subprocess.check_call(["rm", "-rf", "/var/lib/dhcp"])
|
||||
|
||||
def run(self):
|
||||
# Delete previous deployment if it's present
|
||||
output = subprocess.check_output(["ostree", "admin", "status"])
|
||||
if output.count(b"origin refspec") != 1:
|
||||
subprocess.check_call(["ostree", "admin", "undeploy", "1"])
|
||||
|
||||
self.setup_dirs()
|
||||
|
||||
installed_packages = self.get_installed_cockpit_packages()
|
||||
self.remove_packages(installed_packages)
|
||||
|
||||
packages_to_install = self.package_basenames(installed_packages)
|
||||
for p in self.packages_force_install:
|
||||
if not p in packages_to_install:
|
||||
if self.verbose:
|
||||
print("adding package %s (forced)" % (p))
|
||||
packages_to_install.append(p)
|
||||
|
||||
packages_to_install = list(filter(lambda p: any(os.path.split(p)[1].startswith(base) for base in packages_to_install), self.rpms))
|
||||
|
||||
if self.verbose:
|
||||
print("packages to install:")
|
||||
print(packages_to_install)
|
||||
|
||||
if self.external_packages:
|
||||
names = self.external_packages.keys()
|
||||
if self.verbose:
|
||||
print("external packages to install:")
|
||||
print(list(names))
|
||||
|
||||
downloader = URLopener()
|
||||
for name, url in self.external_packages.items():
|
||||
downloader.retrieve(url, name)
|
||||
|
||||
self.install_packages(names, replace=True)
|
||||
|
||||
for name in names:
|
||||
os.remove(name)
|
||||
|
||||
self.install_packages(packages_to_install)
|
||||
no_deps = [x for x in self.rpms \
|
||||
if os.path.split(x)[-1].startswith("cockpit-tests") or
|
||||
os.path.split(x)[-1].startswith("cockpit-machines")]
|
||||
self.install_packages(no_deps, deps=False, replace=True)
|
||||
|
||||
# If firewalld is installed, we need to poke a hole for cockpit, so
|
||||
# that we can run firewall tests on it (change firewall-cmd to
|
||||
# --add-service=cockpit once all supported atomics ship with the
|
||||
# service file)
|
||||
if subprocess.call(["systemctl", "enable", "--now", "firewalld"]) == 0:
|
||||
subprocess.call(["firewall-cmd", "--permanent", "--add-port=9090/tcp"])
|
||||
|
||||
self.commit_to_repo()
|
||||
self.switch_to_local_tree()
|
||||
self.update_container()
|
||||
self.clean_network()
|
||||
|
||||
parser = argparse.ArgumentParser(description='Install Cockpit in Atomic')
|
||||
parser.add_argument('-v', '--verbose', action='store_true', help='Display verbose progress details')
|
||||
parser.add_argument('-q', '--quick', action='store_true', help='Build faster')
|
||||
parser.add_argument('--build', action='store_true', help='Build')
|
||||
parser.add_argument('--install', action='store_true', help='Install')
|
||||
parser.add_argument('--extra', action='append', default=[], help='Extra packages to install inside the container')
|
||||
parser.add_argument('--skip', action='append', default=[], help='Packes to skip during installation')
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.build:
|
||||
sys.stderr.write("Can't build on Atomic\n")
|
||||
sys.exit(1)
|
||||
|
||||
if args.install:
|
||||
os.chdir("build-results")
|
||||
# Force skip cockpit-dashboard
|
||||
if args.skip:
|
||||
skip = list(args.skip)
|
||||
else:
|
||||
skip = []
|
||||
skip.append("cockpit-dashboard")
|
||||
|
||||
rpms = [os.path.abspath(f) for f in os.listdir(".")
|
||||
if (f.endswith(".rpm") and not f.endswith(".src.rpm")
|
||||
and not any(f.startswith(s) for s in args.skip))]
|
||||
cockpit_installer = AtomicCockpitInstaller(rpms=rpms, extra_rpms=args.extra, verbose=args.verbose)
|
||||
cockpit_installer.run()
|
||||
|
||||
# vim: ft=python
|
|
@ -1,78 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
# This file is part of Cockpit.
|
||||
#
|
||||
# Copyright (C) 2015 Red Hat, Inc.
|
||||
#
|
||||
# Cockpit is free software; you can redistribute it and/or modify it
|
||||
# under the terms of the GNU Lesser General Public License as published by
|
||||
# the Free Software Foundation; either version 2.1 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# Cockpit is distributed in the hope that it will be useful, but
|
||||
# WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
# Lesser General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU Lesser General Public License
|
||||
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
set -ex
|
||||
|
||||
# The docker pool should grow automatically as needed, but we grow it
|
||||
# explicitly here anyway. This is hopefully more reliable.
|
||||
# Newer Fedora versions configure docker to use the root LV
|
||||
# HACK: docker falls over regularly, print its log if it does
|
||||
systemctl start docker || journalctl -u docker
|
||||
lvresize atomicos/root -l+60%FREE -r
|
||||
if lvs atomicos/docker-pool 2>/dev/null; then
|
||||
lvresize atomicos/docker-pool -l+100%FREE
|
||||
elif lvs atomicos/docker-root-lv; then
|
||||
lvresize atomicos/docker-root-lv -l+100%FREE
|
||||
fi
|
||||
|
||||
# docker images that we need for integration testing
|
||||
/var/lib/testvm/docker-images.setup
|
||||
|
||||
# Download the libssh RPM plus dependencies which we'll use for
|
||||
# package overlay. The only way to do this is via a container
|
||||
. /etc/os-release
|
||||
REPO="updates"
|
||||
if [ "$ID" = "rhel" ]; then
|
||||
subscription-manager repos --enable rhel-7-server-extras-rpms
|
||||
REPO="rhel-7-server-extras-rpms"
|
||||
ID="rhel7"
|
||||
fi
|
||||
docker run --rm --volume=/etc/yum.repos.d:/etc/yum.repos.d:z --volume=/root/rpms:/tmp/rpms:rw,z "$ID:$VERSION_ID" /bin/sh -cex "yum install -y findutils createrepo yum-utils && (cd /tmp/; yumdownloader --enablerepo=$REPO libssh) && find /tmp -name '*.$(uname -m).*rpm' | while read rpm; do mv -v \$rpm /tmp/rpms; done; createrepo /tmp/rpms"
|
||||
rm -f /etc/yum.repos.d/*
|
||||
cat >/etc/yum.repos.d/deps.repo <<EOF
|
||||
[deps]
|
||||
baseurl=file:///root/rpms
|
||||
enabled=1
|
||||
EOF
|
||||
|
||||
# fully upgrade host. Anything past this point can't touch /etc
|
||||
# Upgrade host if there is a valid upgrade available (we might be on a RC)
|
||||
if rpm-ostree upgrade --check; then
|
||||
atomic host upgrade
|
||||
# HACK - Find a better way to compute the ref.
|
||||
# https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2016-July/msg00015.html
|
||||
|
||||
checkout=$(atomic host status --json | python -c 'import json; import sys; j = json.loads(sys.stdin.readline()); print j["deployments"][0]["origin"]')
|
||||
else
|
||||
checkout=$(atomic host status --json | python -c 'import json; import sys; j = json.loads(sys.stdin.readline()); print [x for x in j["deployments"] if x["booted"]][0]["checksum"]')
|
||||
fi
|
||||
|
||||
# Checkout the just upgraded os branch since we'll use it every time
|
||||
# we build a new tree.
|
||||
|
||||
ostree checkout "$checkout" /var/local-tree
|
||||
|
||||
# reduce image size
|
||||
/var/lib/testvm/zero-disk.setup
|
||||
|
||||
# Prevent SSH from hanging for a long time when no external network access
|
||||
echo 'UseDNS no' >> /etc/ssh/sshd_config
|
||||
|
||||
# Final tweaks
|
||||
rm -rf /var/log/journal/*
|
|
@ -1,5 +0,0 @@
|
|||
FROM fedora:30
|
||||
|
||||
ADD setup.sh /setup.sh
|
||||
|
||||
RUN /setup.sh
|
|
@ -1,5 +0,0 @@
|
|||
Cockpit Base
|
||||
===========================
|
||||
|
||||
Simple base container that installs cockpit-ws dependencies. Used in testing
|
||||
and development to speed up container build times.
|
|
@ -1,26 +0,0 @@
|
|||
#! /bin/sh
|
||||
|
||||
upgrade() {
|
||||
# https://bugzilla.redhat.com/show_bug.cgi?id=1483553
|
||||
dnf -v -y update 2>err.txt
|
||||
ecode=$?
|
||||
if [ $ecode -ne 0 ] ; then
|
||||
grep -q -F -e "BDB1539 Build signature doesn't match environment" err.txt
|
||||
if [ $? -eq 0 ]; then
|
||||
set -eu
|
||||
rpm --rebuilddb
|
||||
dnf -v -y update
|
||||
else
|
||||
cat err.txt
|
||||
exit ${ecode}
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
upgrade
|
||||
|
||||
set -eu
|
||||
|
||||
dnf install -y sed findutils glib-networking json-glib libssh openssl python3
|
||||
|
||||
dnf clean all
|
|
@ -1,16 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
set -eu
|
||||
|
||||
# Download cockpit.spec, replace `npm-version` macro and then query all build requires
|
||||
curl -s https://raw.githubusercontent.com/cockpit-project/cockpit/master/tools/cockpit.spec |
|
||||
sed 's/%{npm-version:.*}/0/' |
|
||||
sed '/Recommends:/d' |
|
||||
rpmspec -D "$1" --buildrequires --query /dev/stdin |
|
||||
sed 's/.*/"&"/' |
|
||||
tr '\n' ' '
|
||||
|
||||
# support for backbranches
|
||||
if [ "$1" = "rhel 7" ] || [ "$1" = "centos 7" ]; then
|
||||
echo "golang-bin golang-src"
|
||||
fi
|
|
@ -1,35 +0,0 @@
|
|||
#!/bin/bash
|
||||
# This file is part of Cockpit.
|
||||
#
|
||||
# Copyright (C) 2016 Red Hat, Inc.
|
||||
#
|
||||
# Cockpit is free software; you can redistribute it and/or modify it
|
||||
# under the terms of the GNU Lesser General Public License as published by
|
||||
# the Free Software Foundation; either version 2.1 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# Cockpit is distributed in the hope that it will be useful, but
|
||||
# WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
# Lesser General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU Lesser General Public License
|
||||
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
|
||||
set -ex
|
||||
|
||||
# HACK: docker falls over regularly, print its log if it does
|
||||
systemctl start docker || journalctl -u docker
|
||||
|
||||
for NAME in bastion
|
||||
do
|
||||
mkdir -p "/var/tmp/containers/$NAME/rpms"
|
||||
cp -f /var/tmp/build-results/*.rpm "/var/tmp/containers/$NAME/rpms/"
|
||||
cd "/var/tmp/containers/$NAME/"
|
||||
sed -i -e "s#FROM .*#FROM cockpit/base#" Dockerfile
|
||||
docker build --build-arg OFFLINE=1 -t "cockpit/$NAME" . 1>&2;
|
||||
rm -r "/var/tmp/containers/$NAME/rpms"
|
||||
done
|
||||
|
||||
journalctl --flush || true
|
||||
journalctl --sync || killall systemd-journald || true
|
||||
rm -rf /var/log/journal/* || true
|
|
@ -1,13 +0,0 @@
|
|||
#! /bin/bash
|
||||
|
||||
set -ex
|
||||
|
||||
out=$1
|
||||
|
||||
# download cloud image; re-use a previously downloaded image
|
||||
image_url="$2"
|
||||
image=tmp/$(basename $image_url)
|
||||
mkdir -p $(dirname $image)
|
||||
[ -f "$image" ] || curl -o "$image" "$image_url"
|
||||
cp "$image" "$out"
|
||||
qemu-img resize -f qcow2 "$out" +8G
|
|
@ -1,97 +0,0 @@
|
|||
#! /bin/sh
|
||||
|
||||
set -ex
|
||||
|
||||
export DEB_BUILD_OPTIONS=""
|
||||
|
||||
do_build=
|
||||
do_install=
|
||||
stdout_dest="/dev/null"
|
||||
args=$(getopt -o "vqs:" -l "verbose,quick,skip:,build,install" -- "$@")
|
||||
eval set -- "$args"
|
||||
while [ $# -gt 0 ]; do
|
||||
case $1 in
|
||||
-v|--verbose)
|
||||
stdout_dest="/dev/stdout"
|
||||
;;
|
||||
-q|--quick)
|
||||
DEB_BUILD_OPTIONS="$DEB_BUILD_OPTIONS nocheck"
|
||||
;;
|
||||
--build)
|
||||
do_build=t
|
||||
;;
|
||||
--install)
|
||||
do_install=t
|
||||
;;
|
||||
--)
|
||||
shift
|
||||
break
|
||||
;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
tar="$1"
|
||||
|
||||
|
||||
# Build
|
||||
|
||||
if [ -n "$do_build" ]; then
|
||||
rm -rf build-results
|
||||
mkdir build-results
|
||||
resultdir=$PWD/build-results
|
||||
upstream_ver=$(ls cockpit-*.tar.gz | sed 's/^.*-//; s/.tar.gz//' | head -n1)
|
||||
|
||||
ln -sf cockpit-*.tar.gz cockpit_${upstream_ver}.orig.tar.gz
|
||||
|
||||
rm -rf cockpit-*/
|
||||
tar -xzf cockpit-*.tar.gz
|
||||
( cd cockpit-*/
|
||||
cp -rp tools/debian debian
|
||||
# put proper version into changelog, as we have versioned dependencies
|
||||
sed -i "1 s/(.*)/($upstream_ver-1)/" debian/changelog
|
||||
# Hack: Remove PCP build dependencies while pcp is not in testing
|
||||
# (https://tracker.debian.org/pcp)
|
||||
sed -i '/libpcp.*-dev/d' debian/control
|
||||
dpkg-buildpackage -S -uc -us -nc
|
||||
)
|
||||
|
||||
# Some unit tests want a real network interface
|
||||
echo USENETWORK=yes >>~/.pbuilderrc
|
||||
|
||||
# pbuilder < 0.228.6 has broken /dev/pts/ptmx permissions; affects Ubuntu < 17.04
|
||||
# see https://bugs.debian.org/841935
|
||||
if ! grep -q ptmxmode /usr/lib/pbuilder/pbuilder-modules; then
|
||||
echo "Fixing /dev/pts/ptmx mode in pbuilder"
|
||||
sed -i '/mount -t devpts none/ s/$/,ptmxmode=666,newinstance/' /usr/lib/pbuilder/pbuilder-modules
|
||||
fi
|
||||
|
||||
pbuilder build --buildresult "$resultdir" \
|
||||
--logfile "$resultdir/build.log" \
|
||||
cockpit_${upstream_ver}-1.dsc >$stdout_dest
|
||||
lintian $resultdir/cockpit_*_$(dpkg --print-architecture).changes >&2
|
||||
fi
|
||||
|
||||
# Install
|
||||
|
||||
if [ -n "$do_install" ]; then
|
||||
packages=$(find build-results -name "*.deb")
|
||||
dpkg --install $packages
|
||||
|
||||
# FIXME: our tests expect cockpit.socket to not be running after boot, only
|
||||
# after start_cockpit().
|
||||
systemctl disable cockpit.socket
|
||||
|
||||
# HACK: tuned breaks QEMU (https://launchpad.net/bugs/1774000)
|
||||
systemctl disable tuned.service 2>/dev/null || true
|
||||
|
||||
# avoid random dpkg database locks, they break our package related tests
|
||||
systemctl disable apt-daily-upgrade.timer
|
||||
|
||||
firewall-cmd --add-service=cockpit --permanent
|
||||
# not managed by NM, so enable interface manually
|
||||
firewall-cmd --zone=public --permanent --add-interface=eth1
|
||||
|
||||
journalctl --flush
|
||||
journalctl --sync || killall systemd-journald
|
||||
rm -rf /var/log/journal/*
|
||||
fi
|
|
@ -1,36 +0,0 @@
|
|||
#!/bin/bash
|
||||
set -ex
|
||||
|
||||
# This file is part of Cockpit.
|
||||
#
|
||||
# Copyright (C) 2016 Red Hat, Inc.
|
||||
#
|
||||
# Cockpit is free software; you can redistribute it and/or modify it
|
||||
# under the terms of the GNU Lesser General Public License as published by
|
||||
# the Free Software Foundation; either version 2.1 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# Cockpit is distributed in the hope that it will be useful, but
|
||||
# WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
# Lesser General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU Lesser General Public License
|
||||
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
if [ $(uname -m) = x86_64 ]; then
|
||||
docker pull busybox:latest
|
||||
docker pull busybox:buildroot-2014.02
|
||||
docker pull gcr.io/google_containers/pause:0.8.0
|
||||
docker pull k8s.gcr.io/pause-amd64:3.1
|
||||
# some aliases for different k8s variants
|
||||
docker tag k8s.gcr.io/pause-amd64:3.1 gcr.io/google_containers/pause-amd64:3.0
|
||||
docker tag k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1
|
||||
fi
|
||||
|
||||
# Download the i386 image and rename it
|
||||
if [ $(uname -m) = i686 ]; then
|
||||
docker pull i386/busybox:latest
|
||||
docker tag docker.io/i386/busybox busybox
|
||||
docker rmi docker.io/i386/busybox
|
||||
fi
|
|
@ -1,120 +0,0 @@
|
|||
#! /bin/bash
|
||||
|
||||
set -ex
|
||||
|
||||
# don't update already installed cockpit packages
|
||||
installed=$(rpm --query --all --queryformat "%{NAME}-\[0-9\]\n" "cockpit*")
|
||||
skip="cockpit-doc-[0-9]"
|
||||
if [ -n "$installed" ]; then
|
||||
skip="$skip
|
||||
$installed"
|
||||
fi
|
||||
|
||||
do_build=
|
||||
do_install=
|
||||
# we build RHEL 7.x in a CentOS mock, thus we can't parse os-release in the .spec
|
||||
mock_opts="--define='os_version_id $(. /etc/os-release; echo $VERSION_ID)'"
|
||||
args=$(getopt -o "vqs:" -l "verbose,quick,skip:,build,install,rhel,HACK-no-bootstrap-chroot" -- "$@")
|
||||
eval set -- "$args"
|
||||
while [ $# -gt 0 ]; do
|
||||
case $1 in
|
||||
-v|--verbose)
|
||||
mock_opts="$mock_opts --verbose"
|
||||
;;
|
||||
-q|--quick)
|
||||
mock_opts="$mock_opts --nocheck --define='selinux 0'"
|
||||
;;
|
||||
-s|--skip)
|
||||
skip="$skip
|
||||
$2"
|
||||
shift
|
||||
;;
|
||||
--build)
|
||||
do_build=t
|
||||
;;
|
||||
--install)
|
||||
do_install=t
|
||||
;;
|
||||
--rhel)
|
||||
# For RHEL we actually build in EPEL, which is based
|
||||
# on CentOS. On CentOS, the spec file has both
|
||||
# %centos and %rhel defined, but it gives precedence
|
||||
# to %centos, as it must. To make it produce the RHEL
|
||||
# packages, we explicitly undefine %centos here.
|
||||
mock_opts="$mock_opts --define='centos 0'"
|
||||
;;
|
||||
--HACK-no-bootstrap-chroot)
|
||||
mock_opts="$mock_opts --no-bootstrap-chroot"
|
||||
;;
|
||||
--)
|
||||
shift
|
||||
break
|
||||
;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
tar=$1
|
||||
|
||||
# Build
|
||||
|
||||
if [ -n "$do_build" ]; then
|
||||
# Some tests need a non-loopback internet address, so we allow
|
||||
# networking during build. Note that we use "--offline" below, so
|
||||
# we should still be protected against unexpected package
|
||||
# installations.
|
||||
echo "config_opts['rpmbuild_networking'] = True" >>/etc/mock/site-defaults.cfg
|
||||
# don't destroy the mock after building, we want to run rpmlint
|
||||
echo "config_opts['cleanup_on_success'] = False" >>/etc/mock/site-defaults.cfg
|
||||
# HACK: don't fall over on unavailable repositories, as we are offline
|
||||
# (https://bugzilla.redhat.com/show_bug.cgi?id=1549291)
|
||||
sed --follow-symlinks -i '/skip_if_unavailable=False/d' /etc/mock/default.cfg
|
||||
|
||||
rm -rf build-results
|
||||
srpm=$(/var/lib/testvm/make-srpm "$tar")
|
||||
LC_ALL=C.UTF-8 su builder -c "/usr/bin/mock --offline --no-clean --resultdir build-results $mock_opts --rebuild $srpm"
|
||||
|
||||
cat <<EOF >/tmp/rpmlint
|
||||
#! /bin/bash
|
||||
rm -rf /builddir/build
|
||||
if type rpmlint >/dev/null 2>&1; then
|
||||
# blacklist "E: no-changelogname-tag" rpmlint error, expected due to our template cockpit.spec
|
||||
mkdir -p ~/.config
|
||||
echo 'addFilter("E: no-changelogname-tag")' > ~/.config/rpmlint
|
||||
# we expect the srpm to be clean
|
||||
echo
|
||||
echo '====== rpmlint on srpm ====='
|
||||
rpmlint /builddir/build/SRPMS/*.src.rpm
|
||||
# this still has lots of errors, run it for information only
|
||||
echo
|
||||
echo '====== rpmlint binary rpms (advisory) ====='
|
||||
rpmlint /builddir/build/RPMS/ || true
|
||||
else
|
||||
echo '====== skipping rpmlint check, not installed ====='
|
||||
fi
|
||||
EOF
|
||||
chmod +x /tmp/rpmlint
|
||||
su builder -c "/usr/bin/mock --offline --copyin /tmp/rpmlint /var/tmp/rpmlint"
|
||||
su builder -c "/usr/bin/mock --offline --shell /var/tmp/rpmlint"
|
||||
fi
|
||||
|
||||
# Install
|
||||
|
||||
if [ -n "$do_install" ]; then
|
||||
packages=$(find build-results -name "*.rpm" -not -name "*.src.rpm" | grep -vG "$skip")
|
||||
rpm -U --force $packages
|
||||
|
||||
if type firewall-cmd > /dev/null 2> /dev/null; then
|
||||
systemctl start firewalld
|
||||
firewall-cmd --add-service=cockpit --permanent
|
||||
fi
|
||||
|
||||
# Make sure we clean out the journal
|
||||
journalctl --flush
|
||||
journalctl --sync || killall systemd-journald
|
||||
rm -rf /var/log/journal/*
|
||||
rm -rf /var/lib/NetworkManager/dhclient-*.lease
|
||||
fi
|
||||
|
||||
if [ -n "$do_build" ]; then
|
||||
su builder -c "/usr/bin/mock --clean"
|
||||
fi
|
|
@ -1,46 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
# Kubernetes is delivered in a non-functional state on Fedora and similar operating systems
|
||||
# The following commands are needed to get it running.
|
||||
|
||||
cd /etc/kubernetes/
|
||||
|
||||
cat <<EOF > openssl.conf
|
||||
oid_section = new_oids
|
||||
[new_oids]
|
||||
[req]
|
||||
encrypt_key = no
|
||||
string_mask = nombstr
|
||||
req_extensions = v3_req
|
||||
distinguished_name = v3_name
|
||||
[v3_name]
|
||||
commonName = kubernetes
|
||||
[v3_req]
|
||||
basicConstraints = CA:FALSE
|
||||
subjectAltName = @alt_names
|
||||
[alt_names]
|
||||
DNS.1 = kubernetes
|
||||
DNS.2 = kubernetes.default
|
||||
DNS.3 = kubernetes.default.svc
|
||||
DNS.4 = kubernetes.default.svc.cluster.local
|
||||
IP.1 = 127.0.0.1
|
||||
IP.2 = 10.254.0.1
|
||||
EOF
|
||||
|
||||
openssl genrsa -out ca.key 2048
|
||||
openssl req -x509 -new -nodes -key ca.key -days 3072 -out ca.crt -subj '/CN=kubernetes'
|
||||
openssl genrsa -out server.key 2048
|
||||
openssl req -config openssl.conf -new -key server.key -out server.csr -subj '/CN=kubernetes'
|
||||
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 3072 -extensions v3_req -extfile openssl.conf
|
||||
# make keys readable for "kube" group and thus for kube-apiserver.service on newer OSes
|
||||
if getent group kube >/dev/null; then
|
||||
chgrp kube ca.key server.key
|
||||
chmod 640 ca.key server.key
|
||||
fi
|
||||
|
||||
echo -e '{"user":"admin"}\n{"user":"scruffy","readonly": true}' > /etc/kubernetes/authorization
|
||||
echo -e 'fubar,admin,10101\nscruffy,scruffy,10102' > /etc/kubernetes/passwd
|
||||
|
||||
echo 'KUBE_API_ARGS="--service-account-key-file=/etc/kubernetes/server.key --client-ca-file=/etc/kubernetes/ca.crt --tls-cert-file=/etc/kubernetes/server.crt --tls-private-key-file=/etc/kubernetes/server.key --basic-auth-file=/etc/kubernetes/passwd --authorization-mode=ABAC --authorization-policy-file=/etc/kubernetes/authorization"' >> apiserver
|
||||
echo 'KUBE_CONTROLLER_MANAGER_ARGS="--root-ca-file=/etc/kubernetes/ca.crt --service-account-private-key-file=/etc/kubernetes/server.key"' >> controller-manager
|
||||
|
|
@ -1,33 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
set -eu
|
||||
|
||||
tar=$1
|
||||
|
||||
version=$(echo "$1" | sed -n 's|.*cockpit-\([^ /-]\+\)\.tar\..*|\1|p')
|
||||
if [ -z "$version" ]; then
|
||||
echo "make-srpm: couldn't parse version from tarball: $1"
|
||||
exit 2
|
||||
fi
|
||||
|
||||
# We actually modify the spec so that the srpm is standalone buildable
|
||||
modify_spec() {
|
||||
sed -e "/^Version:.*/d" -e "1i\
|
||||
%define wip wip\nVersion: $version\n"
|
||||
}
|
||||
|
||||
tmpdir=$(mktemp -d $PWD/srpm-build.XXXXXX)
|
||||
tar xaf "$1" -O cockpit-$version/tools/cockpit.spec | modify_spec > $tmpdir/cockpit.spec
|
||||
|
||||
rpmbuild -bs \
|
||||
--quiet \
|
||||
--define "_sourcedir $(dirname $1)" \
|
||||
--define "_specdir $tmpdir" \
|
||||
--define "_builddir $tmpdir" \
|
||||
--define "_srcrpmdir `pwd`" \
|
||||
--define "_rpmdir $tmpdir" \
|
||||
--define "_buildrootdir $tmpdir/.build" \
|
||||
$tmpdir/cockpit.spec
|
||||
|
||||
rpm --qf '%{Name}-%{Version}-%{Release}.src.rpm\n' -q --specfile $tmpdir/cockpit.spec | head -n1
|
||||
rm -rf $tmpdir
|
Binary file not shown.
Binary file not shown.
|
@ -1,50 +0,0 @@
|
|||
#!/bin/bash
|
||||
|
||||
# This file is part of Cockpit.
|
||||
#
|
||||
# Copyright (C) 2016 Red Hat, Inc.
|
||||
#
|
||||
# Cockpit is free software; you can redistribute it and/or modify it
|
||||
# under the terms of the GNU Lesser General Public License as published by
|
||||
# the Free Software Foundation; either version 2.1 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# Cockpit is distributed in the hope that it will be useful, but
|
||||
# WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
|
||||
# Lesser General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU Lesser General Public License
|
||||
# along with Cockpit; If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
# We don't want to delete the pbuilder caches since we need them during build.
|
||||
# Mock with --offline and dnf is sometimes happy without caches, and with yum it
|
||||
# never is, so we provide an option to also leave the mock caches in place.
|
||||
#
|
||||
# We also want to keep cracklib since otherwise password quality
|
||||
# checks break on Debian.
|
||||
|
||||
if [ -f /root/.skip-zero-disk ]; then
|
||||
echo "Skipping zero-disk.setup as /root/.skip-zero-disk exists"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
keep="! -path /var/cache/pbuilder ! -path /var/cache/cracklib ! -path /var/cache/tomcat"
|
||||
while [ $# -gt 0 ]; do
|
||||
case $1 in
|
||||
--keep-mock-cache)
|
||||
keep="$keep ! -path /var/cache/mock"
|
||||
;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
if [ -d "/var/cache" ]; then
|
||||
find /var/cache/* -maxdepth 0 -depth -name "*" $keep -exec rm -rf {} \;
|
||||
fi
|
||||
rm -rf /var/tmp/*
|
||||
rm -rf /var/log/journal/*
|
||||
|
||||
dd if=/dev/zero of=/root/junk || true
|
||||
sync
|
||||
rm -f /root/junk
|
|
@ -1,3 +0,0 @@
|
|||
BOOTPROTO="dhcp"
|
||||
DEVICE="eth0"
|
||||
ONBOOT="yes"
|
|
@ -1,3 +0,0 @@
|
|||
BOOTPROTO="none"
|
||||
DEVICE="eth1"
|
||||
ONBOOT="no"
|
|
@ -1,4 +0,0 @@
|
|||
#! /bin/bash
|
||||
|
||||
BASE=$(dirname $0)
|
||||
$BASE/virt-install-fedora "$1" x86_64 "http://mirror.centos.org/centos/7/os/x86_64/"
|
|
@ -1,2 +0,0 @@
|
|||
#!/bin/sh
|
||||
# By default this does nothing
|
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue