Commit Graph

284 Commits

Author SHA1 Message Date
Hans-Christoph Steiner 65cd3e3b0a
gitlab-ci: tag docker buildserver image with "stretch"
This should make the migration to bullseye easier because we'll have a
fallback stretch image available as needed.

!1012
2022-04-20 15:14:36 +02:00
Hans-Christoph Steiner 3027dc014e
gitlab-ci: fix sudo test in "fdroid build" job 2022-04-18 14:48:16 +02:00
FestplattenSchnitzel 9721b60d7c
Use buildserver image from fdroidserver
This is derived from the "fdroid build" job in fdroiddata
2022-04-18 13:46:10 +02:00
FestplattenSchnitzel 14ab546365
Fix CI for users with uppercase letters in username 2022-02-26 18:15:58 +01:00
Hans-Christoph Steiner 55b3905951 gitlab-ci: fix servergitmirrors by using Debian packages
this is like:
* ca5ebcb30f
* 972bfe346f
* 23bd62c31c
* !1035
* !1034
* !1029
* !1028
2022-02-03 16:41:57 +01:00
Hans-Christoph Steiner ca5ebcb30f
gitlab-ci: fix plugin_fetchsrclibs by using Debian packages
* https://gitlab.com/eighthave/fdroidserver/-/jobs/1961701458

Collecting pynacl>=1.0.1 (from paramiko->fdroidserver==2.1a0)
  Downloading 27582568be88ce80e4db396c8986da/PyNaCl-1.5.0.tar.gz (3.4MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  Complete output from command /builds/eighthave/fdroidserver/env/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-xokvr6uk --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools>=40.8.0 wheel "cffi>=1.4.1; python_implementation != 'PyPy'":
  Collecting setuptools>=40.8.0
    Using cached 0dd4c796053d79f8bc8b1b6c8e6217/setuptools-60.5.0-py3-none-any.whl
  Collecting wheel
    Using cached 003e593296ee2bca4f6d0fe55c6d00/wheel-0.37.1-py2.py3-none-any.whl
  Collecting cffi>=1.4.1
    Downloading 92de7e12172e0c4530e6eb2ba9282a/cffi-1.15.0.tar.gz (484kB)
  Collecting pycparser (from cffi>=1.4.1)
    Downloading 5f610ebe4298517d912eb1c76e1a53/pycparser-2.21-py2.py3-none-any.whl (118kB)
  Building wheels for collected packages: cffi
    Running setup.py bdist_wheel for cffi: started
    Running setup.py bdist_wheel for cffi: finished with status 'error'
    Complete output from command /builds/eighthave/fdroidserver/env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-djg9jc8p/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-d1knhl7l --python-tag cp37:
    unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
    unable to execute 'x86_64-linux-gnu-gcc': No such file or directory

        No working compiler found, or bogus compiler options passed to
        the compiler from Python's standard "distutils" module.  See
        the error messages above.  Likely, the problem is not related
        to CFFI but generic to the setup.py of any Python package that
        tries to compile C code.  (Hints: on OS/X 10.8, for errors about
        -mno-fused-madd see http://stackoverflow.com/questions/22313407/
        Otherwise, see https://wiki.python.org/moin/CompLangPython or
        the IRC channel #python on irc.libera.chat.)

        Trying to continue anyway.  If you are trying to install CFFI from
        a build done in a different context, you can ignore this warning.

    running bdist_wheel
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.7
    creating build/lib.linux-x86_64-3.7/cffi
    copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/lock.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/recompiler.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/__init__.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/cparser.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/verifier.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/api.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/commontypes.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/error.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/model.py -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/_embedding.h -> build/lib.linux-x86_64-3.7/cffi
    copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.7/cffi
    running build_ext
    building '_cffi_backend' extension
    creating build/temp.linux-x86_64-3.7
    creating build/temp.linux-x86_64-3.7/c
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/ffi -I/usr/include/libffi -I/builds/eighthave/fdroidserver/env/include -I/usr/include/python3.7m -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.7/c/_cffi_backend.o
    unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
    Failed building wheel for cffi
    Running setup.py clean for cffi
  Failed to build cffi
  Installing collected packages: setuptools, wheel, pycparser, cffi
    Running setup.py install for cffi: started
      Running setup.py install for cffi: finished with status 'error'
      Complete output from command /builds/eighthave/fdroidserver/env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-djg9jc8p/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-ek80c81s/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-xokvr6uk --compile --install-headers /builds/eighthave/fdroidserver/env/include/site/python3.7/cffi:
      unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
      unable to execute 'x86_64-linux-gnu-gcc': No such file or directory

          No working compiler found, or bogus compiler options passed to
          the compiler from Python's standard "distutils" module.  See
          the error messages above.  Likely, the problem is not related
          to CFFI but generic to the setup.py of any Python package that
          tries to compile C code.  (Hints: on OS/X 10.8, for errors about
          -mno-fused-madd see http://stackoverflow.com/questions/22313407/
          Otherwise, see https://wiki.python.org/moin/CompLangPython or
          the IRC channel #python on irc.libera.chat.)

          Trying to continue anyway.  If you are trying to install CFFI from
          a build done in a different context, you can ignore this warning.

      running install
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-3.7
      creating build/lib.linux-x86_64-3.7/cffi
      copying cffi/backend_ctypes.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/vengine_gen.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/vengine_cpy.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/pkgconfig.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/lock.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/recompiler.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/__init__.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/cparser.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/ffiplatform.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/verifier.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/cffi_opcode.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/setuptools_ext.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/api.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/commontypes.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/error.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/model.py -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/_cffi_include.h -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/parse_c_type.h -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/_embedding.h -> build/lib.linux-x86_64-3.7/cffi
      copying cffi/_cffi_errors.h -> build/lib.linux-x86_64-3.7/cffi
      running build_ext
      building '_cffi_backend' extension
      creating build/temp.linux-x86_64-3.7
      creating build/temp.linux-x86_64-3.7/c
      x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/ffi -I/usr/include/libffi -I/builds/eighthave/fdroidserver/env/include -I/usr/include/python3.7m -c c/_cffi_backend.c -o build/temp.linux-x86_64-3.7/c/_cffi_backend.o
      unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
      error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

      ----------------------------------------
  Command "/builds/eighthave/fdroidserver/env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-djg9jc8p/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-ek80c81s/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-xokvr6uk --compile --install-headers /builds/eighthave/fdroidserver/env/include/site/python3.7/cffi" failed with error code 1 in /tmp/pip-install-djg9jc8p/cffi/

  ----------------------------------------
Command "/builds/eighthave/fdroidserver/env/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-xokvr6uk --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools>=40.8.0 wheel "cffi>=1.4.1; python_implementation != 'PyPy'"" failed with error code 1 in None
2022-01-13 21:22:25 +01:00
Hans-Christoph Steiner 839f5b109a
gitlab-ci: publish buildserver image to Docker Registry 2022-01-13 21:21:26 +01:00
FestplattenSchnitzel 565d9e4994
Format setup.py with Black 2021-12-16 20:34:38 +01:00
FestplattenSchnitzel 23a4135ccd
Compile translation files on install
Solution based on https://stackoverflow.com/questions/40051076/compile-translation-files-when-calling-setup-py-install
2021-12-16 20:13:24 +01:00
Hans-Christoph Steiner 778e600269 gitlab-ci: enable pyjks in lint_mypy job now that it is in backports 2021-12-07 10:24:27 +01:00
Jochen Sprickerhof 6756f842e4 [CI] install python3-matplotlib from apt 2021-11-19 18:17:36 +01:00
Jochen Sprickerhof 46abffa0fa [CI] install python3-pil from apt in servergitmirrors
New version would need to be compiled, otherwise.
2021-10-23 15:02:57 +02:00
Jochen Sprickerhof b8285ebd0e [CI] Install python3-pil from apt
New version would need to be compiled, otherwise.
2021-10-23 14:33:06 +02:00
Jochen Sprickerhof 972bfe346f Fix servergitmirrors ci 2021-10-14 14:40:31 +02:00
Jochen Sprickerhof 23bd62c31c [CI] fetchsrclibs use apt cffi 2021-10-14 13:55:51 +02:00
Jochen Sprickerhof 60532f0912 Ignore failing Windows CI 2021-10-14 13:40:19 +02:00
FestplattenSchnitzel ef20e1b329 Add CI job "black" for code format checks
The CI job will fail, if a file needs to be reformatted.
Only files already well formatted will be checked at the moment.
2021-09-25 15:45:37 +00:00
Hans-Christoph Steiner 3b95d3de64
update: AllowedAPKSigningKeys metadata to enforce APK signers
This field lets you specify which signing certificates should be
trusted for APKs in a binary repo.
2021-08-05 16:43:48 +02:00
Hans-Christoph Steiner 7ed75abc2f
gitlab-ci: use CVC for `fdroid build` job for more reliably runs 2021-07-01 14:45:39 +02:00
Benedikt Brückmann 1e943a22df rewrite docstrings to match numpy style guide 2021-06-25 06:28:47 +02:00
Hans-Christoph Steiner 660520e3e1
gitlab-ci: switch ubuntu_lts_ppa job to keyserver.ubuntu.com
fixes:

Executing: /tmp/apt-key-gpghome.ZwePp5ymqm/gpg.1.sh --keyserver hkp://pool.sks-keyservers.net --recv-key 9AAC253193B65D4DF1D0A13EEC4632C79C5E0151
gpg: keyserver receive failed: No name

https://gitlab.com/fdroid/fdroidserver/-/jobs/1374517971
2021-06-24 21:34:29 +02:00
Hans-Christoph Steiner fc989d94b3
gitlab-ci: add Windows job 2021-06-18 11:12:27 +02:00
linsui 466c8a44d7 add vscode settings 2021-06-18 06:50:50 +00:00
Hans-Christoph Steiner ac86a2af89
gitlab-ci: use mypy from Debian/bullseye to prevent random breakage
mypy 0.901 was released and everything broke:
https://gitlab.com/linsui/fdroidserver/-/jobs/1330206567

My point is to reduce the number of false job fails like this one. We
have a lot of checkers running, I really think they need to prove they are
adding value before we invest any time maintaining them. mypy is still
"wait and see" in terms of the adding any value.

!951
2021-06-09 12:35:50 +02:00
Hans-Christoph Steiner bcd3e89614
gitlab-ci: fix whitespace 2021-06-09 12:25:00 +02:00
Hans-Christoph Steiner 9d44fa7919
gitlab-ci: auto-generate merge request when NDK release found
Following the pattern of the gradle bot, this will check the transparency
log for any new NDK release.  If there are any, it will make a merge
request from @fdroid-bot.
2021-05-25 17:06:30 +02:00
Hans-Christoph Steiner e4a7726033 gitlab-ci: fix archlinux docker image 2021-05-13 17:20:41 +02:00
Benedikt Brückmann 6791c3a550 Only publish pages on default branch 2021-05-10 14:31:51 +00:00
Benedikt Brückmann 245aeec972 Publish docu to GL Pages 2021-05-10 14:31:51 +00:00
Benedikt Brückmann bdcd843d75 Publish docu to GL Pages 2021-05-10 14:31:51 +00:00
Benedikt Brückmann a15581faa9 Draft: set up docs based on docstrings and publish HTML versions 2021-05-10 14:31:51 +00:00
Hans-Christoph Steiner 022d0796d8 gitlab-ci: use latest alpine/bandit to fix CVE flagged by safety 2021-05-10 16:05:59 +02:00
Hans-Christoph Steiner 634864f206
gitlab-ci: document jobs 2021-04-16 09:40:12 +02:00
Hans-Christoph Steiner 5346ea3c19 gitlab-ci: prevent dualing linters: pyflakes vs mypy
* pyflakes says: "typing.Dict' imported but unused"
* mypy says: "Name 'Dict' is not defined"
2021-04-14 23:18:05 +02:00
Hans-Christoph Steiner 9f60516fcb gitlab-ci: fix broken download link for fdroiddata tarball
fdroiddata!8730
fdroiddata!8732
rfp@f49178ea0aed808607f8d6d753285b101e53f0be
2021-04-06 10:41:59 +02:00
Hans-Christoph Steiner cf1fa84c77
gitlab-ci: rm HTML index gen test from job, fails on stretch
```console
  Downloading 0d6598b1ceaba8ddceeec3fa59997f/slimit-0.8.1.zip (88kB)
Collecting tinycss2>=0.4 (from html5print->fdroidserver==2.1a0.dev0)
  Downloading 63bf697a7c9e93963d6f5790d77d82/tinycss2-1.1.0-py3-none-any.whl
tinycss2 requires Python '>=3.6' but the running Python is 3.5.3
```
2021-04-06 10:23:17 +02:00
Benedikt Brückmann 8f0712e89b Add a mypy run to the pipeline 2021-04-02 08:57:56 +00:00
Hans-Christoph Steiner 20c299358d gitlab-ci: modernize ubuntu_lts_ppa job using updates to PPA
The PPA now provides apksigner that does not need binfmt, and defusedxml is
a hard Depends:.
2021-03-24 11:20:00 +01:00
Hans-Christoph Steiner 67e65ce2ea purge complete-ci-tests, those are better covered in .gitlab-ci.yml 2021-03-22 23:36:04 +01:00
Hans-Christoph Steiner 93ca13fa16 run-tests: remove broken test that never ran due to a syntax error 2021-03-22 22:31:29 +01:00
Hans-Christoph Steiner cc3b88ab30 gradlew-fdroid: use gradle zipballs from local cachedir, if present 2021-03-11 08:40:34 +01:00
Hans-Christoph Steiner d75ba81be0
gitlab-ci: arch job needs to build python-cryptography 2021-02-09 09:26:07 +01:00
Benedikt Brückmann b1bde9e407 * Skip tidy check if html5print is not available
* install test dependencies on arch
2021-02-08 12:25:02 +01:00
Hans-Christoph Steiner c887c14be1 gitlab-ci: use python3-cryptography to avoid building rust sources 2021-02-08 11:40:41 +01:00
Hans-Christoph Steiner 3d536d4515
gitlab-ci: use python3-cryptography to avoid building rust sources 2021-02-08 10:59:48 +01:00
Hans-Christoph Steiner e4087f1798 gitlab-ci: hide massively verbose messages from sdkmanager 2021-01-28 22:25:33 +01:00
Hans-Christoph Steiner 87477cd910
gitlab-ci: gradle job needs git history to find what changed 2021-01-25 11:05:22 +01:00
Hans-Christoph Steiner d133b38f22 fdroid-bot creates merge request when new gradle versions are found 2021-01-25 00:26:14 +01:00
Hans-Christoph Steiner eec0d3a336 gitlab-ci: gradle job generates required changes for new versions 2021-01-24 16:58:38 +01:00
Hans-Christoph Steiner aebf70a763 tests: run pip tests using fdroid that pip installed 2021-01-23 20:13:17 +01:00
Hans-Christoph Steiner f6b5f74ec6 gitlab-ci: drop support for Ubuntu/xenial, test on bionic now
xenial will become obsolete in April
2021-01-23 13:36:23 +01:00
Jochen Sprickerhof 7cf64ea48f Run test from sdist source 2021-01-15 19:57:32 +01:00
Hans-Christoph Steiner 40197295ae gitlab-ci: remove refs: section to hopefully get reliable job starts
"fdroid build" and "plugin_fetchsrclibs" totally failed to run here:
https://gitlab.com/eighthave/fdroidserver/-/pipelines/240160903
2021-01-11 22:13:51 +01:00
Hans-Christoph Steiner 4b7d29097d gitlab-ci: new job to test plugins API + `fdroid fetchsrclibs` 2021-01-11 20:58:50 +01:00
Hans-Christoph Steiner 21b0a6b463 gitlab-ci: update metadata_v0 test for parsing change
`Builds:` is directly in the .yml metadata, and the internal representation
now uses the same "Builds" name.  Reading in .txt metadata files used the
name "builds" for that.  So this test now uses "Builds" as canonical.
2020-12-15 08:55:05 +01:00
Hans-Christoph Steiner ba854cbc0f index: fix GitLab Raw URLs with gitlab.com and recent versions
GitLab seems to be moving to always having "-" as the first path segment in
all the project URLs.  So the URL without a "-" is now a redirect.
2020-12-15 08:30:48 +01:00
Hans-Christoph Steiner 8329be2d22 gitlab-ci: make `fdroid build` job actually build apps 2020-12-10 10:54:47 +01:00
Hans-Christoph Steiner 15a6e2d3a0 gitlab-ci: switch archived ci-images-server to ci-images-base
ci-images-server was created separately from ci-images-base back in the day
when we had to support Debian/stretch without stretch-backports.  Those
days are gone now, so ci-images-server has been archived and should no
longer be used.
2020-12-04 11:25:02 +01:00
Hans-Christoph Steiner ed0102754a gitlab-ci: remove rules:, they create weird "pipelines" 2020-12-04 11:23:22 +01:00
Hans-Christoph Steiner 4b9297dfa6 gitlab-ci: only run arch/pip job on final merged commits
Each of these takes a while to run, and this one rarely catches
issues separately from the main jobs.
2020-12-03 21:48:37 +01:00
Hans-Christoph Steiner 00f5ff8762 gitlab-ci: remove workaround, Debian's apksigner now works w/o binfmt 2020-12-03 16:21:08 +01:00
Jochen Sprickerhof df160170ca Run test on sdist 2020-12-03 13:34:42 +01:00
Hans-Christoph Steiner a1df5ef86a gitlab-ci: try rules:changes: to limit "fdroiddata fdroid build" runs
https://docs.gitlab.com/ee/ci/yaml/#using-onlychanges-with-pipelines-for-merge-requests
2020-11-10 17:00:58 +01:00
Hans-Christoph Steiner 10d5aa6bc4
gitlab-ci: test 'fdroid build' CI job setup 2020-11-09 14:55:09 +01:00
Hans-Christoph Steiner b2b6e62b8d gitlab-ci: fix fedora test failing on Python setup
Avoid having to compile matplotlib and its C dependencies.
2020-11-02 15:38:46 +01:00
Hans-Christoph Steiner e7edd96a02
gitlab-ci: use Alpine 3.11 for lint_format_safety_bandit_checks
This should provide a current version of pip to make safety stop
complaining.

3.12/3.11 breaks bandit, probably because of Python 3.8
2020-10-01 13:21:47 +02:00
Hans-Christoph Steiner af4a2ab736 gitlab-ci: speed up test runs that do not need git history
GIT_DEPTH sets how many commits of history to clone in CI Jobs.
gitlab.com defaults to 50 with a max of 1000.  The metadata_v0 job is
the only job that needs history, and it needs more than 50.  So this
sets the default to 1, then metadata_v0 to 1000.

https://docs.gitlab.com/ee/ci/pipelines/settings.html#git-shallow-clone
2020-09-17 15:25:56 +02:00
Hans-Christoph Steiner 59018a887b gitlab-ci: ensure android-23 is present for `fdroid build` test
This test builds https://gitlab.com/fdroid/ci-test-app, which uses android-23
2020-09-17 15:25:56 +02:00
Marcus Hoffmann f6b7572b10 fix fedora test
minimum build tools version is determined by apksigner now.
2020-09-10 18:59:39 +02:00
Marcus Hoffmann 2367461465 tests: debian: apksigner is required for the tests to run now
We need to use a shell wrapper for apksigner though because docker and
binfmt don't play well together
2020-09-10 18:59:39 +02:00
Marcus Hoffmann d9a6bfb0a9 CI: install pyjks as dependency for tests 2020-08-24 21:11:55 +02:00
Hans-Christoph Steiner 62c8fd5999
add Liberapay: field with username as data
Liberapay was originally included using a numeric ID, since they had
not yet finalized the public URLs.  Now it is a username.  So this
logic prefers the username in Liberapay: field, and keeps the old
LiberapayID: to ease migration.  LiberapayID: will not override
Liberapay:.  Clients are expected to prefer Liberapay: over LiberapayID:
2020-06-16 15:35:28 +02:00
Hans-Christoph Steiner bde65aa54d gitlab-ci: switch metadata_v0 test to commit that supports only .yml
37f37ebd88
2020-06-10 10:44:27 +02:00
Hans-Christoph Steiner 37f37ebd88
use default accepted_formats since all the files are .yml anyway 2020-06-10 10:43:21 +02:00
Hans-Christoph Steiner cfa88a5335 gitlab-ci: fix binfmt support for focal to run apksigner
This manually mounts the binfmt_misc dir if its not present.
It seems the Ubuntu/focal release stopped auto-mounting binfmt_misc:
https://bugs.launchpad.net/binfmt-support/+bug/1878413
2020-05-14 11:55:09 +02:00
Hans-Christoph Steiner 0700242416 gitlab-ci: use latest pylint to avoid safety error about vuln 2020-05-11 17:45:33 +02:00
Hans-Christoph Steiner 052e22284b gitlab-ci: show clear error message when one step of job fails 2020-05-11 17:40:21 +02:00
Jochen Sprickerhof 86beac22e2 Use libarchive instead of the Python implementation 2020-04-15 18:27:13 +00:00
Marcus 9d24f2e4a7 add opencollective metadata and index field 2020-03-10 14:56:03 +00:00
Hans-Christoph Steiner ed46afe262
gitlab-ci: ensure git is installed for pip_install job 2020-02-20 16:40:31 +01:00
Hans-Christoph Steiner 3de2d0f56f add basic test suite for gradlew-fdroid
!707
fdroiddata#6216

The se.manyver app is licensed MPL, the files came from:
81d247a6cd
2020-02-13 22:32:51 +01:00
Hans-Christoph Steiner 0fa1f91a23
gitlab-ci: long timeout and many retries for pip installs 2020-01-31 15:38:05 +01:00
Hans-Christoph Steiner 3df276cc3c
fix all bandit B310 urllib_urlopen
"Audit url open for permitted schemes. Allowing use of ‘file:’’ or custom
schemes is often unexpected."

https://bandit.readthedocs.io/en/latest/blacklists/blacklist_calls.html#b310-urllib-urlopen
2020-01-31 15:37:30 +01:00
Hans-Christoph Steiner d8f3d94997
gitlab-ci: remove dscanner exclusions from bandit 2020-01-31 15:37:28 +01:00
Hans-Christoph Steiner cca78114cb
gitlab-ci: fedora no longer installs difftools by default 2019-11-05 15:30:34 +01:00
Hans-Christoph Steiner 8d3512763d
gitlab-ci: use a template for a complete apt CI setup
# Conflicts:
#	.gitlab-ci.yml
2019-09-23 11:50:51 +02:00
Hans-Christoph Steiner 0e40387805
gitlab-ci: switch debian/testing back to pure testing, no sid
The sid packages were needed as a temporary workaround while the new
androguard packages were settling into Debian.
2019-09-23 11:33:00 +02:00
Hans-Christoph Steiner 25548023e0
gitlab-ci: check gradle checksums against official list 2019-09-12 14:18:07 +02:00
Hans-Christoph Steiner a9b8687e94 gitlab-ci: the ubuntu_lts test also tests the PPA 2019-07-02 22:17:06 +02:00
Hans-Christoph Steiner e2351f6c53 gitlab-ci: move pip job to Xenial, Trusty is over 2019-07-02 22:17:06 +02:00
Michael Pöhn 716f84ec5e use actually working bandit version when running tests on alpine 2019-05-24 23:14:20 +02:00
Taco 457cf22361 Added newer ndks, gradles, latest sdk-license, and update java 1.8 version 2019-03-30 17:10:21 -04:00
Michael Pöhn 102340ec5a fix fedora ci tests: install @development-tools
Sometime pip dependencies required a C compiler because they need to
compile something during installation.
2019-03-18 16:01:34 +01:00
Hans-Christoph Steiner 7133cede89 gitlab-ci: add cache, extend timeouts/retries to stabilize Fedora job
This is happening too often:
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: Error downloading packages:
  Curl error (28): Timeout was reached for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f29&arch=x86_64 [Connection timed out after 30001 milliseconds]
ERROR: Job failed: exit code 1

https://gitlab.com/fdroid/fdroidserver/-/jobs/152719443

# Conflicts:
#	.gitlab-ci.yml
2019-01-29 13:40:55 +01:00
Hans-Christoph Steiner 3b84a82728 gitlab-ci: fix locale checks in pip_install job 2019-01-29 13:01:25 +01:00
Hans-Christoph Steiner 6925083e57 gitlab-ci: compile locales using gettext and babel 2019-01-28 14:42:29 +01:00
Hans-Christoph Steiner 9a524fa85d bump RELEASE_COMMIT_ID for fixing metadata_v0 test
fdroid/fdroidserver!564
2018-12-06 12:30:46 +01:00
Hans-Christoph Steiner e10c12ffc4 gitlab-ci: fix tests on Fedora
* install `which` and `find` commands used in ./tests/run-tests
* sdkmanager on Fedora needs JAVA_HOME to be set
2018-12-06 12:15:27 +01:00
Hans-Christoph Steiner d9e9cc20aa gitlab-ci: install python3-defusedxml in debian_testing and ubuntu_lts
fdroid/fdroidserver!578
2018-09-19 16:48:00 +02:00
Hans-Christoph Steiner 0cd1e0b172 gitlab-ci: include fdroid in bandit scans 2018-09-07 10:34:56 +02:00
Hans-Christoph Steiner 3ffe2860f3 gitlab-ci: add 'bandit' security scanner to all runs
bandit is used by Radically Open Security and is part of the GitLab Ultimate
Static Application Security Testing (SAST) suite.

https://docs.gitlab.com/ee/user/project/merge_requests/sast.html
2018-08-29 17:48:06 +02:00
Michael Pöhn 14730be812 bump RELEASE_COMMIT_ID for fixing metadata_v0 test 2018-08-02 18:18:07 +02:00
Hans-Christoph Steiner 60ee69b8bd gitlab-ci: pylint<2.0 workaround to avoid typed-ast's gcc requirement
To keep those tests light and small, no gcc please!
* https://gitlab.com/eighthave/fdroidserver/-/jobs/82274815
2018-07-18 17:11:21 +02:00
Hans-Christoph Steiner e3bd293f43 gitlab-ci: point to fixed NoSourceSince commit 2018-07-10 23:48:25 +02:00
Hans-Christoph Steiner 6ca09e1bb2 gitlab-ci: update metadata_v0 test for latest metadata fields
This is a lot easier than trying to do some elaborate multiline sed regexp!

https://gitlab.com/fdroid/fdroidserver/merge_requests/529#note_86955227
2018-07-10 17:26:07 +02:00
Hans-Christoph Steiner a87df29135 gitlab-ci: pep8 has been replaced by pycodestyle 2018-05-29 11:28:08 +02:00
Hans-Christoph Steiner 14127bf418 gitlab-ci: combine all lint/syntax/safety checks into a single job
This should make it easier to accept merge requests where there are only
cosmetic problems with them.  pep8/pylint/pyflakes runs can then be disabled
in the 'test' job by not installing the in the ci-images-server base image.
2018-05-25 12:12:40 +02:00
Hans-Christoph Steiner 74fb07b302 gitlab-ci: switch pyup_io_safety_check to Alpine to be lighter 2018-05-25 09:54:08 +02:00
Hans-Christoph Steiner 6570e85a2b gitlab-ci: new test case of pip installs on Ubuntu/trusty
Ubuntu/trusty is used as the base image for CI systems like Travis, as well
as Microsoft Subsystem for Linux.  SO we need to provide working options.
2018-05-14 15:25:33 +02:00
Hans-Christoph Steiner 54b21a6d22 move pylint run to standalone gitlab-ci job 2018-05-14 15:25:33 +02:00
Hans-Christoph Steiner 3c9cc59c38 gitlab-ci: include fdroiddata yml files in metadata_v0 test 2018-05-14 15:25:33 +02:00
Hans-Christoph Steiner 0bd276de1c gitlab-ci: add new security scanner pyup.io/safety
https://pyup.io/safety/
2018-05-14 15:25:33 +02:00
Hans-Christoph Steiner 21a18cf26b gitlab-ci: fix ubuntu_lts test now that its bionic rather than xenial
The new ubuntu:latest image is not set up for non-interactive, so this
gitlab-ci job now needs to do that.
2018-05-04 15:11:25 +02:00
Hans-Christoph Steiner 02107cc5bc gitlab-ci: make ubuntu_lts job test of fdroid/fdroidserver PPA
Now that androguard is working, there should be no need for a specific aapt
version.  The aapt included in Ubuntu LTS should always work fine when
androguard handles the bulk of the work.
2018-03-09 12:00:31 +01:00
Hans-Christoph Steiner 01a73071c7 gitlab-ci: set metadata_v0 test to use 1.0.2 as the baseline 2018-02-23 22:48:44 +01:00
Hans-Christoph Steiner f420a037d5 gitlab-ci: remove apt-get progress dumps from build log 2018-02-22 15:08:54 +01:00
Hans-Christoph Steiner 0e9252db37 gitlab-ci: try to download PPA keys until they succeed
This download occasionally fails, so this keeps retrying till it succeeds.
The CI job has a time limit, so no need to figure out an exit condition.
2018-02-22 15:08:54 +01:00
Hans-Christoph Steiner 89498208fc gitlab-ci: test against latest build-tools 27.0.3 2018-02-22 15:08:54 +01:00
Hans-Christoph Steiner 11bed21071 gitlab-ci: androguard from Debian/unstable til it stabilizes 2018-02-22 15:08:54 +01:00
Hans-Christoph Steiner 9f553186e8 gitlab-ci: switch debian_testing to use androguard 2018-02-20 16:30:06 +01:00
Hans-Christoph Steiner 05eef5a454 metadata: add new Translation app field for URL for contributions
This is a field requested from a bunch of translators so they can easily
find where to translate apps:
https://forum.f-droid.org/t/translation-field-for-app-entries/1403/5
2018-02-05 15:45:12 +01:00
Hans-Christoph Steiner b170ef7ce8 fix metadata test: remove timeout= from comparison 2018-01-30 11:20:30 +01:00
Hans-Christoph Steiner 513c95894c build: remove unused, unmaintained Kivy build method
This code has never been used and contains some insecure uses of shell=True
Building Kivy apps should be done with the buildozer=yes method.  The
buildozer method should probably be moved to a provisioner once that is in
place.
2018-01-23 23:16:05 +01:00
Hans-Christoph Steiner 22563bdf17 gitlab-ci: make metadata_v0 test work even when tags are missing
This uses the commit ID of the release tags, rather than the release tag
itself so that contributor forks do not need to include the tags in them
for this test to work.

The COMMIT_ID should be bumped after each release, so that the list of sed
hacks needed does not continuously grow.
2018-01-22 14:00:20 +01:00
Hans-Christoph Steiner fba4fa6ca6 get minimum aapt version from fdroidserver/common.py for CI tests 2018-01-03 14:54:30 +01:00
Hans-Christoph Steiner c8e32c2802 gitlab-ci: and one last stoopid error in debian_testing target 2017-12-29 16:06:45 +01:00
Hans-Christoph Steiner c7762ad7d7 gitlab-ci: fix syntax error 2017-12-29 15:35:04 +01:00
Hans-Christoph Steiner 6816ef15dd gitlab-ci: temp disable some tests on Debian/testing
build-tools needs an update before these tests will work again.
2017-12-29 15:29:28 +01:00
Hans-Christoph Steiner b70ede6761 gitlab-ci: set Fedora build to use a supported build-tools version
Using 25.0.2, not all of the permissions were being output:

--- /builds/fdroid/fdroidserver/tests/repo/index.xml	2017-12-28 22:33:53.624704459 +0000
+++ repo/index.xml	2017-12-28 22:41:52.207849097 +0000
@@ -35,7 +35,6 @@
 			<added>2017-12-22</added>
 			<sig>2d337e40aef77564bf62781ac424595c</sig>
 			<permissions>ACCESS_NETWORK_STATE,ACCESS_WIFI_STATE,CHANGE_WIFI_MULTICAST_STATE,INTERNET,READ_EXTERNAL_STORAGE,WRITE_EXTERNAL_STORAGE</permissions>
-			<uses-permission maxSdkVersion="18" name="android.permission.READ_EXTERNAL_STORAGE"/>
 			<uses-permission maxSdkVersion="18" name="android.permission.WRITE_EXTERNAL_STORAGE"/>
 			<uses-permission-sdk-23 maxSdkVersion="27" name="android.permission.REQUEST_IGNORE_BATTERY_OPTIMIZATIONS"/>
 		</package>
2017-12-29 15:12:49 +01:00
Hans-Christoph Steiner 8b9b18ceed fix metadata_v0 tests
The old metadata format didn't know anything of LiberapayID, so no need to
check for it.
2017-12-12 16:54:35 +01:00
Hans-Christoph Steiner 05abbfbabd gitlab-ci: move sdist test run to new fedora job
A full run of the test suite takes quite a bit of time.  This removes one
of the 3 runs from the main 'tests' job, and puts it into the Fedora job.
That test run is mostly to make sure the setup.py and source tarball are
correctly, so that doesn't affect merge requests very often.

This also tests `pip install --user`, which was not really being tested
before.
2017-12-06 20:20:17 +01:00
Hans-Christoph Steiner dda9c8b774 gitlab-ci: add `pip install` test on Arch Linux 2017-12-06 20:20:17 +01:00
Hans-Christoph Steiner 1b1475c982 gitlab-ci: add test runs on Ubuntu, Debian/testing, Fedora 2017-12-06 12:30:47 +01:00
Hans-Christoph Steiner 30c0d5f4d5 gitlab-ci: ensure release tag is present for metadata_v0 tests
Forks won't necessarily have all the tags, so this ensures that they are
present.
2017-09-15 11:35:30 +02:00
Marcus Hoffmann baef08725b CI: bump fdroid version to test against to 0.8 remove legacy handling
Bump to latest released version, remove all handling of things that
changed in a backward incompatible way before.
2017-08-31 21:06:34 +02:00
Hans-Christoph Steiner 619c66e1dc gitlab-ci: filter new build fields in metadata_v0 test 2017-07-19 16:24:28 +02:00
Hans-Christoph Steiner dfb07808d3 support configing buildserver VM per-build with sudo=
This adds the 'sudo' build field, which is just a script that is run as
root.  For more info, see the issue that this closes:

refs #318
closes #317
2017-07-06 17:29:23 +02:00
Hans-Christoph Steiner df99c85ca6 support manually adding per-build antiFeatures in metadata
For cases like the OpenVPN vuln that was recently announced, it is useful
for fdroiddata maintainers to be able to mark builds that have known
vulnerabilities.
2017-07-06 12:25:48 +02:00
Hans-Christoph Steiner b29fcde16b gitlab-ci: remove hacks need to support Debian/jessie 2017-07-04 14:53:21 +02:00
Hans-Christoph Steiner 607d1d2553 gitlab-ci: switch to new Debian/stretch image
ci-images-server!1
2017-07-03 20:41:26 +02:00
Hans-Christoph Steiner f2432f7fa4 gitlab-ci: apt upgrade so that tests run with current updates
The MD5 signature stuff was failing in tests because the CI image was
using a quite old version of Java's jarsigner, which had not yet disabled
MD5.
2017-07-03 10:02:51 +02:00
Hans-Christoph Steiner 7f9f47496f gitlab-ci: fix metadata_v0 test to run on the right commits
Checking out master will often mean its testing the wrong commit, since
merge requests rarely are in master.
2017-06-28 23:14:57 +02:00
Hans-Christoph Steiner 6e72ffb043 gitlab-ci: fix metadata_v0 test for new buildozer build flag
Just remove it, since 0.7.0 does not know about that build flag.
2017-06-22 16:33:11 +02:00
Hans-Christoph Steiner 5e7e17e800 gitlab-ci: properly quote colons 2017-06-20 20:27:23 +02:00
Hans-Christoph Steiner 4939b8a530 gitlab-ci: ignore blank Description in metadata_v0 test
We'll eventually be moving to having the internal representation use a
'' or None when there is no Description, so no use in fixing this.
2017-06-20 18:15:12 +02:00
Hans-Christoph Steiner 2c6945dac7 gitlab-ci: add index v0 metadata parsing test
This test is very handy for making sure the old index.xml v0 format does
not inadvertantly change.
2017-05-30 21:13:49 +02:00
Hans-Christoph Steiner e58ad330f4 encode filenames as bytes to handle all locale setups
This was failing on environments that did not have any LANG or LC_* locale
variables set.  This is a valid setup, and is common in headless setups, so
it needs to be handled.

This also adds a new pass of the test suite without the locale env vars set
so that this situation is also tests on gitlab-ci, not only gpjenkins.

The error this caused was:
UnicodeEncodeError: 'ascii' codec can't encode characters in position 6-18: ordinal not in range(128)
2017-04-03 20:24:00 +02:00
Hans-Christoph Steiner 36272656fc gitlab-ci: workaround "ImportError: No module named 'packaging'"
https://github.com/pypa/setuptools/issues/937

fdroid/ci-images#1
2017-02-06 17:28:07 +01:00
Daniel Martí 926aafa58b Bump CI image, now with build-tools 25.0.2 2016-12-23 17:46:09 +01:00
Daniel Martí 2e96fc1cd9 CI: use new docker image with texinfo
docs/gendocs.sh uses makeinfo.
2016-10-31 14:02:39 +00:00