Commit Graph

6079 Commits

Author SHA1 Message Date
Marcus Hoffmann f6b7572b10 fix fedora test
minimum build tools version is determined by apksigner now.
2020-09-10 18:59:39 +02:00
Marcus Hoffmann 2367461465 tests: debian: apksigner is required for the tests to run now
We need to use a shell wrapper for apksigner though because docker and
binfmt don't play well together
2020-09-10 18:59:39 +02:00
Marcus Hoffmann 89f63b3e1c tests: use yaml.Loader on older yaml versions 2020-09-10 18:38:43 +02:00
Marcus Hoffmann 709f4c9b18 pickle -> yaml rename 2020-09-10 18:38:43 +02:00
Marcus Hoffmann e613b65098 we need FullLoader for one test, we are dumping custom objects 2020-09-10 18:38:43 +02:00
Marcus Hoffmann 9bf0758f19 make update.Testcase tests work standalone 2020-09-10 18:38:43 +02:00
Marcus Hoffmann 4cd96d4a9f use yaml safeloader in tests
Try to use CSafeLoader when possible because its significantly faster.
Use the normal Safeloader otherwise. (This mirrors the non-test code
behaviour)
2020-09-10 14:18:36 +02:00
Hans-Christoph Steiner 06766ba48b Merge branch 'signing' into 'master'
sign using apksigner

Closes #827 and #634

See merge request fdroid/fdroidserver!736
2020-09-10 11:49:02 +00:00
Marcus Hoffmann dfecdcc1bd set minimum apksigner version to 26.0.2 2020-09-10 13:20:01 +02:00
Marcus Hoffmann 145ba9db54 fix apksigner smartcardoptions
apksigner documents the options as --ks-provider-class and --ks-provider-arg
those seem to be accepted but fail when actually making a signature with
weird internal exceptions. The new options actually work.
From: https://geoffreymetais.github.io/code/key-signing/#scripting
2020-09-09 22:51:08 +02:00
Marcus Hoffmann 74af61f255 remove providerName from default smartcardoptions
apksigner doesn't recognize the SunPKCS11-OpenSC set via providerName
Neither jarsigner nor apksigner need this to work.
2020-09-09 22:50:58 +02:00
Marcus Hoffmann 768a91370c publish: use apksigner for signing apks with targetSDK>=30
This makes apksigner a hard requirement of the signing procedure.
We'll first try to find a globally installed version from PATH and if
that's not available fall back to using a version from build-tools.

Future TODO: always sign with apksigner, blocked on signature transplant
support for apksigv2/v3

Closes fdroid/fdroidserver#634
Closes fdroid/fdroidserver#827
2020-09-09 18:09:28 +02:00
Marcus b1128325b5 Merge branch 'scanner_fix_ci' into 'master'
scanner: check for `test` in path relative to build dir

See merge request fdroid/fdroidserver!790
2020-08-30 18:32:42 +00:00
Marcus Hoffmann c90a72e14d scanner: check for `test` in path relative to build dir
This was degrading scanner errors to warnings whenever the path from the
current running install of fdroidserver contained test, as has been
happening in the CI image builder:

https://gitlab.com/fdroid/ci-images-server/-/blob/master/test#L6
2020-08-30 20:09:19 +02:00
Marcus 9ae8d7ae6c Merge branch 'pyasn1_modules' into 'master'
unpin pyasn1_modules

See merge request fdroid/fdroidserver!788
2020-08-28 16:07:56 +00:00
Marcus Hoffmann 637c73a882 unpin pyasn1_modules
It currently  runs fine on archlinux with pyasn1-modules 0.2.8.
And guix just reported packaging it with 0.2.2
2020-08-28 17:46:53 +02:00
Hans-Christoph Steiner d595948616
jenkins-test: remove rewritemeta, it is well covered elsewhere
The jenkins.debian.net tests are brittle enough as it is.

[skip ci] - this doesn't run on gitlab-ci
2020-08-26 10:40:15 +02:00
Marcus Hoffmann eb57723096 fix accidentally changed hash of gradle 6.5.1
This was changed in 394ad8f863 when
deleting 6.6 from makebuildserver but deleting one line up instead of
down.
2020-08-25 23:15:25 +02:00
Marcus Hoffmann bc6fa986d1 fix vcs test
It doesn't need a valid sdk_path, so just set it to something.
The test was failing when ANDROID_HOME wasn't set in the env.
2020-08-25 23:09:28 +02:00
Marcus Hoffmann 394ad8f863 add gradle 6.6.1 2020-08-25 23:06:21 +02:00
Marcus e169238c60 Merge branch 'extlib_scanignore' into 'master'
minor bugfixes

Closes #795, #796, and #759

See merge request fdroid/fdroidserver!771
2020-08-25 19:53:44 +00:00
Marcus Hoffmann 34717fe88a fix typoes 2020-08-25 21:32:54 +02:00
Marcus Hoffmann 059ebd4bc9 tests: add check that we trigger a scanner error without setting it as extlib 2020-08-25 21:32:29 +02:00
Marcus Hoffmann d07b4123e9 scanner: docstrings for handleproblem functions 2020-08-25 21:32:29 +02:00
Marcus Hoffmann a656be82ae update: calculate added date for an app over all apks
This was accidentally changed in !756 because the functionality was
hidden in `apply_info_from_latest_apk` which is a less than stellar name
for something that also applies infos from app->apk and in this case did
apply info from *oldest* apk->app.
So instead move that into a separate step.

Note: This restores the previous behaviour. There's discussion in #801
on further changes to make the added date also work for repos which
don't keep an archive at all.
2020-08-25 21:32:29 +02:00
Marcus Hoffmann a301a1ba93 add test for correct `added` date for apps 2020-08-25 21:32:29 +02:00
Marcus Hoffmann d5311fff09 vcs: don't fail when git remote set-head fails
There's valid use-cases for setups where set-head --auto fails. This
happens when building an app from a gitlab CI checkout where no remote
tracking branches are setup.

This isn't really a fatal error. When a remote HEAD exists we'll
continue setting it and if none exists and something requires this being
set up later on (either a build script or fdroid checkupdates) then
we'll fail later on with "origin/HEAD not being known to git".
By not failing early we allow the majority of use-cases that don't need
a remote HEAD to continue with just a warning.

The setup in which this can be reproduced is as follows:
(This is roughly what gitlab runner does when setting up a git checkout
for CI)
- mkdir test && cd test
- git init
- git remote add https://gitlab.com/Bubu/fdroidclassic.git
- git fetch --all
- git checkout db0d2a9a5d1d89101a344169013ac5d518185f31
- mkdir nested_repo && cd nested_repo
- git clone .. .
- git remote set-head origin --auto
  > error: Cannot determine remote HEAD
2020-08-25 21:32:29 +02:00
Marcus Hoffmann fce4721695 add a vcs.TestCase 2020-08-25 21:32:29 +02:00
Marcus Hoffmann 03e723b1af fix crash when scanner wants to remove the same file more than once
A file can be flagged for multiple problems (i.e. multiple unknown maven
repos in one build.gradle file that is included in a scandelete path).

The scanner will try to delete it once for every problem detected, we
don't really care, as long as the file is gone.

fixes fdroid/fdroidserver#759
2020-08-25 21:32:29 +02:00
Marcus Hoffmann 61736f3f50 scanner: add test for #759 2020-08-25 21:32:29 +02:00
Marcus Hoffmann 78491a0a5b add used extlibs to scanignore path
fixes fdroid/fdroidserver#795
2020-08-25 21:32:29 +02:00
Marcus Hoffmann f46e99a5c4 test for #796
The extlib test is in build because it tests the interaction between
prepare_source with a later scan as it is run from build.py
2020-08-25 21:31:07 +02:00
Marcus 8801d37649 Merge branch 'publish_json' into 'master'
Some publish.py improvements

Closes #820

See merge request fdroid/fdroidserver!787
2020-08-25 12:33:39 +00:00
Marcus Hoffmann 882f8cfe19 test_check_for_key_collisions: test with an actual collision
Genrated with this script:
https://gitlab.com/fdroid/fdroidserver/-/merge_requests/787#note_401275883
2020-08-24 21:19:59 +02:00
Marcus Hoffmann d9a6bfb0a9 CI: install pyjks as dependency for tests 2020-08-24 21:11:55 +02:00
Marcus Hoffmann a114c73c2d publish: factor out the signing key creation into a method 2020-08-24 19:34:08 +02:00
Marcus Hoffmann 7813a17cf8 publish: extract a few functions out of main
publish is currently not reusable from other modules as everything is
happening in main. It's also not testable from python unittests.

There's already a function for getting the key_alias, so we can use
that.

Introduce tests for the split out functions.
2020-08-24 19:34:08 +02:00
Marcus Hoffmann eaca3d5faa publish: better json reporting
* newKeyAliases wasn't providing any useful information
* generatedKeys now contains the used keyalias as well
* signedApks now also records the used keyalias for each apk
2020-08-24 19:34:08 +02:00
Marcus Hoffmann ca86c18e33 publish: reformat 2020-08-24 19:34:08 +02:00
Marcus 3954814a3b Merge branch 'post-txt-refactor' into 'master'
Post .txt refactor

See merge request fdroid/fdroidserver!786
2020-08-24 17:05:39 +00:00
Hans-Christoph Steiner 0b92e60266 handle file type detection using Pythonic methods
This ditches the custom common.get_extension() for straight core Python
methods.  This should make the code closer to Python conventions.  For
example, pathlib also includes the "." in the extension it returns.
2020-08-24 17:05:49 +02:00
Hans-Christoph Steiner da31120b5a build: refactor missing srclibs error reporting for only .yml 2020-08-24 17:05:49 +02:00
Hans-Christoph Steiner bd1f05e370 Merge branch 'yaml-is-king' into 'master'
remove all code for handling txt metadata

See merge request fdroid/fdroidserver!772
2020-08-24 14:57:19 +00:00
Michael Pöhn 5c3db9a7cc rewritemeta: overwrite existing metadata only if no exception occurred 2020-08-20 20:40:15 +02:00
Michael Pöhn c45ef453fd remove SUPPORTED_FORMATS list from rewritemeta 2020-08-20 20:40:15 +02:00
Michael Pöhn fac033314a fix a comment and a warning 2020-08-20 20:40:15 +02:00
Michael Pöhn f5a5fffb10 purge accepted_formats from config 2020-08-20 20:40:15 +02:00
Michael Pöhn 40cbbd3173 update changelog: remove txt metadata support 2020-08-20 20:40:15 +02:00
Michael Pöhn f8bc51399e remove txt from nightly default supported metadata formats 2020-08-20 20:40:15 +02:00
Michael Pöhn 4dc503ed28 remove txt form accepted format list 2020-08-20 20:40:15 +02:00