Commit Graph

6064 Commits

Author SHA1 Message Date
Marcus Hoffmann 637c73a882 unpin pyasn1_modules
It currently  runs fine on archlinux with pyasn1-modules 0.2.8.
And guix just reported packaging it with 0.2.2
2020-08-28 17:46:53 +02:00
Hans-Christoph Steiner d595948616
jenkins-test: remove rewritemeta, it is well covered elsewhere
The jenkins.debian.net tests are brittle enough as it is.

[skip ci] - this doesn't run on gitlab-ci
2020-08-26 10:40:15 +02:00
Marcus Hoffmann eb57723096 fix accidentally changed hash of gradle 6.5.1
This was changed in 394ad8f863 when
deleting 6.6 from makebuildserver but deleting one line up instead of
down.
2020-08-25 23:15:25 +02:00
Marcus Hoffmann bc6fa986d1 fix vcs test
It doesn't need a valid sdk_path, so just set it to something.
The test was failing when ANDROID_HOME wasn't set in the env.
2020-08-25 23:09:28 +02:00
Marcus Hoffmann 394ad8f863 add gradle 6.6.1 2020-08-25 23:06:21 +02:00
Marcus e169238c60 Merge branch 'extlib_scanignore' into 'master'
minor bugfixes

Closes #795, #796, and #759

See merge request fdroid/fdroidserver!771
2020-08-25 19:53:44 +00:00
Marcus Hoffmann 34717fe88a fix typoes 2020-08-25 21:32:54 +02:00
Marcus Hoffmann 059ebd4bc9 tests: add check that we trigger a scanner error without setting it as extlib 2020-08-25 21:32:29 +02:00
Marcus Hoffmann d07b4123e9 scanner: docstrings for handleproblem functions 2020-08-25 21:32:29 +02:00
Marcus Hoffmann a656be82ae update: calculate added date for an app over all apks
This was accidentally changed in !756 because the functionality was
hidden in `apply_info_from_latest_apk` which is a less than stellar name
for something that also applies infos from app->apk and in this case did
apply info from *oldest* apk->app.
So instead move that into a separate step.

Note: This restores the previous behaviour. There's discussion in #801
on further changes to make the added date also work for repos which
don't keep an archive at all.
2020-08-25 21:32:29 +02:00
Marcus Hoffmann a301a1ba93 add test for correct `added` date for apps 2020-08-25 21:32:29 +02:00
Marcus Hoffmann d5311fff09 vcs: don't fail when git remote set-head fails
There's valid use-cases for setups where set-head --auto fails. This
happens when building an app from a gitlab CI checkout where no remote
tracking branches are setup.

This isn't really a fatal error. When a remote HEAD exists we'll
continue setting it and if none exists and something requires this being
set up later on (either a build script or fdroid checkupdates) then
we'll fail later on with "origin/HEAD not being known to git".
By not failing early we allow the majority of use-cases that don't need
a remote HEAD to continue with just a warning.

The setup in which this can be reproduced is as follows:
(This is roughly what gitlab runner does when setting up a git checkout
for CI)
- mkdir test && cd test
- git init
- git remote add https://gitlab.com/Bubu/fdroidclassic.git
- git fetch --all
- git checkout db0d2a9a5d1d89101a344169013ac5d518185f31
- mkdir nested_repo && cd nested_repo
- git clone .. .
- git remote set-head origin --auto
  > error: Cannot determine remote HEAD
2020-08-25 21:32:29 +02:00
Marcus Hoffmann fce4721695 add a vcs.TestCase 2020-08-25 21:32:29 +02:00
Marcus Hoffmann 03e723b1af fix crash when scanner wants to remove the same file more than once
A file can be flagged for multiple problems (i.e. multiple unknown maven
repos in one build.gradle file that is included in a scandelete path).

The scanner will try to delete it once for every problem detected, we
don't really care, as long as the file is gone.

fixes fdroid/fdroidserver#759
2020-08-25 21:32:29 +02:00
Marcus Hoffmann 61736f3f50 scanner: add test for #759 2020-08-25 21:32:29 +02:00
Marcus Hoffmann 78491a0a5b add used extlibs to scanignore path
fixes fdroid/fdroidserver#795
2020-08-25 21:32:29 +02:00
Marcus Hoffmann f46e99a5c4 test for #796
The extlib test is in build because it tests the interaction between
prepare_source with a later scan as it is run from build.py
2020-08-25 21:31:07 +02:00
Marcus 8801d37649 Merge branch 'publish_json' into 'master'
Some publish.py improvements

Closes #820

See merge request fdroid/fdroidserver!787
2020-08-25 12:33:39 +00:00
Marcus Hoffmann 882f8cfe19 test_check_for_key_collisions: test with an actual collision
Genrated with this script:
https://gitlab.com/fdroid/fdroidserver/-/merge_requests/787#note_401275883
2020-08-24 21:19:59 +02:00
Marcus Hoffmann d9a6bfb0a9 CI: install pyjks as dependency for tests 2020-08-24 21:11:55 +02:00
Marcus Hoffmann a114c73c2d publish: factor out the signing key creation into a method 2020-08-24 19:34:08 +02:00
Marcus Hoffmann 7813a17cf8 publish: extract a few functions out of main
publish is currently not reusable from other modules as everything is
happening in main. It's also not testable from python unittests.

There's already a function for getting the key_alias, so we can use
that.

Introduce tests for the split out functions.
2020-08-24 19:34:08 +02:00
Marcus Hoffmann eaca3d5faa publish: better json reporting
* newKeyAliases wasn't providing any useful information
* generatedKeys now contains the used keyalias as well
* signedApks now also records the used keyalias for each apk
2020-08-24 19:34:08 +02:00
Marcus Hoffmann ca86c18e33 publish: reformat 2020-08-24 19:34:08 +02:00
Marcus 3954814a3b Merge branch 'post-txt-refactor' into 'master'
Post .txt refactor

See merge request fdroid/fdroidserver!786
2020-08-24 17:05:39 +00:00
Hans-Christoph Steiner 0b92e60266 handle file type detection using Pythonic methods
This ditches the custom common.get_extension() for straight core Python
methods.  This should make the code closer to Python conventions.  For
example, pathlib also includes the "." in the extension it returns.
2020-08-24 17:05:49 +02:00
Hans-Christoph Steiner da31120b5a build: refactor missing srclibs error reporting for only .yml 2020-08-24 17:05:49 +02:00
Hans-Christoph Steiner bd1f05e370 Merge branch 'yaml-is-king' into 'master'
remove all code for handling txt metadata

See merge request fdroid/fdroidserver!772
2020-08-24 14:57:19 +00:00
Michael Pöhn 5c3db9a7cc rewritemeta: overwrite existing metadata only if no exception occurred 2020-08-20 20:40:15 +02:00
Michael Pöhn c45ef453fd remove SUPPORTED_FORMATS list from rewritemeta 2020-08-20 20:40:15 +02:00
Michael Pöhn fac033314a fix a comment and a warning 2020-08-20 20:40:15 +02:00
Michael Pöhn f5a5fffb10 purge accepted_formats from config 2020-08-20 20:40:15 +02:00
Michael Pöhn 40cbbd3173 update changelog: remove txt metadata support 2020-08-20 20:40:15 +02:00
Michael Pöhn f8bc51399e remove txt from nightly default supported metadata formats 2020-08-20 20:40:15 +02:00
Michael Pöhn 4dc503ed28 remove txt form accepted format list 2020-08-20 20:40:15 +02:00
Michael Pöhn fd2cfb0c7d remove code for copying txt srclibs to buildserver 2020-08-20 20:40:15 +02:00
Michael Pöhn 681a27546c fix linking to yml metadata in wiki 2020-08-20 20:40:15 +02:00
Michael Pöhn cb368a674c remove txt srclib support 2020-08-20 20:40:15 +02:00
Michael Pöhn c8f25c2652 remove txt metadata support 2020-08-20 20:40:15 +02:00
Michael Pöhn 2ec90bb490 remove support for rewriting to txt 2020-08-20 20:40:15 +02:00
Michael Pöhn bc1398f594 remove json metadata support 2020-08-20 20:40:15 +02:00
Hans-Christoph Steiner 29eea3f739 Merge branch 'androguard_minsdk' into 'master'
common: use androguard to figure out minSDK version

Closes #816

See merge request fdroid/fdroidserver!785
2020-08-20 12:45:28 +00:00
Marcus Hoffmann ba28b44ae7 common: use androguard to figure out minSDK version
Closes: #816
2020-08-19 19:49:39 +02:00
Marcus 89dac62b10 Merge branch 'fix-gradlew-fdroid' into 'master'
gradlew-fdroid: fix random failures due to empty $line, closes #815

Closes #815

See merge request fdroid/fdroidserver!784
2020-08-19 15:12:26 +00:00
Hans-Christoph Steiner f8a1b45df5 gradlew-fdroid: fix random failures due to empty $line, closes #815 2020-08-19 16:28:58 +02:00
Hans-Christoph Steiner 6258a939e6 Merge branch 'smartcard_hsm' into 'master'
switch publish to common.sign_apk and support smartcard HSMs for app signing

See merge request fdroid/fdroidserver!782
2020-08-17 10:09:35 +00:00
Marcus Hoffmann a8e9653b96 update: make --create-key work with a HSM 2020-08-14 17:27:58 +02:00
Marcus Hoffmann 004d13a48a make publish and update work with a smartcard HSM
Followup to fdroid/fdroidserver!779.

We need to add smartcardoptions to every call to keytool and jarsigner
as well as handle when keypass not being required and not allowed for
pkcs11 keystores.
2020-08-14 17:27:58 +02:00
Marcus Hoffmann 066978cbcf publish: use common signing method
This is currently still jarsigner based but will at least use sha256
when possible
2020-08-14 15:07:04 +02:00
Marcus Hoffmann 6128f93d26 publish: keystore "NONE" is a special case and doesn't need to exist 2020-08-14 15:06:33 +02:00