Commit Graph

877 Commits

Author SHA1 Message Date
Hans-Christoph Steiner 9d2cc1ecc5 fix pylint C1803: 'icons_src == {}' can be simplified to 'not icons_src' as an empty dict is falsey (use-implicit-booleaness-not-comparison) 2023-02-02 16:02:49 +01:00
linsui 5858d8fcb4 remove liberapayID 2023-01-16 17:02:56 +00:00
FestplattenSchnitzel aa71dd57f6
update.TestCase: Avoid creating metadata with duplicate keys 2023-01-15 12:52:41 +01:00
FestplattenSchnitzel 671a264dfe
Use ruamel.yaml in metadata.py and metadata.TestCase
This is to read metadata based on YAML 1.2 rather than 1.1.
2023-01-15 12:52:41 +01:00
Hans-Christoph Steiner 947d94e0a9
deploy: support GitLab Job Artifacts as a mirror 2022-11-16 20:16:32 +01:00
Hans-Christoph Steiner f24613b701
index: fix git-mirror size check for GitLab Pages
The test case had the wrong folder setup, this was confirmed on a production
repo setup.
2022-11-16 20:16:25 +01:00
Hans-Christoph Steiner c2567d71d1
nightly: fail if *-nightly git repo is not publicly available 2022-11-16 20:16:21 +01:00
Hans-Christoph Steiner 15bd7057f0 nightly: add tests 2022-11-16 16:50:14 +01:00
Hans-Christoph Steiner 1c5506ae05 nightly: support OpenSSL 3.0 with Paramiko
OpenSSL 3.0 changed the default output format from PKCS#1 to PKCS#8,
which paramiko does not support.

https://www.openssl.org/docs/man3.0/man1/openssl-rsa.html#traditional
https://github.com/paramiko/paramiko/issues/1015
2022-11-16 14:40:08 +01:00
Hans-Christoph Steiner 01f7dfd0b3 index-v2.jar is not a valid file, remove references
entry.jar is the signed file, it references index-v2.json.
2022-11-15 13:22:09 +00:00
linsui d25995c084 Set ArchivePolicy based on VercodeOperation/signature 2022-11-15 18:50:16 +08:00
Michael Pöhn 24d88705fa 🔍 add `scanner_signature_sources` config option
This adds the option to configure which set of signatures `fdroid
scanner` should use, by configuring it in `config.yml`. It allows
fetching signatures in our custom json format. It also adds 3 additional
sources: 'suss', 'exodus', 'etip'
2022-11-15 09:20:26 +00:00
Hans-Christoph Steiner bd51b2e99f build: read VM CPUs/RAM from builder/Vagrantfile
makebuildserver.config.py is no more, builder/Vagrantfile is now where the
CPU and memory is configured for the buildserver VM.  In fact, that was
always the actual place, the makebuildserver.config.py thing was just
confused.

This should have been part of !1222
2022-11-15 08:57:41 +01:00
Jochen Sprickerhof 1bb963d768
jarsigner: allow weak signatures
openjdk-11 11.0.17 in Debian unstable fails to verify weak signatures:

jarsigner -verbose -strict -verify tests/signindex/guardianproject.jar

         131 Fri Dec 02 20:10:00 CET 2016 META-INF/MANIFEST.MF
         252 Fri Dec 02 20:10:04 CET 2016 META-INF/1.SF
        2299 Fri Dec 02 20:10:04 CET 2016 META-INF/1.RSA
           0 Fri Dec 02 20:09:58 CET 2016 META-INF/
 m  ?  48743 Fri Dec 02 20:09:58 CET 2016 index.xml

  s = signature was verified
  m = entry is listed in manifest
  k = at least one certificate was found in keystore
  ? = unsigned entry

- Signed by "EMAILADDRESS=root@guardianproject.info, CN=guardianproject.info, O=Guardian Project, OU=FDroid Repo, L=New York, ST=New York, C=US"
    Digest algorithm: SHA1 (disabled)
    Signature algorithm: SHA1withRSA (disabled), 4096-bit key

WARNING: The jar will be treated as unsigned, because it is signed with a weak algorithm that is now disabled by the security property:

  jdk.jar.disabledAlgorithms=MD2, MD5, RSA keySize < 1024, DSA keySize < 1024, SHA1 denyAfter 2019-01-01, include jdk.disabled.namedCurves
2022-11-14 17:59:56 +01:00
Jochen Sprickerhof 24630dfe68
Drop old getsig.java test
getsig.java was replaced by a Python implementation in 6e2d0a9e (2014)
and the test was only there to compare the results for the transition.

Dropping this as it no longer works starting with 11.0.17+8.
2022-11-14 17:58:37 +01:00
Jochen Sprickerhof 557fe87d44 Run shell scripts with -e (Closes: #1035)
Make sudo, init prebuild, build and Prepare fields lists and only
concatenate them with '; ' before execution. This allows arbitrary
commands inside the fileds (even && and ';') as we don't need to split
the commands again for rewritemeta.
2022-11-03 12:25:31 +00:00
Hans-Christoph Steiner 97f3e78fa8 fixup from 8b484b37bd in !1240 2022-10-31 10:56:02 +01:00
Hans-Christoph Steiner 99e0c28d37 run black on tests/updates.TestCase to fix lint failure 2022-10-31 10:19:43 +01:00
Hans-Christoph Steiner 8b484b37bd
add tests for Fastlane changelogs including default.txt 2022-10-31 09:42:19 +01:00
linsui 618e2c7d72 change VercodeOperation to TYPE_LIST 2022-10-27 19:11:37 +08:00
Jochen Sprickerhof 5e1bdfc278
Fix ipfs test config 2022-10-24 11:43:47 +02:00
FC Stegerman d1ddd525c1
net.download_file(): retry on errors 2022-10-22 23:15:13 +02:00
Hans-Christoph Steiner 2b5c35829f tests/run-tests: fix "fatal: transport 'file' not allowed"
https://gitlab.com/eighthave/fdroidserver/-/jobs/3202418899
2022-10-20 17:21:43 +02:00
linsui 68b58c043f Change VercodeOperation to list
... so that we can calculate multiple version codes to support multi-arch autoupdate
2022-10-15 17:19:49 +08:00
linsui 0d39169840 checkupdates.TestCase: remove indentation
These assert functions doesn't need to be in the mock context.
2022-10-15 16:09:19 +08:00
linsui 5a28f20301 make versionCode/build.timeout an integer
Co-authored-by: Jochen Sprickerhof <git@jochen.sprickerhof.de>
2022-10-14 08:42:26 +00:00
linsui 6f73a87eb1 fix typo: Build -> Builds 2022-10-14 08:42:26 +00:00
Hans-Christoph Steiner 353ee84a6f
test that get_apk_id throws exceptions (issuebot#60) 2022-10-11 16:36:11 +02:00
Michael Pöhn e967fc61cf scanner: add regression tests for signautre load logic 2022-10-06 14:50:30 +02:00
Michael Pöhn b7233dfb2e ignore cache write test case on older python versions 2022-10-06 12:09:07 +02:00
Michael Pöhn 59b1899d79 scanner: switch form iso8601 to unix timestamp for python3.5 support 2022-10-06 12:09:07 +02:00
Michael Pöhn 9560ed955c avoid running into native-date object issue
Seem I ran into this issue: https://bugs.python.org/issue47228  This
change tries to fix it by using utcnow insteas of astimezone.
2022-10-06 12:09:07 +02:00
Michael Pöhn 0921863fa6 scanner: update suss defaults after removal of allowlisted feature 2022-10-06 12:09:07 +02:00
Michael Pöhn a8bcaa3d70 scanner: implement caching rules for suss 2022-10-06 12:09:06 +02:00
Michael Pöhn bfcc30b854 add --refresh to scanner 2022-10-06 12:09:06 +02:00
Michael Pöhn 1e6694112a rename to suss 2022-10-06 12:09:06 +02:00
Michael Pöhn c9b59b525d fix timestamp check; remove dead code 2022-10-06 12:09:05 +02:00
Michael Pöhn c10633eac5 convert fdroid scanner --exodus to SignatureDataController 2022-10-06 12:08:26 +02:00
Michael Pöhn d5ef1b2e95 add --clear-cache option to scanner 2022-10-06 12:08:26 +02:00
Michael Pöhn f56b1f3012 basic downloading for scan_binary signatures 2022-10-06 12:08:23 +02:00
Hans-Christoph Steiner 3de6063a01 scanner: open DEX/ZIP by file magic; throw errors on bad filenames 2022-09-30 17:56:15 +00:00
Hans-Christoph Steiner aa190d532f scanner.TestCase: manually convert to black code format
I manually changed some code structures to give a decent code format.
2022-09-30 17:56:15 +00:00
Jochen Sprickerhof f96f247095
Add test for git getref 2022-09-15 13:32:11 +02:00
linsui ac96e43e13 scanner: add some rules 2022-09-14 15:02:37 +00:00
pmmayero 0ad45a94a8 Addition of IPFS CIDv1 to Index
IPFS CIDv1 is only generated for APKs and "repo files"
2022-09-14 09:28:02 +00:00
Jochen Sprickerhof 483d6da51c
Cleanup tests/dump_internal_metadata_format.py
- Update usage documentation.
- Use argparse.
- Simplify config handling (similar to readmeta.py).
- Drop code for fdroidserver before 0.7.0.
- Reformat with black.
2022-09-08 21:40:26 +02:00
Hans-Christoph Steiner b854f3bab7 update: fully isolate testInsertStoreMetadata test
The test was failing due to something on my local filesystem, this
runs the test in a new dir every time.
2022-09-08 18:45:30 +02:00
Jochen Sprickerhof 5b79e7aea8
Fix trailing whitespace 2022-09-05 17:50:13 +02:00
Jochen Sprickerhof c89a9f0e8b
Add timeout argument to requests.(get,post) 2022-09-05 17:50:10 +02:00
Hans-Christoph Steiner c6cf0468ed nightly: update Raw URLs to fix breakage and avoid redirects 2022-08-26 07:18:19 +00:00
FestplattenSchnitzel 7c89e923f6 Move methods specific to import to it's module 2022-08-24 22:21:37 +02:00
FestplattenSchnitzel 7b7f863c65 [import] Rename to import_subcommand internally
This enables normal import of the module without the need for
workarounds.
2022-08-24 22:21:35 +02:00
linsui 832ed18fca scanner: also scan classpath 2022-08-24 21:34:55 +02:00
Hans-Christoph Steiner 9e58fc8cda
verify: normalize dicts via JSON for reliable comparisons
13016c5d63 in !602 used a set to prevent
duplicate entries, but that worked poorly because it required lots of
data wrapping.  Instead, just normalize to JSON, then equality is easy.
2022-08-24 19:16:33 +02:00
linsui 516a0c2ce8 Fix tag match with GitPython 2022-08-24 10:50:09 +00:00
linsui f99dbec012 scanner: fix maven url check for kts files 2022-08-08 15:52:47 +00:00
Jochen Sprickerhof eb79522a36 Fix flake8 E275 missing whitespace after keyword 2022-07-31 09:48:14 +02:00
Jochen Sprickerhof 7822db2881 Catch DefusedXmlException (as ValueError)
defusedxml can't handle the nbsp in the strings.xml (etree can).
2022-07-25 09:30:25 +02:00
Michael Pöhn ab579be6b5 add some basic tests for scanner.main 2022-07-17 15:52:52 +02:00
Michael Pöhn 42d9ac446c get tests working on ci 2022-07-17 15:50:07 +02:00
Michael Pöhn 1c2b084410 🔧 improve scanner.scan_apk tests
Refactor test function it a TestCase and split into separate test cases.
Fix and improve tests for scanning apks with embedded apks.
2022-07-17 15:50:07 +02:00
Michael Pöhn 07a366a4d6 add tests for scanner.load_exodus_trackers_signatures 2022-07-17 15:50:07 +02:00
Michael Pöhn a1677b5cb0 add test case for _exodus_compile_signatures 2022-07-17 15:50:07 +02:00
Hans-Christoph Steiner e2d9dedbb1 build: add test to ensure the flags are passed to the buildserver
!1141
2022-06-08 21:15:51 +02:00
Hans-Christoph Steiner 2bf6848391 test_find_apksigner_system_package_android_home if build_tools is new enough 2022-06-08 20:39:44 +02:00
Hans-Christoph Steiner fe22958476
run-tests: skip tests that require apksigner when running on Java8
The buildserver VM has not been upgraded yet to bullseye, so it is still on
Debian/stretch.  The buildserver VM does not need to run `fdroid update`,
`fdroid signindex`, etc. so this new apksigner requirement should not
affect app builds even though they are stuck on Debian/stretch.
2022-06-07 16:56:25 +02:00
Hans-Christoph Steiner 3182b77d18
use apksigner to sign index-v2 with modern, supported algorithms
The current signing method uses apksigner to sign the JAR so that it
will automatically select algorithms that are compatible with Android
SDK 23, which added the most recent algorithms:
https://developer.android.com/reference/java/security/Signature

This signing method uses then inherits the default signing algothim
settings, since Java and Android both maintain those.  That helps
avoid a repeat of being stuck on an old signing algorithm.  That means
specifically that this call to apksigner does not specify any of the
algorithms.

The old indexes must be signed by SHA1withRSA otherwise they will no
longer be compatible with old Androids.

apksigner 30.0.0+ is available in Debian/bullseye, Debian/buster-backports,
Ubuntu 21.10, and Ubuntu 20.04 from the fdroid PPA.  Here's a quick way to
test:

for f in `ls -1 /opt/android-sdk/build-tools/*/apksigner | sort ` /usr/bin/apksigner; do printf "$f : "; $f sign --v4-signing-enabled false; done

closes #1005
2022-06-07 16:56:23 +02:00
Hans-Christoph Steiner 7544761e86
index: add test for --nosign config and file generation 2022-06-07 13:22:06 +02:00
Hans-Christoph Steiner c6dcc82ca4 allow common.get_apk_id() to be used in the API
If a project uses fdroidserver as a library, then just calls
common.get_apk_id(), it will now work.  Before, that project would have had
to include something like `common.config = {}` to avoid a stacktrace.
2022-05-26 22:18:21 +02:00
Hans-Christoph Steiner 2639909f90 add test cases for parsing smartcardoptions config field 2022-05-26 16:49:43 +02:00
Hans-Christoph Steiner cbd4828561 include index-v2 in signindex tests 2022-05-24 11:35:46 +02:00
Hans-Christoph Steiner 2448f070e9 fix tests and docstring error 2022-05-23 15:34:30 +02:00
Jochen Sprickerhof b07d23ff5c Don't include disabled apks in the index
This needs a rerun of `fdroid update --clean`.

In case a build is disabled delete_disabled_builds takes care of
deleting it from the repo. But this only works if the apk follows the
normal name pattern. Otherwise it will stay in the folder and be picked
up by process_apks and added to the index.

Closes: #1002
2022-05-23 11:47:53 +00:00
Jochen Sprickerhof d70e5c2cd9 Index v2 2022-05-23 10:39:17 +00:00
Hans-Christoph Steiner 73c31e6e63
signindex: do not remove index-v1.json after signing
With ~index-v2, the model is changing to offer the plain JSON file for easy
consumption.  Then gpgsign will also provide a detached PGP signature for
systems that would rather verify based on PGP signatures than JAR signatures.

!1080
closes #969
2022-05-17 15:38:06 +02:00
Jochen Sprickerhof 08e3e445fb Merge androguard_test into update.TestCase 2022-05-10 14:40:03 +00:00
Jochen Sprickerhof e82888837e Fix androguard_test
- SdkToolsPopen expects a list.
- info.guardianproject.urzip.apk is not in the repo.
- scan_apk_aapt was removed in 08931f45 so define output.
2022-05-10 14:40:03 +00:00
Hans-Christoph Steiner 107593d3ba net: improve parsing filenames from URL
This is more relevant to issuebot than fdroidserver, but it can't hurt here
also.
2022-05-09 20:59:15 +02:00
linsui df1e0e30d5 fix tests 2022-05-07 02:17:26 +00:00
linsui 90bc8e1e8f scan gradle.kts 2022-05-07 00:43:55 +08:00
Jochen Sprickerhof 94ccd022c4 Add apk.embedded_1.apk test 2022-05-03 14:59:52 +00:00
Jochen Sprickerhof 9c8e128064 [scanner] replace apkanalyzer by dexdump 2022-05-03 14:59:52 +00:00
Hans-Christoph Steiner 9f477dee56
signindex: added simple test case 2022-04-25 22:38:34 +02:00
Sergey Bobrenok 46f4205fff Avoid zero-length prefixes in PATH
A zero-length prefix in PATH is a legacy feature that indicates the
current working directory.

https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html#tag_08_03

Found in ru.nsu.bobrofon.easysshfs, see:
https://gitlab.com/fdroid/fdroiddata/-/merge_requests/10953#note_921802636
2022-04-24 16:33:33 +03:00
Jochen Sprickerhof 5f3eb601df Add check for repo/archive_url 2022-04-21 11:16:14 +02:00
Jochen Sprickerhof 0c390c943d [checkupdates] update submodules according to tag
In case the version information is inside a submodule we need to
checkout the submodule at the version of the tag we test.

Found with org.courville.nova.

Closes: #622
2022-03-10 12:49:03 +01:00
Simon Ser 757e30ad85 Add test case for Triple-T metadata in a Flutter project 2022-03-07 12:40:37 +01:00
Hans-Christoph Steiner 707cd7addb
publish index-v1.json and include a gpg signature
This gives a more flexible and direct way for many clients to consume the
index file. #969
2022-02-22 22:58:09 +01:00
Hans-Christoph Steiner d1fd58681e
nightly: stop stripping APKs before signing, apksigner does it
* https://gitlab.com/fdroid/fdroidserver/-/merge_requests/1033#note_742563869
* https://github.com/wardvl/f-droid-nightly-action/issues/3
2022-02-22 22:29:09 +01:00
Jochen Sprickerhof b73663967b Only ignore opening a second MR 2021-12-30 19:23:05 +00:00
Jochen Sprickerhof 460dffe82b Don't open second MR for gradle (Closes: #914)
Traceback (most recent call last):
  File "/builds/fdroid/fdroidserver/./tests/gradle-release-checksums.py", line 130, in <module>
    mr = project.mergerequests.create({
  File "/usr/lib/python3/dist-packages/gitlab/exceptions.py", line 281, in wrapped_f
    raise error(e.error_message, e.response_code, e.response_body) from e
gitlab.exceptions.GitlabCreateError: 409: ['Another open merge request already exists for this source branch: !1064']
2021-12-30 10:00:46 +01:00
jugendhacker b236773455 Fix subproject regex to allow subdir without colon 2021-11-30 08:28:17 +00:00
Jochen Sprickerhof a5deaa80d8 Parse single digit sizes
Closes: #946
2021-11-27 21:44:47 +01:00
Gaurav Ujjwal aead3310bd NDK Install: Handle symlinks present in NDK zip 2021-11-03 08:13:21 +00:00
Hans-Christoph Steiner da10acfe2c
update: add test with APK for icon_id related exceptions 2021-10-01 16:44:14 +02:00
Hans-Christoph Steiner c71c2465d3
tests: silence the linters 2021-09-20 10:51:54 +02:00
Hans-Christoph Steiner d6fd165444 test whether NDK version parsing is working properly 2021-09-20 10:18:16 +02:00
Hans-Christoph Steiner 8ea154c125 fix IndexTest.test_gitlab_get_mirror_service_urls
This makes it actually call the method rather than mocking it.
2021-09-13 13:18:21 +02:00
Hans-Christoph Steiner 7987c746de index: do not include GitLab Pages mirror if it can't be deployed
GitLab Pages sites are limited to 1GB on gitlab.com, so the CI/CD job will
fail if the repo is bigger than that.  It should not be included as a
mirror in that case.

https://docs.gitlab.com/ee/user/gitlab_com/#gitlab-pages
2021-09-11 15:30:55 +00:00
Hans-Christoph Steiner a56d377c9e
update: modernize options handling in tests
This moves everything to the Options class, and resets the options between
test runs.
2021-08-05 16:44:40 +02:00
Hans-Christoph Steiner 3b95d3de64
update: AllowedAPKSigningKeys metadata to enforce APK signers
This field lets you specify which signing certificates should be
trusted for APKs in a binary repo.
2021-08-05 16:43:48 +02:00
Felix C. Stegerman d518971204
add test_insert_triple_t_anysoftkeyboard() 2021-08-04 17:33:38 +02:00
Felix C. Stegerman 4c4a283ae2
update test_insert_triple_t_multiple_metadata() test data 2021-08-03 20:53:56 +02:00
Jochen Sprickerhof 121e06e4b7 Fix CheckupdatesTest 2021-07-29 11:52:22 +02:00
Jochen Sprickerhof 8f836b3b01 [checkupdates] Move log messages into method 2021-07-28 00:09:40 +02:00
Jochen Sprickerhof 6f7a1ecf01 [checkupdates] Don't catch exceptions
Basically moves all code one level up.
2021-07-28 00:09:40 +02:00
Jochen Sprickerhof a2db8f4a62 [checkupdates] Exit 1 in case of errors 2021-07-28 00:09:40 +02:00
Jochen Sprickerhof 331b4830dd Support multiple flavours in parse_androidmanifests
Previously only the last flavour was checked.
2021-07-24 21:20:19 +02:00
Pierre Rudloff d04c3f65c2 New com.jens.automation2 test case 2021-07-06 08:30:08 +00:00
Felix C. Stegerman 73f28a611c
add test_insert_triple_t_multiple_metadata() 2021-07-03 21:20:18 +02:00
Hans-Christoph Steiner 40c77892a2
do not crash when config.yml is 0 bytes or empty of data 2021-07-01 14:45:35 +02:00
Hans-Christoph Steiner 5267699d81
index: make download_repo_index() handle common URL mistakes 2021-07-01 14:43:58 +02:00
Jochen Sprickerhof 70a11b3d79 Ignore ~/.gitconfig in tests and drop version test
git version 2.3 was released in 2015.
2021-06-25 13:22:58 +02:00
Jochen Sprickerhof 674786db96 [checkupdates] Ignore broken submodule
In case the app repository has a broken submodule, checkupdates failed
and did not search for any version updates. Ignoring the error let's us
at least find new version in the main repo (which is probably the right
place anyhow) and thus an improvement.
2021-06-25 12:15:41 +02:00
Hans-Christoph Steiner 9500b9d2e9
gradle-release-checksums.py: check for GitLab token before committing 2021-06-24 21:34:34 +02:00
linsui 763a2ee80d fix invalid extension of output apk 2021-06-24 13:42:10 +00:00
Hans-Christoph Steiner 78d37bb13b switch to allow/block list terminology throughout code base
allowlist and blocklist are much clearer terms with no cultural baggage.
This changes all "whitelist" references to "allowlist", and all "blacklist"
references to "blocklist".
2021-06-18 18:26:50 +00:00
Hans-Christoph Steiner 36849b2fba
skip tests on Windows that need bash 2021-06-18 11:12:22 +02:00
Hans-Christoph Steiner 1f5534d060
require config.yml use UTF-8 as encoding
config.yml requires ASCII or UTF-8 encoding because this code does not
auto-detect the file's encoding.  That is left up to the YAML library.
YAML allows ASCII, UTF-8, UTF-16, and UTF-32 encodings.  Since it is a
good idea to manage config.yml (WITHOUT PASSWORDS!) in git, it makes
sense to use a globally standard encoding.
2021-06-18 11:12:20 +02:00
Hans-Christoph Steiner 48c4354629
always open Android source files as UTF-8
Android Studio recommends "you use UTF-8 encoding whenever possible",
so this code assumes the files use UTF-8.  UTF-8 is also the default
encoding on GNU/Linux and macOS.
https://sites.google.com/a/android.com/tools/knownissues/encoding

Windows will probably default to UTF16, since that's the native
encoding for files.  So forcing things to use UTF-8 should help
compatibility.
2021-06-18 11:12:18 +02:00
Gerhard Olsson 0c31c4a5ab
win fixes 2021-06-18 10:30:59 +02:00
Hans-Christoph Steiner a6d35a7ee1
metadata: always open metadata files as UTF-8
Windows seems to require this, otherwise this happens:

Traceback (most recent call last):

  File "tests/update.TestCase", line 737, in test_translate_per_build_anti_features

    apps = fdroidserver.metadata.read_metadata(xref=True)

  File "C:\Users\travis\build\fdroidtravis\fdroidserver\fdroidserver\metadata.py", line 813, in read_metadata

    app = parse_metadata(metadatapath, appid in check_vcs, refresh)

  File "C:\Users\travis\build\fdroidtravis\fdroidserver\fdroidserver\metadata.py", line 1023, in parse_metadata

    parse_yaml_metadata(mf, app)

  File "C:\Users\travis\build\fdroidtravis\fdroidserver\fdroidserver\metadata.py", line 1073, in parse_yaml_metadata

    yamldata = yaml.safe_load(mf)

  File "C:\python37\lib\site-packages\yaml\__init__.py", line 162, in safe_load

    return load(stream, SafeLoader)

  File "C:\python37\lib\site-packages\yaml\__init__.py", line 112, in load

    loader = Loader(stream)

  File "C:\python37\lib\site-packages\yaml\loader.py", line 34, in __init__

    Reader.__init__(self, stream)

  File "C:\python37\lib\site-packages\yaml\reader.py", line 85, in __init__

    self.determine_encoding()

  File "C:\python37\lib\site-packages\yaml\reader.py", line 124, in determine_encoding

    self.update_raw()

  File "C:\python37\lib\site-packages\yaml\reader.py", line 178, in update_raw

    data = self.stream.read(size)

  File "C:\python37\lib\encodings\cp1252.py", line 23, in decode

    return codecs.charmap_decode(input,self.errors,decoding_table)[0]

UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 37: character maps to <undefined>
2021-06-18 10:30:55 +02:00
Hans-Christoph Steiner 4514983e58
fix typo 2021-06-18 10:30:50 +02:00
Hans-Christoph Steiner 1abbc9732e
metadata: add test_build_ndk_path 2021-06-18 10:30:44 +02:00
Jochen Sprickerhof 39c55d799b [checkupdates] Only update if version code is grater
Don't change the current versions in case there is an error in the check_* methods or upstream screwed up.
2021-06-17 11:52:37 +00:00
Jochen Sprickerhof 3809b4d424 Ignore git submodule failure in gotorevisionx
gotorevisionx tries to clean up the git repo before checking out a new
revision. In b848b99ba this was changed to reset and clean any submodule
as well. In case upstream has a broken submodule configuration this
could fail and we can't checkout the new revision. As we are doing a
reset and clean after checking out the new revision anyhow, this change
ignores submodule errors before the checkout and only makes sure that
the main repo is reset and clean.

This broke checkupdates for apps where old versions had broken
submodules. It checkout out the old version and got stuck, not able to
checkout any other version.
2021-06-15 20:39:18 +02:00
Jochen Sprickerhof cebdcdd67c [checkupdates] UpdateCheckData tag for verocode if no regex
Use the tag as the version code if no regex was specified. This allows:

UpdateCheckData: '|||'

meaning the tag should be used for version code and name.
2021-06-15 19:27:29 +02:00
Jochen Sprickerhof 4e97b58d8c latesttags revert to git log and fix comma handling
2de34312 tried to fix the comma handling by relying on git tag --sort.
This did not work out so this reverts to the method used before.
2021-06-15 08:39:59 +02:00
Jochen Sprickerhof 2de3431296 Use git tag in latesttags 2021-06-14 18:56:59 +00:00
Jochen Sprickerhof 1e6de7eb34 Support '{' in extra line in parse_androidmanifests
If the flavour group starts in a separate line don't count it as a
second group.

Closes: #899
2021-06-13 07:23:42 +02:00
Jochen Sprickerhof 0fefecde1e Fix matching substring flavour detection
com.github.jameshnsears.quoteunquote defines flavours 'fdroid' and
'fdroidS'. The old code used flavour in line, which matches both and the
wrong one was selected.

Closes: #912
2021-06-13 00:09:02 +02:00
Jochen Sprickerhof bdec7d8652 [checkupdates] UpdateCheckData use tag by default
Use the tag as version, if no version file was specified:

UpdateCheckData: app/build.gradle|versionCode\s(\d+)||

Extract version from tag, if a regex was specified:

UpdateCheckData: app/build.gradle|versionCode\s(\d+)||Android-([\d.]+)

Use the tag for both if no file was specified:

UpdateCheckData: |\+(\d+)||Android-([\d.]+)
2021-06-09 14:02:02 +02:00
Jochen Sprickerhof a9a336a12b Don't allow '_' in first place of version code
Fixes a regression of 2cb0ff45. Example:

versionCode project.versionCode_plus

931b0a3087/app/build.gradle (L38)
2021-06-09 11:04:25 +00:00
Jochen Sprickerhof 69a0a7da39 [checkupdates] UpdateCheckData warn if file was not found
Instead of throwing an exception.
2021-06-09 11:52:11 +02:00
linsui 6bafb036ee lint.py: use pathlib and support Windows 2021-06-09 15:46:52 +08:00
Hans-Christoph Steiner 490f578d1c remove redundant call to Path() 2021-06-08 16:50:48 +02:00
linsui 8f21f1e510 metadata.py/rewritemeta.py: use pathlib and support Windows 2021-06-08 21:31:55 +08:00
linsui d6eece6395 import.py: use pathlib and support Windows 2021-06-08 18:33:22 +08:00
linsui 84b74d481e fix tests for !935 2021-06-07 15:33:57 +00:00
linsui 92ec6265a3 checkupdates.py: use pathlib and support Windows 2021-06-07 21:02:03 +08:00
Jochen Sprickerhof 133b626b22 Support UpdateCheckData in check_tags 2021-06-07 13:47:18 +02:00
Hans-Christoph Steiner d05ff9db1d
easy changes to black code format in test cases
This does not change areas of code that should be manually reformatted.
2021-06-07 11:53:58 +02:00
Hans-Christoph Steiner cb09a16133
tests/ndk-release-checksums.py: fix parsing of NDK r10e
This wasn't finding r10e properly, so it would submit merge requests
to remove r10e from _fdroidserver_.

* !940
* !930
2021-06-07 09:58:12 +02:00
Jochen Sprickerhof 185da60d93 Support AutoUpdateMode: Version without pattern
Since 24dd6740 UpdateCheckMode: Tags uses the found tag instead of
regenerating it from the AutoUpdateMode pattern making the pattern
superfluous. This adds support for dropping the pattern and a test case.
2021-06-01 13:59:49 +02:00
Hans-Christoph Steiner 52ac0f0176 ndk-release-checksums.py: fix crash, there is no version here 2021-05-28 10:32:09 +02:00
Hans-Christoph Steiner 09fa49a7a3
make get_android_tools_versions() search ndk_paths from config 2021-05-28 09:13:38 +02:00
Hans-Christoph Steiner 7a1d236c8d
only support zipballs in NDK provisioning
Since I discovered there is an r10e zipball, this can now get all NDKs
in zipball form.
fdroid/android-sdk-transparency-log@447fea86e7

closes #902
2021-05-28 09:13:36 +02:00
Hans-Christoph Steiner 9f77044d0d
auto-detect NDKs installed in standard paths
'ndk_paths' will be automatically filled out from well known sources
like $ANDROID_HOME/ndk-bundle and $ANDROID_HOME/ndk/*.  If a required
version is missing in the buildserver VM, it will be automatically
downloaded and installed into the standard $ANDROID_HOME/ndk/
directory.  Manually setting it here will override the auto-detected
values.  The keys can either be the "release" (e.g. r21e) or the
"revision" (e.g. 21.4.7075529).

https://developer.android.com/studio/projects/configure-agp-ndk#agp_version_41
* sdkmanager installs "ndk;12.3.4567890" into $ANDROID_SDK_ROOT/ndk/
* sdkmanager installs "ndk-bundle" into $ANDROID_SDK_ROOT/ndk-bundle/
2021-05-28 09:13:34 +02:00
Hans-Christoph Steiner 4686c06f62 metadata: allow `ndk:` to be str or list of release or revision
There are two version numbers used for NDKs: the "release" and the
"revision".  The "release" is used in the download URL and zipball and the
"revision" is used in the source.properties and the gradle ndkVersion field.

Also, there are some builds which need multiple NDKs installed, so this
makes it possible to have a list of release/revision entries in build.ndk.
This does not yet add full support since _fdroidserver/build.py_ will also
need changes.
2021-05-28 09:13:27 +02:00
Hans-Christoph Steiner 9d44fa7919
gitlab-ci: auto-generate merge request when NDK release found
Following the pattern of the gradle bot, this will check the transparency
log for any new NDK release.  If there are any, it will make a merge
request from @fdroid-bot.
2021-05-25 17:06:30 +02:00
Hans-Christoph Steiner 69fcd6a024
build: auto-download missing NDKS if they're known and can be verified
refs #517 #717
2021-05-25 17:06:24 +02:00
Hans-Christoph Steiner 0b0bc803b7 build: --test now keeps unsigned APKs in tmp/ that fail to reproduce
Before, whenever an unsigned APK failed to reproduce, it was just deleted.
That makes debugging hard.  This makes it keep the unsigned APK, which is
written in tmp/ when using --test.

@jspricke this is related to !864
2021-04-16 09:46:23 +02:00
Hans-Christoph Steiner 3d69e767d8
common: test abs and rel paths in get_all_gradle_and_manifests() 2021-04-16 09:40:19 +02:00
Jochen Sprickerhof f01a3caf77 Bump METADATA_VERSION for apksigner transition
We switched to apksigner in 50f0534d but old apks where still verified
with jarsigner (or an old apksigner version). Bumping the
METADATA_VERSION to force a rebuild of apkcache.
Hopefully this resolves de.chagemann.regexcrossword getting the
KnownVuln, DisabledAlgorithm tags.
2021-04-14 22:08:26 +00:00
Hans-Christoph Steiner 2946c90dd4 publish: rename vars to match naming in JAR Signature docs
https://docs.oracle.com/javase/tutorial/deployment/jar/intro.html

closes #892
2021-04-14 23:01:47 +02:00
Hans-Christoph Steiner 44d481768f publish: add test for reproduble builds with signatures 2021-04-14 23:01:47 +02:00
Felix C. Stegerman 202fd8b25a
vendor & use apksigcopier v0.4.0-12-g93d8e14 2021-04-14 21:06:20 +02:00
Felix C. Stegerman 67a0f3ae5b
use subclass hack for better ZIP cloning
See https://bugs.python.org/issue43547 for more info on the details.

thanks to @obfusk for the technique
2021-04-14 15:17:56 +02:00
Hans-Christoph Steiner cff575f402 common: add test case for read_pkg_args() 2021-04-12 09:11:58 +02:00
Hans-Christoph Steiner b5485ded64
skip test_find_apksigner_system_package_android_home if no apksigner 2021-03-23 23:29:50 +01:00
Hans-Christoph Steiner b69b95103e add complete tests for finding apksigner; fix minor detection bug
find_apksigner() was preferring the oldest valid version rather than the
newest.
2021-03-23 21:01:53 +01:00
Hans-Christoph Steiner 67e65ce2ea purge complete-ci-tests, those are better covered in .gitlab-ci.yml 2021-03-22 23:36:04 +01:00
Hans-Christoph Steiner 93ca13fa16 run-tests: remove broken test that never ran due to a syntax error 2021-03-22 22:31:29 +01:00
Hans-Christoph Steiner 7ec2a8ebb3 add apksigner, jarsigner, and keytool to relevant status JSON 2021-03-22 22:31:29 +01:00
Hans-Christoph Steiner 903a7396b1 switch to preferring apksigner, requiring for `fdroid publish` 2021-03-22 22:30:49 +01:00
Jochen Sprickerhof ea9f917ddc Support versionCode and versionName in brackets
Example: versionCode(29)
https://github.com/florisboard/florisboard/blob/v0.3.9/app/build.gradle.kts#L24
2021-03-19 13:58:02 +01:00
Hans-Christoph Steiner dbf80ad771
metadata: remove git repo handling from metadata parsing
This is a vestige of implementing builds from a .fdroid.yml file directly
in the app's source repo.  It was never fully complete and seems to not be
used in any apps in fdroiddata.  This makes `fdroid build --all` runs much
faster since it does not need to do any git handling for apps that do not
have any new builds to run.

4e8e29794
948689281
a4e431080
e37be9b06
e775d330c
2021-03-17 13:44:24 +01:00
Hans-Christoph Steiner cc3b88ab30 gradlew-fdroid: use gradle zipballs from local cachedir, if present 2021-03-11 08:40:34 +01:00
Hans-Christoph Steiner 65836d9e9b run-tests: fix local_copy_dir test, it broke when /tmp/fdroid existed 2021-03-09 17:55:10 +01:00
Hans-Christoph Steiner 649347dcd4 common: properly parse version from NDK's source.properties 2021-03-09 15:55:45 +01:00
Hans-Christoph Steiner 6a875a1821 update: stop extracting and storing XML icons, they're useless
APKs that only have XML icons in them should include an icon.png in the
metadata or fastlane/triple-t

closes #344
2021-03-09 15:40:30 +01:00
Hans-Christoph Steiner 544a45c16a index: raise error rather than crash on bad repo file
If a non-APK is added with the appid/packageName that matches some APKs, it
should through an error.

Traceback (most recent call last):
  File "/home/hans/code/fdroid/server/fdroid", line 22, in <module>
    fdroidserver.__main__.main()
  File "/home/hans/code/fdroid/server/fdroidserver/__main__.py", line 211, in main
    mod.main()
  File "/home/hans/code/fdroid/server/fdroidserver/update.py", line 2343, in main
    index.make(apps, sortedids, apks, repodirs[0], False)
  File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 142, in make
    fdroid_signing_key_fingerprints)
  File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 166, in make_v1
    v1_sort_packages(packages, fdroid_signing_key_fingerprints)
  File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 292, in v1_sort_packages
    packages.sort(key=v1_sort_keys)
  File "/home/hans/code/fdroid/server/fdroidserver/index.py", line 288, in v1_sort_keys
    .format(apkfilename=package['apkName']))
fdroidserver.exception.FDroidException: at.roteskreuz.stopcorona_8.jobf does not have a valid signature!
2021-03-09 15:40:30 +01:00
Jochen Sprickerhof 7f2cbd1876 Fix serverwebroot test 2021-02-12 10:20:30 +00:00
Jochen Sprickerhof 33bff6b298 Publish unsigned builds
This should make debugging reproducible builds easier.
2021-02-12 10:20:30 +00:00
Hans-Christoph Steiner 525dcb8f98 update: handle large, corrupt, or inaccessible fastlane/triple-t files
```
Traceback (most recent call last):
  File "../fdroid", line 22, in <module>
    fdroidserver.__main__.main()
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/__main__.py", line 227, in main
    raise e
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/__main__.py", line 208, in main
    mod.main()
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2340, in main
    repoapps = prepare_apps(apps, apks, repodirs[0])
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 2176, in prepare_apps
    copy_triple_t_store_metadata(apps_with_packages)
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 1076, in copy_triple_t_store_metadata
    _set_author_entry(app, 'authorWebSite', os.path.join(root, f))
  File "/var/lib/jenkins/userContent/reproducible/reproducible_fdroid_build_apps/fdroidserver/update.py", line 784, in _set_author_entry
    with open(f, errors='replace') as fp:
FileNotFoundError: [Errno 2] No such file or directory: 'build/player.efis.cfd/pfd/src/main/play/contact-website.txt'
```
2021-02-11 17:42:13 +01:00
Benedikt Brückmann 9c609aa361 Fix copy-paste error 2021-02-08 12:25:02 +01:00
Benedikt Brückmann 23931e6818 Install [test] dependencies when running complete-ci-tests 2021-02-08 12:25:02 +01:00
Benedikt Brückmann b1bde9e407 * Skip tidy check if html5print is not available
* install test dependencies on arch
2021-02-08 12:25:02 +01:00
Benedikt Brückmann 022d73b3b6 Add HTML/CSS tidy test
and tidy up the content
2021-02-08 12:25:02 +01:00
Hans-Christoph Steiner 5cd9a8e396
tests: make init apksigner test use existing test keystore
This test does not need to generate a keystore, so using the existing one
reduces the run time from 4 seconds to 0.040 seconds.  Also, if makes this
mystery error on macOS go away:

```
Traceback (most recent call last):
  File "/Users/travis/build/fdroidtravis/fdroidserver/tests/init.TestCase", line 66, in test_main_in_empty_dir
    fdroidserver.init.main()
  File "/Users/travis/build/fdroidtravis/fdroidserver/fdroidserver/init.py", line 231, in main
    common.genkeystore(c)
  File "/Users/travis/build/fdroidtravis/fdroidserver/fdroidserver/common.py", line 3434, in genkeystore
    cmd = [config['keytool'], '-genkey',
KeyError: 'keytool'

```
2021-01-29 09:06:21 +01:00
Hans-Christoph Steiner 32a0c61010 init: enable apksigner by default if it is found 2021-01-28 22:25:33 +01:00
Hans-Christoph Steiner 0af157244a
gradle-release-checksums: runs after commits push to fdroid/fdroidserver 2021-01-25 11:05:20 +01:00
Hans-Christoph Steiner 669ce10d9a
enable complete lint/CI checking on gradle-release-checksums.py 2021-01-25 11:05:16 +01:00
Hans-Christoph Steiner d133b38f22 fdroid-bot creates merge request when new gradle versions are found 2021-01-25 00:26:14 +01:00
Hans-Christoph Steiner eec0d3a336 gitlab-ci: gradle job generates required changes for new versions 2021-01-24 16:58:38 +01:00
Hans-Christoph Steiner f527b17715 sync repo examples with defaults and comment out example repo settings
!359
2021-01-24 09:26:11 +01:00
Hans-Christoph Steiner aebf70a763 tests: run pip tests using fdroid that pip installed 2021-01-23 20:13:17 +01:00
Hans-Christoph Steiner 5ae2703477 init: no error if examples/ dir cannot be found; overhaul repo_icon
There are so many possible installation paths for Python modules, it has
been very hard to even find and test them all.  This adds a fallback option
if the examples dir cannot be found.  A repo can work without an icon or
the example config.py.

This removes the fake assumption that the icon can be a full path in the
config.py.  While the path was being properly passed through to the index
file, the file was never copied properly into place nor rsynced to the web
server.
2021-01-18 14:07:20 +01:00
Hans-Christoph Steiner 94cb19f571 set METADATA_VERSION to 20000 to leave room for 1.1.x changes 2021-01-15 14:33:21 +01:00
Hans-Christoph Steiner fff59e5197 only use AutoName: in checkupdates
AutoName: is only needed for the commit messages generated by checkupdates,
and it makes the logic for localized names confusing.

closes #654
refs #304
2021-01-14 20:36:09 +01:00
Hans-Christoph Steiner 0f6b638986 common.get_app_display_name() for finding app names 2021-01-14 20:34:15 +01:00
Hans-Christoph Steiner 81b55e0d8f
tests: use wc -m to support BSD and macOS 2021-01-07 17:09:44 +01:00
Hans-Christoph Steiner bf25b4ca03 eliminate app.builds everywhere, it should be app['Builds']
The .txt format was the last place where the lowercase "builds" was used,
this converts references everywhere to be "Builds".  This makes it possible
to load metadata YAML files with any YAML parser, then have it possible to
use fdroidserver methods on that data, like metadata.write_metadata().

The test files in tests/metadata/dump/*.yaml were manually edited by cutting
the builds: block and putting it the sort order for Builds: so the contents
should be unchanged.

```
sed -i \
 -e 's/app\.builds/app.get('Builds', \[\])/g' \
 -e "s/app\.get(Builds, \[\]) =/app\['Builds'] =/g" \
 -e "s/app\.get(Builds, \[\]) =/app\['Builds'] =/g" \
 -e "s/app\.get(Builds, \[\])/app.get('Builds', \[\])/g" \
 -e "s/app\.get('Builds', \[\])\.append/app\['Builds'\].append/g" \
 -e "s/app\['builds'\]/app.get('Builds', [])/g" \
 */*.*
```
2020-12-15 08:55:05 +01:00
Hans-Christoph Steiner fb628c2cb2 include modified and untracked files in status JSON
Ideally, an fdroid repo should be running from a clean git repo, so that
all changes are tracked in git.  This change is useful in seeing which
changes and/or files are not in git.  If there are modified files, the
dirty flag will be set, so this info can help debugging that.
2020-12-15 08:30:48 +01:00
Hans-Christoph Steiner 384922118f index: sanitize fingerprint arg, extract_pubkey() returns with spaces
The key fingerprint should be only hex digits, everything else can be
discarded.  That makes it easy to use this function various fingerprint
formats, including the common, human-readable forms spaces between pairs or
quartets.
2020-12-15 08:30:48 +01:00
Hans-Christoph Steiner d9171f11e0 update: improve logging when exiting due to bad APK file
closes #851
2020-12-15 08:30:48 +01:00
Hans-Christoph Steiner ba854cbc0f index: fix GitLab Raw URLs with gitlab.com and recent versions
GitLab seems to be moving to always having "-" as the first path segment in
all the project URLs.  So the URL without a "-" is now a redirect.
2020-12-15 08:30:48 +01:00
Hans-Christoph Steiner 531c36f310
fix typo in ac3eb0e641 2020-12-14 23:08:32 +01:00
Hans-Christoph Steiner ac3eb0e641 tests: fix mirror test setup on macOS
I guess macOS has an ancient version of bash which does not include
support for $RANDOM
2020-12-14 20:33:28 +01:00
Hans-Christoph Steiner 9cfd12f9ed tests: include test http.server.log in the test's tmp dir 2020-12-10 10:45:28 +01:00