Commit Graph

5971 Commits

Author SHA1 Message Date
Marcus Hoffmann 03881154c6 metadata: make linkresolver an actual object
Previously this was magically capturing the apps dict when passing it around as a
function. This also moved the code to the metadata module.

Add a test doing read_metadata where the linkresolver is used. This
happens when the apps we read have a [[app.id]] link to another app.
2020-06-24 23:53:35 +02:00
Marcus Hoffmann a4177e5ec3 add test for correct whatsnew handling without CVC 2020-06-24 23:53:35 +02:00
Hans-Christoph Steiner 238f048257 update: fix crash when liberapay: or open_collective: not in FUNDING.yml
closes #799
2020-06-24 21:39:43 +02:00
Hans-Christoph Steiner 82eceebd13 Merge branch 'parse-donation-links-from-funding.yml' into 'master'
update: insert donation links based on FUNDING.yml

See merge request fdroid/fdroidserver!754
2020-06-18 04:44:08 +00:00
Hans-Christoph Steiner 0b4112f743 Merge branch 'scanner-safe_path_test' into 'master'
scanner: add test with abs/rel paths as run from `fdroid build`

See merge request fdroid/fdroidserver!768
2020-06-17 08:56:42 +00:00
Hans-Christoph Steiner 2c4e9beacb scanner: add test with abs/rel paths as run from `fdroid build`
https://gitlab.com/fdroid/fdroidserver/-/issues/791#note_361018050
!767
2020-06-17 10:33:55 +02:00
Hans-Christoph Steiner 62c8fd5999
add Liberapay: field with username as data
Liberapay was originally included using a numeric ID, since they had
not yet finalized the public URLs.  Now it is a username.  So this
logic prefers the username in Liberapay: field, and keeps the old
LiberapayID: to ease migration.  LiberapayID: will not override
Liberapay:.  Clients are expected to prefer Liberapay: over LiberapayID:
2020-06-16 15:35:28 +02:00
Hans-Christoph Steiner 0183592526
update: insert donation links based on FUNDING.yml
GitHub has specified FUNDING.yml, a file to include in a git repo for
pointing people to donation links.  Since F-Droid also points people
to donation links, this parses them to fill out Donate:
and OpenCollective:.  Specifying those in the metadata file takes
precedence over the FUNDING.yml.  This follows the same pattern as how
`fdroid update` includes Fastlane/Triple-T metadata.  This lets the
git repo maintain those specific donations links themselves.

https://help.github.com/en/articles/displaying-a-sponsor-button-in-your-repository#about-funding-files

The test file was generated using:

```python
import os, re, yaml

found = dict()
for root, dirs, files in os.walk('.'):
    for f in files:
        if f == 'FUNDING.yml':
            with open(os.path.join(root, f)) as fp:
                data = yaml.safe_load(fp)
            for k, v in data.items():
                if k not in found:
                    found[k] = set()
                if not v:
                    continue
                if isinstance(v, list):
                    for i in v:
                        found[k].add(i)
                else:
                    found[k].add(v)

            with open('gather-funding-names.yaml', 'w') as fp:
                output = dict()
                for k, v in found.items():
                    output[k] = sorted(v)
                yaml.dump(output, fp, default_flow_style=False)
```
2020-06-16 15:35:26 +02:00
Marcus 8d517d4583 Merge branch 'simple_apk_scanner' into 'master'
Simple apk scanner

See merge request fdroid/fdroidserver!766
2020-06-15 18:03:19 +00:00
Marcus f84818c15e scanner: add a simple scan for blacklisted classes after build step
add com.android.billing to blacklist, see
https://gitlab.com/fdroid/fdroiddata/-/issues/2070#note_360611289
2020-06-15 18:03:19 +00:00
Marcus 6986e73506 Merge branch 'allow_appcenter' into 'master'
scanner: allow microsoft appcenter

See merge request fdroid/fdroidserver!764
2020-06-15 10:46:51 +00:00
Marcus Hoffmann 670328c9e5 scanner: allow microsoft appcenter, except appcenter-push
This sdk is open source. It was added because appcenter has a dependency
to play-services. It's possible though to build an app using appcenter
that doesn't pull in play services, so we can't blanket ban the sdk.

The appcenter-push modules has obvious refenrences to firebase, so it's
safe to error on that.

Ref: https://phabricator.wikimedia.org/T254980
2020-06-15 12:24:46 +02:00
Marcus 12f4ae3039 Merge branch 'fix_791' into 'master'
scanner: fix wrong path being passed to function

Closes #791

See merge request fdroid/fdroidserver!767
2020-06-13 11:03:30 +00:00
Marcus Hoffmann 3a9f1e845c scanner: fix wrong path being passed to function
Also make rename the parameter in safe_path to make it clear that this
is just a relative path.

Closes fdroid/fdroidserver#791.
2020-06-13 03:01:54 +02:00
Marcus be814bf002 Merge branch 'importsubmodules' into 'master'
Import - Detect .gitmodules and add submodules build line

See merge request fdroid/fdroidserver!765
2020-06-12 19:26:23 +00:00
Licaon_Kter 72fde0f10b Detect .gitmodules and add submodules build line 2020-06-11 16:36:54 +03:00
Marcus Hoffmann 294799e2f5 scanner: fix options handling
closes fdroid/fdroidserver#789
2020-06-11 11:44:00 +02:00
Marcus 0e025e2ae9 Merge branch 'scanner-overhaul' into 'master'
scanner file type overhaul

Closes #394

See merge request fdroid/fdroidserver!758
2020-06-11 09:23:39 +00:00
Marcus 95623361ae Merge branch 'master' into 'master'
Update NDK to r21d and add buildtools 30.0.0

See merge request fdroid/fdroidserver!763
2020-06-10 20:21:26 +00:00
TacoTheDank 07992177bf Add build tools 30.0.0 2020-06-10 15:59:04 -04:00
TacoTheDank 0216513da6 Upgrade NDK r21c to r21d 2020-06-10 15:55:53 -04:00
Hans-Christoph Steiner 0837289935 scanner: fix regex for matching URLs in gradle maven{} blocks
closes #465

This script generated gradle-maven-blocks.yaml:
```python
import os
import re
import yaml

pat = re.compile(r'\smaven\s*{[^}]+}')

finds = set()
for root, dirs, files in os.walk('.'):
    for f in files:
        if '.gradle' in f:
            with open(os.path.join(root, f), errors='surrogateescape') as fp:
                contents = fp.read()
            for m in pat.findall(contents):
                finds.add(m)

with open('finds.yaml', 'w') as fp:
    yaml.dump(sorted(finds), fp, default_flow_style=False)

```
2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner 6590f3869e scanner: error/warn on dex/gz/zip, closes #394 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner d7b3bca1e7 build: pass --verbose flag through to scanner 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner ee54dbce87 scanner: safely check options, for things using this API
https://gitlab.com/fdroid/fdroidserver/-/issues/771#note_353495799
2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner 75acf63efa scanner: remove all gradle wrapper files 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner 2f5d780c49 scanner: always setup JSON data structure so functions work as API 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner 84f225f2f2 scanner: make problem descriptions translationable 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner d898ad0412 scanner: make AARs and JARs trigger an error
refs #491
2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner 3bc246ccad scanner: ignore well known image types that are set executable 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner 83edb5b80a Merge branch 'plugin-system' into 'master'
Plugin system

See merge request fdroid/fdroidserver!709
2020-06-10 17:36:02 +00:00
Michael Pöhn a97b3ca4dd implement plugin system review suggestsions 2020-06-10 18:43:23 +02:00
Michael Pöhn 77167e098e plugin system: regex instead of import bases plugin parsing 2020-06-10 18:43:11 +02:00
Michael Pöhn b257a3411a stick to default python sys.path 2020-06-10 12:11:47 +02:00
Michael Pöhn 3a3803ea2d raise excepten when starting broken plugin 2020-06-10 12:11:47 +02:00
Michael Pöhn bf815251ec rough plugin system implementation 2020-06-10 12:11:47 +02:00
Michael Pöhn 32f09603e1 Merge branch 'move-tests-to-yaml' into 'master'
move tests to YAML

See merge request fdroid/fdroidserver!761
2020-06-10 10:04:36 +00:00
Hans-Christoph Steiner bde65aa54d gitlab-ci: switch metadata_v0 test to commit that supports only .yml
37f37ebd88
2020-06-10 10:44:27 +02:00
Hans-Christoph Steiner 0d1fddb82e
convert inline tests to .yml 2020-06-10 10:43:27 +02:00
Hans-Christoph Steiner f0fb3b64f8
remove obselete tests 2020-06-10 10:43:25 +02:00
Hans-Christoph Steiner 37f37ebd88
use default accepted_formats since all the files are .yml anyway 2020-06-10 10:43:21 +02:00
Hans-Christoph Steiner 1b90aec697
purge .txt and .json from bash-completion
YAML is the way!
2020-06-10 09:56:33 +02:00
Hans-Christoph Steiner b63e9e68c5
move all test metadata files to .yml format
for f in metadata/*.txt; do ../fdroid rewritemeta --to yml $(basename $f | sed 's,\.txt,,'); done
2020-06-10 09:56:31 +02:00
Marcus 3bf2d60bfa Merge branch 'fix_783' into 'master'
strip_and_copy_image: abort on broken symlinks

Closes #783

See merge request fdroid/fdroidserver!760
2020-06-04 11:20:43 +00:00
Marcus Hoffmann 581e433832 strip_and_copy_image: abort on broken symlinks
Also guard against other processes removing the files we are about to
copy.

closes fdroid/fdroidserver#783
2020-06-04 12:58:11 +02:00
Marcus Hoffmann 4c69411387 add gradle 6.5 2020-06-04 00:30:30 +02:00
Marcus 0d609b4b1e Merge branch 'validate-string-int-types' into 'master'
metadata: validate STRING and INT build field types

See merge request fdroid/fdroidserver!752
2020-06-03 22:03:37 +00:00
Hans-Christoph Steiner 13d9a122bf metadata: validate STRING and INT build field types
This converts float/int to string for things like commit: or versionName:.
For versionCode, which must be an integer, it throws an exception if the
data is any other type.
2020-06-02 22:06:01 +02:00
Hans-Christoph Steiner b2daf96284 lint: fix pyflakes 2020-06-02 22:05:18 +02:00
Hans-Christoph Steiner b1eb086259 Merge branch 'patch-2' into 'master'
Add NDK r21c

See merge request fdroid/fdroidserver!755
2020-06-02 19:59:38 +00:00