Commit Graph

6094 Commits

Author SHA1 Message Date
TacoTheDank 3de9e46702 Add platform-29_r04 2020-07-18 19:22:50 -04:00
TacoTheDank cfed640adc Add buildTools 30.0.1 2020-07-18 19:17:42 -04:00
Marcus f5a32aa69e Merge branch 'add_gradle_651' into 'master'
Add gradle 6.5.1

See merge request fdroid/fdroidserver!775
2020-07-13 18:44:14 +00:00
Hans-Christoph Steiner c8b0e9d509 Merge branch 'log_timestamp' into 'master'
Add timstamp to logs

See merge request fdroid/fdroidserver!774
2020-07-07 15:12:13 +00:00
xynngh 393f08ff25 Add gradle 6.5.1 2020-07-07 14:02:56 +04:00
Jochen Sprickerhof d8b73dd241 Add timstamp to logs 2020-07-04 18:56:10 +02:00
Hans-Christoph Steiner 174bbf0bdc
jenkins-setup-build-environment: fix my stoopid typo 2020-06-29 09:56:37 +02:00
Hans-Christoph Steiner e659a5353d
provision-apt-get-install: gpg needs to come from stretch-backports
https://gitlab.com/eighthave/fdroiddata/-/jobs/611438997
Otherwise it fails to install:

```
  gpg : Depends: gpgconf (= 2.2.20-1~bpo9+1)
        Depends: libassuan0 (>= 2.5.0) but 2.4.3-2 is to be installed
        Depends: libgpg-error0 (>= 1.35) but 1.26-2 is to be installed
```

* 7e8c244473
* !769
2020-06-25 18:08:31 +02:00
Hans-Christoph Steiner f70a4e11dd
jenkins-setup-build-environment: --skip-scan on old builds
The scanner has gotten stricter: !758

[skip ci] since this only affects jenkins.debian.net
2020-06-25 11:53:29 +02:00
Marcus 1902bfb6fc Merge branch 'fix_760' into 'master'
update.py (and index.py) refactoring

Closes #524 and #760

See merge request fdroid/fdroidserver!756
2020-06-24 22:37:51 +00:00
Marcus Hoffmann 3ebc44c54f fix tests after they switched to yaml 2020-06-25 00:07:42 +02:00
Marcus Hoffmann 07caa88920 don't include the localized metadata things for /archive
We haven't done this so far and it's a potential big change in archive
size and update performance.
2020-06-25 00:07:42 +02:00
Marcus Hoffmann d720c99ae5 refactor update.py:main
This makes update.py:main a lot saner by removing a lot of the
implicit assumptions between the different stages of generating the
repository index.

* mostly unify repo and archive processing, that means the archive is
  now actually getting the same treatment regarding i.e. fastlane data.
  Previously the archive didn't get considered at all here.
* already filter the list of apps to include in a repo in update.py and
  give that prefiltered list to index. This makes sure we actually only
  copy fastlane/triple-t/etc. stuff for apps ending up in the index.
  This both, can save a lot of time if there are a lot of old /build
  dirs lying around and doesn't clutter /repo with things that aren't
  referenced from the index.

Closes fdroid/fdroidserver#524
2020-06-24 23:53:36 +02:00
Marcus Hoffmann e66683720b move index sorting to index module
This is a historic detail of the index format, so move it there.
For wiki update and status json there's really no reason why this should
be done in alphabetic app name order. Use the default sort order by
appid.
2020-06-24 23:53:36 +02:00
Marcus Hoffmann 8c71637d43 update: make copy_triple_t_store_metadata and insert_localized_app_metadata not assume /repo
This will enable copying the localized metadata to the archive as well.
2020-06-24 23:53:36 +02:00
Marcus Hoffmann ee4ee85cbd update:archive_old_apks: handle apps with no CVC
If an app doesn't have a CVC, we can just skip any special archive
handling.
Also rename weirdly named `res` to `apkList`.
2020-06-24 23:53:36 +02:00
Marcus Hoffmann ee162d9e62 gitignore tests/repo/status 2020-06-24 23:53:35 +02:00
Marcus Hoffmann 03881154c6 metadata: make linkresolver an actual object
Previously this was magically capturing the apps dict when passing it around as a
function. This also moved the code to the metadata module.

Add a test doing read_metadata where the linkresolver is used. This
happens when the apps we read have a [[app.id]] link to another app.
2020-06-24 23:53:35 +02:00
Marcus Hoffmann a4177e5ec3 add test for correct whatsnew handling without CVC 2020-06-24 23:53:35 +02:00
Marcus ed8c87950d Merge branch 'replacement-var-abs-path' into 'master'
Replacement var abs path

Closes #725

See merge request fdroid/fdroidserver!770
2020-06-24 21:01:35 +00:00
Hans-Christoph Steiner 5fce16c847
remove unused, confusing arg to common.getsrclib() 2020-06-24 22:40:27 +02:00
Hans-Christoph Steiner d0f426e076
replace $$srclib$$ with an absolute path
closes #725
2020-06-24 22:40:26 +02:00
Marcus b3f5176654 Merge branch 'fix-bugs' into 'master'
Fix bugs

Closes #725

See merge request fdroid/fdroidserver!769
2020-06-24 20:04:51 +00:00
Hans-Christoph Steiner 1e6f99988a fix typo when looking for build.gradle.kts 2020-06-24 21:43:58 +02:00
Hans-Christoph Steiner 7e8c244473 provision-apt-get-install: ensure gpg is installed
The basebox installs gpg, but when this script is used to provision
a GitLab CI Runner, gpg was missing.  It is used in some builds.
2020-06-24 21:43:58 +02:00
Hans-Christoph Steiner 238f048257 update: fix crash when liberapay: or open_collective: not in FUNDING.yml
closes #799
2020-06-24 21:39:43 +02:00
Hans-Christoph Steiner 82eceebd13 Merge branch 'parse-donation-links-from-funding.yml' into 'master'
update: insert donation links based on FUNDING.yml

See merge request fdroid/fdroidserver!754
2020-06-18 04:44:08 +00:00
Hans-Christoph Steiner 0b4112f743 Merge branch 'scanner-safe_path_test' into 'master'
scanner: add test with abs/rel paths as run from `fdroid build`

See merge request fdroid/fdroidserver!768
2020-06-17 08:56:42 +00:00
Hans-Christoph Steiner 2c4e9beacb scanner: add test with abs/rel paths as run from `fdroid build`
https://gitlab.com/fdroid/fdroidserver/-/issues/791#note_361018050
!767
2020-06-17 10:33:55 +02:00
Hans-Christoph Steiner 62c8fd5999
add Liberapay: field with username as data
Liberapay was originally included using a numeric ID, since they had
not yet finalized the public URLs.  Now it is a username.  So this
logic prefers the username in Liberapay: field, and keeps the old
LiberapayID: to ease migration.  LiberapayID: will not override
Liberapay:.  Clients are expected to prefer Liberapay: over LiberapayID:
2020-06-16 15:35:28 +02:00
Hans-Christoph Steiner 0183592526
update: insert donation links based on FUNDING.yml
GitHub has specified FUNDING.yml, a file to include in a git repo for
pointing people to donation links.  Since F-Droid also points people
to donation links, this parses them to fill out Donate:
and OpenCollective:.  Specifying those in the metadata file takes
precedence over the FUNDING.yml.  This follows the same pattern as how
`fdroid update` includes Fastlane/Triple-T metadata.  This lets the
git repo maintain those specific donations links themselves.

https://help.github.com/en/articles/displaying-a-sponsor-button-in-your-repository#about-funding-files

The test file was generated using:

```python
import os, re, yaml

found = dict()
for root, dirs, files in os.walk('.'):
    for f in files:
        if f == 'FUNDING.yml':
            with open(os.path.join(root, f)) as fp:
                data = yaml.safe_load(fp)
            for k, v in data.items():
                if k not in found:
                    found[k] = set()
                if not v:
                    continue
                if isinstance(v, list):
                    for i in v:
                        found[k].add(i)
                else:
                    found[k].add(v)

            with open('gather-funding-names.yaml', 'w') as fp:
                output = dict()
                for k, v in found.items():
                    output[k] = sorted(v)
                yaml.dump(output, fp, default_flow_style=False)
```
2020-06-16 15:35:26 +02:00
Marcus 8d517d4583 Merge branch 'simple_apk_scanner' into 'master'
Simple apk scanner

See merge request fdroid/fdroidserver!766
2020-06-15 18:03:19 +00:00
Marcus f84818c15e scanner: add a simple scan for blacklisted classes after build step
add com.android.billing to blacklist, see
https://gitlab.com/fdroid/fdroiddata/-/issues/2070#note_360611289
2020-06-15 18:03:19 +00:00
Marcus 6986e73506 Merge branch 'allow_appcenter' into 'master'
scanner: allow microsoft appcenter

See merge request fdroid/fdroidserver!764
2020-06-15 10:46:51 +00:00
Marcus Hoffmann 670328c9e5 scanner: allow microsoft appcenter, except appcenter-push
This sdk is open source. It was added because appcenter has a dependency
to play-services. It's possible though to build an app using appcenter
that doesn't pull in play services, so we can't blanket ban the sdk.

The appcenter-push modules has obvious refenrences to firebase, so it's
safe to error on that.

Ref: https://phabricator.wikimedia.org/T254980
2020-06-15 12:24:46 +02:00
Marcus 12f4ae3039 Merge branch 'fix_791' into 'master'
scanner: fix wrong path being passed to function

Closes #791

See merge request fdroid/fdroidserver!767
2020-06-13 11:03:30 +00:00
Marcus Hoffmann 3a9f1e845c scanner: fix wrong path being passed to function
Also make rename the parameter in safe_path to make it clear that this
is just a relative path.

Closes fdroid/fdroidserver#791.
2020-06-13 03:01:54 +02:00
Marcus be814bf002 Merge branch 'importsubmodules' into 'master'
Import - Detect .gitmodules and add submodules build line

See merge request fdroid/fdroidserver!765
2020-06-12 19:26:23 +00:00
Licaon_Kter 72fde0f10b Detect .gitmodules and add submodules build line 2020-06-11 16:36:54 +03:00
Marcus Hoffmann 294799e2f5 scanner: fix options handling
closes fdroid/fdroidserver#789
2020-06-11 11:44:00 +02:00
Marcus 0e025e2ae9 Merge branch 'scanner-overhaul' into 'master'
scanner file type overhaul

Closes #394

See merge request fdroid/fdroidserver!758
2020-06-11 09:23:39 +00:00
Marcus 95623361ae Merge branch 'master' into 'master'
Update NDK to r21d and add buildtools 30.0.0

See merge request fdroid/fdroidserver!763
2020-06-10 20:21:26 +00:00
TacoTheDank 07992177bf Add build tools 30.0.0 2020-06-10 15:59:04 -04:00
TacoTheDank 0216513da6 Upgrade NDK r21c to r21d 2020-06-10 15:55:53 -04:00
Hans-Christoph Steiner 0837289935 scanner: fix regex for matching URLs in gradle maven{} blocks
closes #465

This script generated gradle-maven-blocks.yaml:
```python
import os
import re
import yaml

pat = re.compile(r'\smaven\s*{[^}]+}')

finds = set()
for root, dirs, files in os.walk('.'):
    for f in files:
        if '.gradle' in f:
            with open(os.path.join(root, f), errors='surrogateescape') as fp:
                contents = fp.read()
            for m in pat.findall(contents):
                finds.add(m)

with open('finds.yaml', 'w') as fp:
    yaml.dump(sorted(finds), fp, default_flow_style=False)

```
2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner 6590f3869e scanner: error/warn on dex/gz/zip, closes #394 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner d7b3bca1e7 build: pass --verbose flag through to scanner 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner ee54dbce87 scanner: safely check options, for things using this API
https://gitlab.com/fdroid/fdroidserver/-/issues/771#note_353495799
2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner 75acf63efa scanner: remove all gradle wrapper files 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner 2f5d780c49 scanner: always setup JSON data structure so functions work as API 2020-06-10 19:42:23 +02:00