Commit Graph

126 Commits

Author SHA1 Message Date
Marcus f84818c15e scanner: add a simple scan for blacklisted classes after build step
add com.android.billing to blacklist, see
https://gitlab.com/fdroid/fdroiddata/-/issues/2070#note_360611289
2020-06-15 18:03:19 +00:00
Marcus 6986e73506 Merge branch 'allow_appcenter' into 'master'
scanner: allow microsoft appcenter

See merge request fdroid/fdroidserver!764
2020-06-15 10:46:51 +00:00
Marcus Hoffmann 670328c9e5 scanner: allow microsoft appcenter, except appcenter-push
This sdk is open source. It was added because appcenter has a dependency
to play-services. It's possible though to build an app using appcenter
that doesn't pull in play services, so we can't blanket ban the sdk.

The appcenter-push modules has obvious refenrences to firebase, so it's
safe to error on that.

Ref: https://phabricator.wikimedia.org/T254980
2020-06-15 12:24:46 +02:00
Marcus Hoffmann 3a9f1e845c scanner: fix wrong path being passed to function
Also make rename the parameter in safe_path to make it clear that this
is just a relative path.

Closes fdroid/fdroidserver#791.
2020-06-13 03:01:54 +02:00
Marcus Hoffmann 294799e2f5 scanner: fix options handling
closes fdroid/fdroidserver#789
2020-06-11 11:44:00 +02:00
Hans-Christoph Steiner 0837289935 scanner: fix regex for matching URLs in gradle maven{} blocks
closes #465

This script generated gradle-maven-blocks.yaml:
```python
import os
import re
import yaml

pat = re.compile(r'\smaven\s*{[^}]+}')

finds = set()
for root, dirs, files in os.walk('.'):
    for f in files:
        if '.gradle' in f:
            with open(os.path.join(root, f), errors='surrogateescape') as fp:
                contents = fp.read()
            for m in pat.findall(contents):
                finds.add(m)

with open('finds.yaml', 'w') as fp:
    yaml.dump(sorted(finds), fp, default_flow_style=False)

```
2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner 6590f3869e scanner: error/warn on dex/gz/zip, closes #394 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner ee54dbce87 scanner: safely check options, for things using this API
https://gitlab.com/fdroid/fdroidserver/-/issues/771#note_353495799
2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner 75acf63efa scanner: remove all gradle wrapper files 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner 2f5d780c49 scanner: always setup JSON data structure so functions work as API 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner 84f225f2f2 scanner: make problem descriptions translationable 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner d898ad0412 scanner: make AARs and JARs trigger an error
refs #491
2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner 3bc246ccad scanner: ignore well known image types that are set executable 2020-06-10 19:42:23 +02:00
Hans-Christoph Steiner 6030445be0 logging.warn() was deprecated in Python 3.3, use logging.warning()
sed -i 's,logging\.warn(,logging.warning(,g' fdroid */*.*

https://docs.python.org/3.3/library/logging.html#logging.Logger.warning
2020-05-27 08:35:14 +02:00
Hans-Christoph Steiner 67332d83a5 scanner: add --json option for outputting machine readable results
* makes per-build entries in per-app entries
* `fdroid scanner --json --verbose` will output logging messages to stderr
* removed " at line N" from one message to make them uniform keys
* this will be used in issuebot

This is a second attempt with tests for how `fdroid build` calls the
scanner functions. closes #771.  It was previously merged in !748 then
reverted in 68c072c72e
2020-05-27 08:35:14 +02:00
Marcus Hoffmann 68c072c72e Revert "scanner: add --json option for outputting machine readable results"
This reverts commit cdaf62e5d9.

See: https://gitlab.com/fdroid/fdroidserver/-/merge_requests/748#note_347769371
2020-05-24 10:24:40 +02:00
Hans-Christoph Steiner cdaf62e5d9 scanner: add --json option for outputting machine readable results
* makes per-build entries in per-app entries
* `fdroid scanner --json --verbose` will output logging messages to stderr
* removed " at line N" from one message to make them uniform keys
* this will be used in issuebot
2020-05-14 21:58:48 +02:00
Hans-Christoph Steiner df563d339a
fix pep8 E741 ambiguous variable name 'l' 2020-05-14 12:11:36 +02:00
Hans-Christoph Steiner ad92b4c678 scanner: add --force option for scanning disabled apps/builds 2020-02-13 13:58:13 +01:00
licaon-kter 135c92f3d0 Add Microsoft Appcenter SDK to forbidden list 2020-02-02 03:50:54 +02:00
Hans-Christoph Steiner b95f66a806
scanner: only allow HTTPS versions of the whitelist 2019-11-13 11:59:24 +01:00
Mike Hardy 3c398e2aa6 remove redundant google analytics match
'google.*play.*services' and 'firebase' will catch both manners of including analytics I think
https://developers.google.com/android/guides/setup#split
https://firebase.google.com/support/release-notes/android#latest_sdk_versions

This has the desired side effect of also allowing the libre analytics library
'net.mikehardy:google-analytics-java' and 'net.mikehardy:google-analytics-java7'
2019-11-02 18:40:58 +00:00
relan bd96a17ef2 scanner: blacklist Bugly
The artifact is licensed under "The Bugly Software License, Version 1.0"
[1], but link to the full text [2] is broken. LICENSE file in the source
code repository is empty.

I guess this library is non-free.

[1] https://mvnrepository.com/artifact/com.tencent.bugly/crashreport/2.8.6.0
[2] http://bugly.qq.com/licenses/LICENSE-1.0.txt
[3] https://github.com/BuglyDevTeam/Bugly-Android
2019-09-28 07:58:19 +03:00
relan 3a7ad650e3 scanner: fix local Debian Maven repo handling
The resulting regex was 'https?://file:///usr/share/maven-repo' causing
scanner error.
2019-08-29 21:32:54 +03:00
Hans-Christoph Steiner f0460dea6e scanner: allow local Debian Maven repo file:///usr/share/maven-repo
It is now possible to build an app using only things in Debian.  Since the
buildserver will always control the contents of file:///usr/share/maven-repo,
it is the most safe repo there is.
2019-05-01 21:16:15 +02:00
Pierre Rudloff b3754af137 Add cloudrail to blacklist 2019-02-07 10:26:03 +01:00
Hans-Christoph Steiner 57556aceee remove redundant open() arg: encoding='utf8'
By default, open() returns a str:
https://docs.python.org/3/library/functions.html#open

By default, str is UTF-8:
https://docs.python.org/3/library/stdtypes.html#str

This used to matter on Python 2.x, but this code is 3.x only now.
2018-10-19 15:01:34 +02:00
Marcus Hoffmann a6dfbc6e2e scanner: add facebook sdk to forbidden libraries
These are sourceavailable but not under a free license.
I made sure that this matches only the facebook sdk's from here:
https://github.com/facebook/facebook-android-sdk and not some real open
source libraries by facebook (fresco, stetho, ...). These seem to be
under a different namespace.

fdroid/fdroidserver#534
2018-08-10 15:20:01 +02:00
Marcus Hoffmann 753e9c8175 scanner: properly combine all gradle compile commands with flavors 2018-07-25 19:00:34 +02:00
Marcus Hoffmann c44ab190bd scanner: fix reformatting fail
This was broken by 44e8f425d :-(.
2018-07-25 19:00:37 +02:00
Marcus Hoffmann 3a6942ca77
scanner: whitelist www.jitpack.io
Only the non-www version was recognized so far.

Closes #498
2018-06-11 12:40:55 +02:00
Hans-Christoph Steiner 44e8f425d5 fix PEP8 formatting issue
fdroidserver/scanner.py:35:34: E241 multiple spaces after ','
fdroidserver/scanner.py:36:35: E241 multiple spaces after ','
fdroidserver/scanner.py:37:30: E241 multiple spaces after ','
fdroidserver/scanner.py:38:41: E241 multiple spaces after ','
fdroidserver/scanner.py:39:30: E241 multiple spaces after ','
fdroidserver/scanner.py:40:38: E241 multiple spaces after ','
fdroidserver/scanner.py:41:38: E241 multiple spaces after ','
2018-03-08 13:55:07 +01:00
relan 44d17663fd scanner: support Gradle plugin 3.0 syntax for dependencies
See https://developer.android.com/studio/build/gradle-plugin-3-0-0-migration.html#new_configurations
2018-03-08 15:23:15 +03:00
Hans-Christoph Steiner 32213ef040 scanner: allow running without versionCode and as API
This lets `fdroid scanner my.package.name` run without requiring that the
versionCode is also specified.  It also allows scanner.scan_source() to be
called as a function in the public API of fdroidserver.
2018-01-26 10:18:41 +01:00
Jan Berkel 62d1c672f3 whitelist some open-source firebase libs 2017-12-14 22:03:48 +01:00
Hans-Christoph Steiner ffc91e301a make _ always be the gettext function, nothing else
This avoids hard bugs where the _() function gets overidden by a str or
something else.
2017-10-25 23:01:25 +02:00
Hans-Christoph Steiner ab02a6fcc3 Revert "scanner: promote jar and aar files present in the repo to errors"
This reverts commit 5f5d3ea896.

These new scanner rules for JARs causes most builds on fdroiddata to
break.  For example, at least the past 20 releases of
org.fdroid.fdroid.  fdroiddata needs to be cleaned up and prepared for
this before we can include 5f5d3ea896

I'm running fdroid scanner on the whole archive now, I'll post the log
to an issue once its done.  Ignoring test files would eliminate some
of these build failures, at least for most org.fdroid.fdroid builds.

refs #!325
2017-10-25 22:55:47 +02:00
Hans-Christoph Steiner 53ce81179c Merge branch 'error_on_jars' into 'master'
RFC: Error on jars

See merge request fdroid/fdroidserver!325
2017-10-12 11:46:59 +00:00
Hans-Christoph Steiner 6c7b987239 fix string formats that are ambiguous for translators 2017-09-16 13:19:38 +02:00
Hans-Christoph Steiner fa0d618d2f rename 'app-id' to standard Android 'applicationId'
* https://developer.android.com/studio/build/application-id.html
* https://sites.google.com/a/android.com/tools/tech-docs/new-build-system/applicationid-vs-packagename

This only changes the term in the human texts, not var names or CLI flags.
2017-09-15 11:41:51 +02:00
Hans-Christoph Steiner 278d67d960 implement gettext localization
This allows all the text to be localized via Weblate.  This is a quick
overview of all the strings, but there are certainly some that were left
out.

closes #342
2017-09-15 11:39:00 +02:00
Hans-Christoph Steiner cb10f0df09 standardize os.walk() var names based on Python 3.5 docs
There were multiple conventions used in the code, but mostly it was already
using the convention from the docs, so this converts things to using that
convention:

https://docs.python.org/3/library/os.html#os.walk
2017-09-15 11:37:21 +02:00
Marcus Hoffmann 5f5d3ea896 scanner: promote jar and aar files present in the repo to errors
This makes sure we we don't include any prebuilt code in the build process.
2017-09-11 17:05:24 +02:00
Marcus Hoffmann 15c4492625 remove gradle-wrapper.jar during scan
We make sure to not use the gradle wrapper anyway so this should have no
impact.
2017-09-11 17:05:21 +02:00
Marcus Hoffmann af2eb2bd50 scanner: warn on aar files 2017-09-11 17:05:16 +02:00
Marcus Hoffmann 5d9eb9b295 scanner: rename variables, use os.path.relpath
* rename fd -> path_in_build_dir
* rename fp -> filepath
* use os.path.reldir instead of string manipulation
2017-09-11 16:49:39 +02:00
Marcus Hoffmann 1ee3f9aad5 scanner: more descriptive variable names from os.walk
This uses the names from python documentation of os.walk().
2017-09-11 16:49:30 +02:00
Boris Kraut c056b95fd5 scanner: allow google maven 2017-05-26 15:51:38 +02:00
Hans-Christoph Steiner 17cd07f1a7 fix pylint unused-argument 2017-05-23 22:34:16 +02:00
Torsten Grote 1fcd8e63a3
Replace sys.exit() in non-main functions by exceptions
Also move all exceptions into one module
2017-05-22 16:51:11 -03:00