sourcehut
/
sr.ht-docs
mirror of https://git.sr.ht/~sircmpwn/sr.ht-docs
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

Update top-level installation/configuration pages

This commit is contained in:
Jason Phan 2020-11-02 12:37:28 -06:00 committed by Drew DeVault
parent b201f05d92
commit f3cc1dcd59
4 changed files with 263 additions and 262 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

137
configuration.md Normal file
View File

@ -0,0 +1,137 @@
---
title: Configuration
---
This document provides general information about the configuration process of
sr.ht services.
Similar to the installation process, configuring sr.ht services typically
involves steps that are specific each service. That being said, they all share
certain aspects which are described below. Be sure to take a look at each
service's configuration page for more details:
- [builds.sr.ht](/builds.sr.ht/configuration.md)
- [dispatch.sr.ht](/dispatch.sr.ht/configuration.md)
- [git.sr.ht](/git.sr.ht/configuration.md)
- [hg.sr.ht](/hg.sr.ht/configuration.md)
- [lists.sr.ht](/lists.sr.ht/configuration.md)
- [man.sr.ht](/man.sr.ht/configuration.md)
- [meta.sr.ht](/meta.sr.ht/configuration.md)
- [paste.sr.ht](/paste.sr.ht/configuration.md)
- [todo.sr.ht](/todo.sr.ht/configuration.md)
# Prerequisites
Before configuring any sr.ht service, ensure that you have the following
services installed. Every sr.ht service makes use of them, so your deployment
will likely not work as expected if they are missing.
- [core.sr.ht][core] - Provides functionality used by all sr.ht services.
- For users installing from packages, this package is automatically pulled
in as a dependency.
- For users installing from source, this package must be built *first*.
<div class="alert alert-info">
<strong>Note:</strong> The dependencies in <code>setup.py</code> may
not be up-to-date. For an up-to-date list of dependencies, consult
the latest
<a href="https://git.sr.ht/~sircmpwn/sr.ht-apkbuilds/tree/master/sr.ht/py3-srht/APKBUILD"
class="alert-link">Alpine package</a>.
</div>
- [meta.sr.ht][meta] - Performs various administrative tasks (e.g., storing
account details, handling logins, SSH and PGP key storage).
[meta]: https://git.sr.ht/~sircmpwn/meta.sr.ht
[core]: https://git.sr.ht/~sircmpwn/core.sr.ht
# Files
## `config.ini`
sr.ht services all use a shared configuration file, located at
`/etc/sr.ht/config.ini`. The specific options you will configure will depend on
the subset of and configuration of sr.ht services you choose to run.
Each service provides an example configuration file, called
`config.example.ini`, in their source code repository. You should consult these
when creating your unified `config.ini`. Additionally, you may refer to each
service's configuration reference pages for a full list of configuration
options:
- [sr.ht](/sr.ht/configuration_reference.md)
- [builds.sr.ht](/builds.sr.ht/configuration_reference.md)
- [dispatch.sr.ht](/dispatch.sr.ht/configuration_reference.md)
- [git.sr.ht](/git.sr.ht/configuration_reference.md)
- [hg.sr.ht](/hg.sr.ht/configuration_reference.md)
- [lists.sr.ht](/lists.sr.ht/configuration_reference.md)
- [man.sr.ht](/man.sr.ht/configuration_reference.md)
- [meta.sr.ht](/meta.sr.ht/configuration_reference.md)
- [paste.sr.ht](/paste.sr.ht/configuration_reference.md)
- [todo.sr.ht](/todo.sr.ht/configuration_reference.md)
## Service Files
Service files for daemons are pre-configured by their distribution packages.
Thus, after setting up your `config.ini` appropriately, you may start them
using your distribution's service manager (e.g., `service git.sr.ht start`).
# Databases
Each service requires its own database. It is recommended to give each service
its own database login, with full permissions for its database so that it may
manage its own schema migrations.
Once you populate the `connection-string` field in your `config.ini`, you may
use the `<module>-initdb` script to populate the schema and stamp the latest
revision for migrations (e.g., `metasrht-initdb` to set up meta.sr.ht's
database). Note that if you have not configured SSL for your PostgreSQL server,
you may have to append `?sslmode=disable` to your connection string for some
services to work.
## Schema Upgrades
We use [alembic](https://alembic.zzzcomputing.com/en/latest/) to manage schema
migrations. We use custom scripts to automatically retrieve database
credentials from your `config.ini`. If you have installed from distribution
packages, set `[<service>] migrate-on-upgrade=yes` to have migrations
automatically performed during normal service upgrades (e.g., `[meta.sr.ht]
migrate-on-upgrade=yes`).
Otherwise, you may use `srht-migrate <service> upgrade head` to run updates for
core.sr.ht migrations, and `<service>-migrate upgrade head` to run
service-specific upgrades. For example, to upgrade the database schema for
git.sr.ht, run `srht-migrate git.sr.ht upgrade head`, then `gitsrht-migrate
upgrade head`. Other alembic commands are available, use `gitsrht-migrate
--help` for more information.
# Upgrade Procedure
1. Stop all services.
2. Update and upgrade your packages (e.g. `apk update && apk upgrade`).
3. Resume all services.
If you have high availability requirements, the web services may be
load-balanced to avoid an outage.
Database migrations will be performed automatically during upgrades. We source
your alembic config from your main `config.ini`, so there's no need to write an
`alembic.ini` file.
Run `srht-migrate <service> stamp head && <service>-migrate stamp head` once to
tell alembic the schema is up-to-date (e.g. `srht-migrate man.sr.ht stamp head
&& mansrht-migrate stamp head`). Future upgrades will be managed automatically
by the package, or if you're using the source code you can run `srht-migrate
<service> upgrade head && <service>-migrate upgrade head` when you pull the
latest version down.
# Integrations
Many sr.ht services are able to integrate with third-party tools, such as:
- [Prometheus](https://prometheus.io/) for monitoring.
- Any S3-compatible object storage (e.g., [Minio](https://min.io/),
[Ceph](https://ceph.io/)).

27
hacking.md
View File

@ -3,23 +3,22 @@ title: Hacking on SourceHut
toc: false
---
For developers interested in hacking on SourceHut, an expedited installation
procedure is available. The bare minimum is a PostgreSQL database and a Redis
server, and you can generally skip the webhooks daemons unless you need them.
It is recommended that you still use our distribution packages to obtain the
dependencies, though.
This document provides information for those interested in hacking on or
contributing to SourceHut.
If you don't intend to work on core.sr.ht, install the package to skip this
step. If you don't intend to work on meta.sr.ht, it may be easier to install the
package and configure it normally on localhost than to try and get it up and
running from source.
# Installation
You can also *usually* get away without configuring the outgoing mail settings,
but you may still have to generate a PGP key to get things started up. You can
also skip the reverse proxy and use our development servers via `python3
run.py`.
Here are some tips to expedite the installation procedure:
# Sending patches
* A PostgreSQL database and Redis server are always required.
* Generally, you can skip the webhooks daemons unless you need them.
* Always prefer package installations for obtaining dependencies.
* You can *usually* get away without configuring the outgoing mail settings,
but you may still have to generate a PGP key to get things started up.
* You can skip setting up a reverse proxy and use our development servers via
`python3 run.py` instead.
# Sending Patches
We use [git send-email](https://git-send-email.io) and discuss patches (and
other development topics) on the

257
installation.md
View File

@ -2,20 +2,30 @@
title: Installation
---
sr.ht is a distributed system. Each service, such as git.sr.ht and builds.sr.ht,
runs independently, with its own database and resources. They communicate with
each other using mainly their respective APIs and webhooks, and are mostly
tolerant to the temporary or permanent absence of their peers. The system is
fairly complex, and is designed more for deployment at scale than for
small-scale installations. However, you can choose a subset of these services to
install as your needs demand, and it will work correctly.
This document covers the installation steps common to all sr.ht services.
Developers wishing to hack on or contribute to sr.ht, see
[hacking.md](/hacking.md).
<div class="alert alert-success">
If you would like to hack on or contribute sr.ht, see the
<a href="/hacking.md" class="alert-link">Hacking</a> page.
</div>
This page documents details common to the installation of all packages, but many
services have specific needs. Specific details to each service are available on
each service's installation page:
<div class="alert alert-warning">
<strong>Warning:</strong> Currently, the only officially supported method for
installing sr.ht software is through packages on Alpine Linux hosts.
</div>
sr.ht is a distributed system in which each service (e.g., git.sr.ht,
builds.sr.ht) runs independently with its own database and resources,
communicates with other services through their respective APIs and webhooks,
and is highly tolerant to the temporary or permanent absence of its peers.
This type of design allows you to choose and deploy any subset of services
depending on your needs, making it ideal for both large-scale deployments and
small-scale installations.
However, being distributed also means that the installation and deployment
process of can become a bit more involved. In particular, many sr.ht services
have their own, unique requirements that necessitate extra installation steps.
Consult each service's installation page for more details:
- [builds.sr.ht](/builds.sr.ht/installation.md)
- [dispatch.sr.ht](/dispatch.sr.ht/installation.md)
@ -27,218 +37,41 @@ each service's installation page:
- [paste.sr.ht](/paste.sr.ht/installation.md)
- [todo.sr.ht](/todo.sr.ht/installation.md)
All sr.ht sysadmins are encouraged to subscribe to the
[sr.ht-admins](https://lists.sr.ht/~sircmpwn/sr.ht-admins) mailing list, a
# sr.ht-admins
If you are a sr.ht system administrator, we encourage you to to subscribe to
the [sr.ht-admins](https://lists.sr.ht/~sircmpwn/sr.ht-admins) mailing list, a
low-volume list of sysadmin-oriented announcements regarding breaking changes,
security vulnerabilities, and so on.
# Packages
The only supported installation is Alpine Linux hosts using the SourceHut
[Alpine Linux package repository](/packages.md). There are also
community-maintained SourceHut distributions for [Debian and Arch
Linux](/packages.md), which are preferred before installing SourceHut from
source.
SourceHut is still in alpha, and has no stable releases. If you are interested
in packaging it for your distribution, please reach out to us for help in
automating the maintenance of a third-party distribution which is automatically
updated following our (frequent) upstream rolling releases. Please refrain from
packaging SourceHut for your upstream distribution repositories until we have
shipped stable versions of our software.
Packages also ship with the correct users and groups rigged up, and with
whichever daemon configurations are particular to your distribution's init
system (e.g. OpenRC scripts or systemd units).
# Prerequisites
Generally speaking, most services require the following services:
Most sr.ht services require the following:
- A PostgreSQL server: persistent storage
- A Redis server: ephemeral storage, caching, work distribution
- A mail server: incoming and outgoing mail
- A cron daemon: running scheduled tasks
- A [PostgreSQL](https://www.postgresql.org/) server - persistent storage
Additionally, many services are able to integrate with some optional tools:
- A [Redis](https://redis.io) server - ephemeral storage, caching, work
distribution
- [Prometheus](https://prometheus.io/) for monitoring
- Any S3-compatible object storage, such as [Minio](https://min.io/) or
[Ceph](https://ceph.io/)
- A mail server - incoming/outgoing mail
# Installing SourceHut services
- A cron daemon - scheduled tasks
These instructions apply generally to all services. Consult service-specific
documentation for amendments to these procedures.
# Installing From Packages
## sr.ht core
1. Add the appropriate [SourceHut package repository](/packages.md) to your
package manager.
[core.sr.ht](https://git.sr.ht/~sircmpwn/core.sr.ht) is a Python package which
provides common functionality for all services. For users installing from
packages, it is not necessary to concern yourself with this, as it is
automatically pulled in as a dependency. For users building from source, you
must build this package first. Be aware that the dependencies in `setup.py` are
not generally kept up-to-date, consult the latest
[Alpine package][alpine package] for an up-to-date and comprehensive list.
2. Install your sr.ht services!
[alpine package]: https://git.sr.ht/~sircmpwn/sr.ht-apkbuilds/tree/master/sr.ht/py3-srht/APKBUILD
<div class="alert alert-info">
<strong>Note:</strong> Each package repository has a dedicated maintainer,
listed on the packages page, who should be your first stop for any
distribution-related issues. Alpine Linux is officially maintained by
SourceHut; the other repositories are community-maintained.
</div>
## meta.sr.ht
# Next Steps
meta.sr.ht is the only service which is required. It is responsible for storing
your account details and profile, handling logins, storing SSH and PGP keys,
keeping a secure audit log, and providing some management interfaces for
administrators.
Consult the [meta.sr.ht installation guide](/meta.sr.ht/installation.md) for
details on the installation procedure.
## Service configuration
All services use a shared configuration file at `/etc/sr.ht/config.ini`. Each
site provides an example configuration in `config.example.ini` in their
respective source code repositories. It is the administrator's responsibility to
consult these examples to produce a unified configuration file which is
applicable to all of the services which they intend to operate.
Some configuration options are applicable to all services.
### Service keys
Encryption is used throughout SourceHut to encrypt and validate communications.
There are three main keys which you need to generate: service keys, the network
key, and the webhook key.
Service keys are used to encrypt session cookies, and if you configure
load-balancing, it must be consistent between all nodes of that service. They
can be generated with `srht-keygen service`.
The network key needs to be consistent throughout all services and nodes in your
sr.ht installation. It is used to encrypt and sign internal communications
between services. This key is generated with `srht-keygen network`.
The webhook key is also consistent throughout all services and nodes, but is an
asymmetric key. It is used to sign webhook payloads to validate the authenticity
of the request. You can generate this with `srht-keygen webhook`, store the
private key in your config file and distribute to the public key to any parties
interested in authenticating webhook payloads from your services. This is also
used to validate webhooks used internally; it is not optional.
### Mail configuration
Outgoing emails from sr.ht are configured in the `[mail]` section. If you fill
out the `error-to` address, the services will send exceptions to this address
for diagnostics. You must also generate a PGP key (without a password) for sr.ht
services to sign outgoing emails with (and optionally encrypt emails to each
recipient).
## Database Configuration
Each service requires its own database, though they can co-exist on the same
server. It is also recommended to give each service its own database login, with
full access rights to that database so that it may manage its own schema
migrations.
After you populate your `config.ini`'s connection string, you may use the
`[module]-initdb` script to populate the schema and stamp the latest revision
for migrations, for example `metasrht-initdb` to set up meta.sr.ht's database.
### Schema Upgrades
We use [alembic](https://alembic.zzzcomputing.com/en/latest/) to manage schema
migrations. We use custom scripts to automatically retrieve database credentials
from your main sr.ht config file. If you have installed from distribution
packages, set `[service] migrate-on-upgrade=yes` (where service is e.g.
`[meta.sr.ht]`) to have migrations automatically performed during normal service
upgrades.
Otherwise, you may use `srht-migrate <service> upgrade head` to run updates for
core.sr.ht migrations, and `<service>-migrate upgrade head` to run
service-specific upgrades. For example, to upgrade the database schema for
git.sr.ht, run `srht-migrate git.sr.ht upgrade head`, then `gitsrht-migrate
upgrade head`. Other alembic commands are available, use `gitsrht-migrate
--help` for more information.
## Upgrade procedure
1. Stop all services
2. Run your distro's update commands (e.g. `apk update && apk upgrade`)
3. Resume all services
We source your alembic config from your main sr.ht `config.ini` -
no need to write an `alembic.ini` file. Run `srht-migrate <service> stamp head
&& <service>-migrate stamp head` once to tell alembic the schema is up-to-date
(e.g. `srht-migrate man.sr.ht stamp head && mansrht-migrate stamp head`).
Future upgrades will be managed automatically by the package, or if you're
using the source code you can run `srht-migrate <service> upgrade head &&
<service>-migrate upgrade head` when you pull the latest version down.
## Start the daemons
Daemon configuration is provided in the distribution packages. Use the service
manager appropriate to your distro to start the daemons, e.g. `service git.sr.ht
start` or `systemctl start git.sr.ht`.
## Proxy configuration
The exact configuration you use will vary depending on your needs. Here is an
example nginx configuration for meta.sr.ht:
server {
listen 80;
server_name meta.sr.ht;
location / {
return 302 https://$server_name$request_uri;
}
location ^~ /.well-known {
root /var/www;
}
}
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name meta.sr.ht;
client_max_body_size 100M;
ssl_certificate /etc/ssl/uacme/meta.sr.ht/cert.pem;
ssl_certificate_key /etc/ssl/uacme/private/meta.sr.ht/key.pem;
location / {
proxy_pass http://127.0.0.1:5002;
}
location /static {
root /usr/lib/python3.6/site-packages/metasrht;
}
}
Once the proxy is configured, you should be able to access your new service.
## OAuth configuration
For services other than meta.sr.ht, you have to create and configure an OAuth
client before users can log into your service. To do this, visit your profile
on your meta.sr.ht instance, select the OAuth tab and register a new client.
Append the path `/oauth/callback` to the URL of your service instance and
choose this value as the base redirect URI (for example
`https://git.sr.ht/oauth/callback`). Update your service configuration file with
the generated client ID and secret.
You then need to configure that client as "preauthorized", i.e. first-party.
This skips the OAuth consent screen that third-party applications are subject to
and enables some extra API features. Log into the meta.sr.ht database and run
the following query (adjusting the client ID as appropriate):
```sql
update oauthclient set preauthorized = true where client_id = 'your client ID';
```
Now you should be able to log into your new service.
Alternatively, you can generally run most SourceHut services by pointing them at
*our* meta.sr.ht instance and registering them as an OAuth client. For example,
you could run a custom builds.sr.ht instance which logs in with hosted
meta.sr.ht accounts by registering for an OAuth client there and skipping the
preauthorized step. YMMV.
After installing your sr.ht services, you'll need to configure them. For more
information, see the [Configuration](/configuration.md) page.

104
packages.md
View File

@ -1,70 +1,102 @@
---
title: Package mirrors
title: Package Repositories
---
sr.ht operates a number of automatically-updated package repositories where you
can find packages for sr.ht software in various distributions at
SourceHut operates a number of package repositories for sr.ht software. These
repositories, which support various distributions, are available at
[mirror.sr.ht](https://mirror.sr.ht).
All of our package repositories accept patches as necessary on
[sr.ht-packages](https://lists.sr.ht/~sircmpwn/sr.ht-packages).
If you would like to contribute to the package repositories, we accept
patches on the [sr.ht-packages](https://lists.sr.ht/~sircmpwn/sr.ht-packages)
mailing list.
# Package Repositories
<div class="alert alert-warning">
<p>
<strong>Warning:</strong> SourceHut is still in alpha, and has no stable
releases. As such, we do not recommend packaging SourceHut for your upstream
distribution repositories until we have shipped stable versions of our
software.
</p>
<p>
If you are still interested in packaging it for your distribution, please
reach out to us for help in automating the maintenance of a third-party
distribution which is automatically updated following our (frequent)
upstream rolling releases.
</p>
</div>
## Alpine Linux
# Alpine Linux
This is the recommended distribution of SourceHut, and the one we use for the
hosted instance upstream. This must be the first line in `/etc/apk/repositories`
&mdash; we shadow some upstream packages with patched versions. You will need
the community repository enabled.
**Mirror**: https://mirror.sr.ht/alpine/v3.12/sr.ht
**Mirror URL**: `https://mirror.sr.ht/alpine/v3.12/sr.ht`
**Signing Key**: [alpine@sr.ht.pub](https://mirror.sr.ht/alpine/alpine%40sr.ht.rsa.pub)
```
$ wget -q -O /etc/apk/keys/alpine@sr.ht.rsa.pub https://mirror.sr.ht/alpine/alpine@sr.ht.rsa.pub
```
**Signing key**: [alpine@sr.ht.pub](https://mirror.sr.ht/alpine/alpine%40sr.ht.rsa.pub)
**Maintainer**: Drew DeVault <sir@cmpwn.com>
**Source**: https://git.sr.ht/~sircmpwn/sr.ht-apkbuilds
## Arch Linux
```sh
# Add the following line to /etc/apk/repositories
https://mirror.sr.ht/alpine/v3.12/sr.ht
**Mirror URL**: `https://mirror.sr.ht/archlinux/sr.ht`
# Install signing key
wget -q -O /etc/apk/keys/alpine@sr.ht.rsa.pub https://mirror.sr.ht/alpine/alpine@sr.ht.rsa.pub
```
[sr.ht]
Server = https://mirror.sr.ht/archlinux/sr.ht
# Update package index
apk update
```
**Signing Key**: `C0AAFC1676BD998617C94C42DC59670F1EB0A189`
<div class="alert alert-warning">
<strong>Warning:</strong> SourceHut shadows some upstream packages with
patched versions. The SourceHut mirror must be the <em>first</em> repository
in <code>/etc/apk/repositories</code>.
</div>
```
$ pacman-key --recv-keys C0AAFC1676BD998617C94C42DC59670F1EB0A189
$ pacman-key --lsign-key C0AAFC1676BD998617C94C42DC59670F1EB0A189
```
<div class="alert alert-warning">
<strong>Warning:</strong> The Alpine Linux community package repository must
be enabled.
</div>
# Arch Linux
**Mirror**: https://mirror.sr.ht/archlinux/sr.ht
**Signing key**: `C0AAFC1676BD998617C94C42DC59670F1EB0A189`
**Maintainer**: Eli Schwartz <eschwartz@archlinux.org>
**Source**: https://git.sr.ht/~sircmpwn/sr.ht-pkgbuilds
## Debian
```sh
# Add the following two lines to /etc/pacman.conf
[sr.ht]
Server = https://mirror.sr.ht/archlinux/sr.ht
**Mirror URL**: `https://mirror.sr.ht/debian`
# Install signing key
sudo pacman-key --recv-keys C0AAFC1676BD998617C94C42DC59670F1EB0A189
sudo pacman-key --lsign-key C0AAFC1676BD998617C94C42DC59670F1EB0A189
```
deb https://mirror.sr.ht/debian sid main
# Update package index
sudo pacman -Syu
```
**Signing Key**: [key.asc](https://mirror.sr.ht/debian/key.asc) `6B1296C65B24472674E7B6520585B50AC6A4914D`
# Debian
**Mirror**: https://mirror.sr.ht/debian
```
$ wget -q https://mirror.sr.ht/debian/key.asc -O - | sudo apt-key --keyring /etc/apt/trusted.gpg.d/debian.sr.ht.gpg add -
```
**Signing key**: [key.asc](https://mirror.sr.ht/debian/key.asc) `6B1296C65B24472674E7B6520585B50AC6A4914D`
**Maintainer**: Denis Laxalde <denis@laxalde.org>
**Source**: https://git.sr.ht/~dlax/sr.ht-debbuilds
```sh
# Add the following line to /etc/apt/sources.list
deb https://mirror.sr.ht/debian sid main
# Install signing key
wget -q https://mirror.sr.ht/debian/key.asc -O - | sudo apt-key --keyring /etc/apt/trusted.gpg.d/debian.sr.ht.gpg add -
# Update package index
sudo apt update
```