builds.sr.ht: add internal deploy docs

builds.sr.ht/deploy.md

@@ -0,0 +1,49 @@
+---
+title: Internal deployment documentation
+---
+
+This is internal documentation for deploying builds.sr.ht updates to the
+SourceHut hosted services.
+
+## Updating images
+
+When a patch for an image comes in, ensure that it meets all of the criteria set
+out in [image maintenance][0] before applying. Every step is essential to ensure
+that there is no downtime for any image while upgrading -- review the patch
+carefully.
+
+[0]: https://man.sr.ht/builds.sr.ht/image-maintenance.md
+
+The deployment process for new images is the following:
+
+1. Apply and push the changes to the upstream repository
+1. Manually submit builds to build and deploy the new images
+1. Tag a new release and push it to the Alpine repos
+1. Log into the build workers and run `apk upgrade -U`. At this point the new
+   image scripts are available and builds can be run against the new images.
+1. Run sanity test builds against the new images to verify them
+1. Apply patches for compatibility.md in sr.ht-docs
+
+When deprecating images, identify users who have submitted builds using the
+affected image in the past 90 days with a SQL query similar to the following:
+
+```
+SELECT
+  DISTINCT "user".email,
+  job.created
+FROM job
+JOIN "user" ON "user".id = job.owner_id
+WHERE image = 'ubuntu/kinetic' AND job.created >= '2023-06-01'
+ORDER BY job.created DESC;
+```
+
+Send an email to each affected user with two weeks notice before deploying an
+image deprecation. The compatibility.md matrix should be updated right away to
+remove the affected image, to avoid new users appearing during the notice
+period.
+
+## crontab & submit_image_build
+
+These are not updated automatically during deployments, when changed you need to
+deploy them manually. submit\_image\_build is stored in /usr/local/bin and the
+crontab runs on sircmpwn's account.

builds.sr.ht/index.md

@@ -29,6 +29,7 @@ on.
 - [Build image support matrix](compatibility.md)
 - [SSH access to build VMs](build-ssh.md)
 - [Information for build image maintainers](image-maintenance.md)
+- [Internal deployment reference](deploy.md)
 
 # How jobs are submitted