Files
community-rule/docs/guides/ops-runbook.md
T

407 lines
17 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Steady-state operator runbook
Day-to-day deploy, rollback, and recovery for CommunityRule on MEDLab
Cloudron. Assumes staging or production is already installed and smoke-tested.
> **First-time install, apex cutover, and legacy decommission** live in
> [`ops-backend-deploy.md`](ops-backend-deploy.md). Use this doc once an
> environment is already running.
## 1. Quick reference
| Item | Value |
| ---- | ----- |
| Cloudron dashboard | `https://my.medlab.host` |
| Cloudron CLI login | `cloudron login my.medlab.host` |
| Staging app | `staging.communityrule.info` |
| Production app | `communityrule.info` (after apex cutover) |
| Container image | `git.medlab.host/communityrule/community-rule:<tag>` |
| Health check | `GET /api/health` → `200 {"ok":true,"database":"connected"}` |
| Manifest version | [`CloudronManifest.json`](../../CloudronManifest.json) `version` field (must increase for each release) |
| Current manifest | `0.1.8` at time of writing — always read the file before deploying |
Replace `<app>` below with the Cloudron location (`staging.communityrule.info`
or `communityrule.info`).
## 2. Prerequisites (one-time per operator)
1. **Cloudron access** — admin login on `my.medlab.host` and a CLI API token
(*Profile → API Tokens* on the dashboard). Save the token in 1Password.
2. **Cloudron CLI** — logged in:
```bash
cloudron login my.medlab.host
```
3. **Docker + buildx** — Docker Desktop or equivalent with `docker buildx`.
4. **Gitea registry auth** — personal access token on `git.medlab.host` with
`read:package` + `write:package`; then:
```bash
docker login git.medlab.host
```
5. **Repo checkout** — clone
[`CommunityRule/community-rule`](https://git.medlab.host/CommunityRule/community-rule)
and work from a clean commit that matches the release you intend to ship.
Images are **`linux/amd64` only** (Cloudron host is x86_64). On Apple
Silicon, the release script still builds amd64 via buildx; a bare
`docker pull` without `--platform linux/amd64` failing on arm64 is expected.
## 3. Deploy a new version
Typical release flow: bump manifest → build/push image → `cloudron update`
→ smoke.
### 3.1 Build and push
1. Check out the commit to release (`main` or a release branch).
2. **Bump** [`CloudronManifest.json`](../../CloudronManifest.json) `version`
(e.g. `0.1.8` → `0.1.9`). Cloudron requires the manifest version to
**increase** for `cloudron update --image` to be accepted.
3. From the repo root, build and push (tag should match the manifest version
for sanity):
```bash
TAG=0.1.9 ./scripts/docker-release.sh
# equivalent:
TAG=0.1.9 npm run docker:release
```
Omit `TAG=` to push `git rev-parse --short HEAD` instead — only do that
for ad-hoc staging experiments; production releases should use semver tags
aligned with the manifest.
4. **Verify anonymous pull** (simulates Cloudron):
```bash
docker logout git.medlab.host
docker pull --platform linux/amd64 \
git.medlab.host/communityrule/community-rule:0.1.9
```
5. **Commit the manifest bump** in git alongside the code that shipped in
this build.
Registry details and one-time Gitea setup: [`ops-backend-deploy.md` §9](ops-backend-deploy.md#9-build-and-push-image-workflow).
### 3.2 Update Cloudron
```bash
cloudron update --app staging.communityrule.info \
--image git.medlab.host/communityrule/community-rule:0.1.9
```
Use `communityrule.info` for production. Cloudron pulls the image (no registry
credentials on the host), restarts the container, and runs
[`scripts/start.sh`](../../scripts/start.sh), which:
1. `chown`s `/app/data` (localstorage mount),
2. runs **`prisma migrate deploy`**,
3. writes the adopted team Rule (document and visibility together),
4. execs the Next.js standalone server.
Watch the app **Logs** tab in the Cloudron dashboard for a clean migration and
`Listening on port 3000`.
### 3.3 Migrations
**Normal case:** migrations apply automatically on container start — no
separate step.
**Manual re-run** (only if debugging a failed deploy or verifying before
traffic):
```bash
cloudron exec --app staging.communityrule.info -- npm run db:deploy
```
(`npm run db:deploy` → `prisma migrate deploy`.)
**Policy:** never run `prisma migrate reset` against staging or production.
Never edit migration files already applied to a shared database. Fix schema
drift by adding a **new** migration locally (`prisma migrate dev`) and
deploying a new image. See [`backend-roadmap.md` §8](backend-roadmap.md#8-prisma-migrations-policy).
### 3.4 Seed data (not every deploy)
Template + facet seed (`MethodFacet` rows for create-flow “Recommended” tags)
is **not** applied at boot. The adopted team Rule is the exception: container
start writes that one row so the public catalog does not keep the old draft
body. Run the template seed once per environment after first install, or
when recommendations return all-zero scores:
```bash
cloudron exec --app staging.communityrule.info -- \
node prisma/seed.bundle.cjs
```
Re-running is safe (idempotent upserts). JSON lives at `/app/seed-data/` in
the image — not under `/app/data` (Cloudron localstorage overwrites that
mount).
### 3.5 Smoke after deploy
**Automated** (from your laptop, repo root):
```bash
./scripts/staging-smoke.sh staging.communityrule.info
# production:
./scripts/staging-smoke.sh communityrule.info
# optional — exercises magic-link request (check inbox manually):
EMAIL=you@example.com ./scripts/staging-smoke.sh staging.communityrule.info
```
**Manual** (still required for full acceptance):
- Click a magic link → signed in → `GET /api/auth/session` returns a user.
- Publish a rule end-to-end → public detail page loads.
- Optional: Save & Exit draft sync; upload with `UPLOAD_ROOT` set.
Full checklist and failure table: [`ops-backend-deploy.md` §10](ops-backend-deploy.md).
## 4. Roll back (code-only)
To revert application code without touching the database:
```bash
cloudron update --app staging.communityrule.info \
--image git.medlab.host/communityrule/community-rule:<previous-tag>
```
Pick a tag you know was healthy (previous manifest version or git tag recorded
at last good deploy).
**Database implications:**
- Rolling back the **image** does **not** undo migrations already applied.
- If the bad release added a migration, rolling back to an older image may
leave the DB schema **ahead** of what that code expects — usually safe if
the migration was additive (new nullable columns, new tables).
- If the bad release broke because of a **destructive or incompatible**
migration, do **not** reset production. Restore from a Cloudron backup
(§5) or fix forward with a corrective migration.
**Never** `prisma migrate reset` on staging or production.
## 5. Restore drill (quarterly)
Verify Cloudron backups are restorable without touching the live app.
**Cadence:** at least once per quarter, or after any backup-policy change.
**Steps:**
1. In the Cloudron dashboard, pick a recent automatic backup of
`<app>` (*Backups* tab).
2. **Restore to a scratch location** — e.g.
`restore-drill-YYYYMMDD.communityrule.info` — not over the live app.
3. After restore completes, confirm the container starts and migrations are
current:
```bash
curl -sS "https://restore-drill-YYYYMMDD.communityrule.info/api/health"
```
Expect `200` with `"database":"connected"`.
4. Optional: `cloudron exec --app restore-drill-YYYYMMDD.communityrule.info -- npm run db:deploy`
if logs show pending migrations on an older snapshot.
5. Run `./scripts/staging-smoke.sh restore-drill-YYYYMMDD.communityrule.info`.
6. **Uninstall** the scratch app when done.
Record the drill date and outcome in your ops notes. Cloudron retains
automatic backups per platform defaults; confirm retention in the dashboard.
## 6. Single-instance limitations
The current Cloudron deploy runs **one container per environment**. Do not
scale to multiple app instances without addressing these per-process limits:
### 6.1 In-memory rate limiter
[`lib/server/rateLimit.ts`](../../lib/server/rateLimit.ts) stores windows in
process memory. Limits apply **per container**, not globally across instances.
| Route / action | Key | Min interval |
| -------------- | --- | ------------ |
| Magic-link request | per email | 60 s |
| Magic-link request | per IP | 20 s |
| Email change request | per email / IP / user | 60 s |
| Organizer inquiry | per email / IP | 60 s / 20 s |
| Publish with stakeholder invites | per IP | 60 s |
| Stakeholder add / resend | per IP / invite | 60 s |
| File upload | per user | 5 s |
Before horizontal scale-out, replace with a shared store (e.g. Redis) or edge
rate limits. See [`backend-roadmap.md` §5](backend-roadmap.md#5-session-and-authentication-v1).
### 6.2 Web vitals storage
Production defaults to **`external`** mode: vitals are structured log lines, not
written to Postgres or local files. Setting `WEB_VITALS_STORAGE=local` uses a
**per-process** file store under `.next/web-vitals` — suitable for dev/admin
only, not multi-instance. See [`backend-roadmap.md` §7](backend-roadmap.md#7-api-responses-errors-and-observability).
## 7. Environment variables (steady-state)
Cloudron **auto-injects** addon vars (`CLOUDRON_POSTGRESQL_URL`,
`CLOUDRON_MAIL_SMTP_*`). Operators set these manually once per app; they
persist across image updates unless changed:
| Variable | Purpose |
| -------- | ------- |
| `SESSION_SECRET` | Session cookie signing (≥ 16 chars). Rotating logs everyone out. |
| `SMTP_FROM` | Visible From on sign-in emails (e.g. `Community Rule <hello@communityrule.info>`). |
| `NEXT_PUBLIC_ENABLE_BACKEND_SYNC` | `true` in staging/production — Postgres draft persistence. |
| `UPLOAD_ROOT` | `/app/data/uploads` on Cloudron — required for file uploads. |
| `OPS_PRUNE_TOKEN` | Bearer token for `POST /api/ops/prune`. Unset hides the route. Optional; the daily scheduler runs the prune inside the container without it. |
| `WORKSHOP_INQUIRY_PURGE_TOKEN` | Bearer token for `POST /api/workshop-inquiries/purge`. Unset hides that route. |
| `RULE_ADMIN_EMAILS` | Named admins. Comma-separated. Can open private Rules and `/rule-counts`. |
| `NEXT_PUBLIC_UMAMI_SRC` | Umami script URL. Unset: no tracker. Pair with `NEXT_PUBLIC_UMAMI_WEBSITE_ID`. |
| `NEXT_PUBLIC_UMAMI_WEBSITE_ID` | Umami website id for this hostname. |
Full detail: [`ops-backend-deploy.md` §3](ops-backend-deploy.md#3-environment-variables).
## 8. Troubleshooting
| Symptom | Likely cause | Action |
| ------- | ------------ | ------ |
| Image pull error on update | Private repo, wrong tag, or amd64 manifest missing | Confirm repo is public; verify pull with `--platform linux/amd64` (§3.1) |
| Health `503` / `database: disconnected` | Postgres addon or `CLOUDRON_POSTGRESQL_URL` missing | Cloudron app → Environment |
| Container crash on start | Migration failure | App logs around `prisma migrate deploy`; fix forward with new migration |
| Magic link not sent | Mail addon, `SMTP_FROM`, or SES DNS | Cloudron mail logs (`CLOUDRON_MAIL_SMTP_*`); inbox/spam; SPF/DKIM for SES (§8.1) |
| Upload `server_misconfigured` | `UPLOAD_ROOT` unset | `cloudron env set --app <app> UPLOAD_ROOT=/app/data/uploads` |
| No “Recommended” on method cards | Seed not run | §3.4 — `node prisma/seed.bundle.cjs` |
| Rate limit too aggressive after deploy | Expected per §6.1 | Single instance only; limits reset on container restart |
App logs: Cloudron dashboard → *Logs* tab, or `cloudron logs --app <app> -f`.
### 8.1 Mail DNS when SES is the relay
`communityrule.info` outbound mail is **Amazon SES SMTP** (`email-smtp.us-east-2.amazonaws.com:587`), not Cloudron's own MTA. Cloudron Mail → domain status therefore **skips SPF and DKIM** ("configure the relay provider") and only checks MX, DMARC (`v=DMARC1; p=reject; pct=100`), and that the SES connection works.
That is expected. Recipients still authenticate the visible `From:` (`staging.app@communityrule.info` on staging) against **SES DKIM/SPF**, not `a:my.medlab.host`.
**Operator steps (AWS + Cloudron DNS, not app code):**
1. In **AWS SES** (us-east-2), open the verified identity for `communityrule.info` (create one if missing). Copy the **DKIM CNAME** records SES shows (three `*._domainkey.communityrule.info` names).
2. In **Cloudron** → *Domains* → `communityrule.info` → DNS, add those CNAMEs. Cloudron's Namecheap provider publishes them to the registrar. Confirm with `dig +short CNAME <selector>._domainkey.communityrule.info`.
3. Optional but recommended for SPF alignment: add `include:amazonses.com` to the existing TXT SPF, e.g. `v=spf1 include:amazonses.com a:my.medlab.host ~all`. Do not remove `a:my.medlab.host` until you know nothing still sends directly from the box.
4. Leave DMARC at `p=reject` once DKIM verifies in SES; if a provider still quarantines after DKIM is live, inspect that provider's headers before relaxing DMARC.
5. Retest: request a magic link to Gmail **and** a non-Gmail inbox (May First / university). Check spam. Staging From is `Community Rule <staging.app@communityrule.info>`.
`SMTP_FROM` should stay the Cloudron mailbox (`staging.app@communityrule.info` on staging, `hello@communityrule.info` on the apex app). The app falls back to `CLOUDRON_MAIL_FROM` if `SMTP_FROM` is unset.
## 9. Retention (logs, backups, unused links)
Periods the privacy policy states. Source of truth:
[`docs/privacy-policy-facts.md`](../privacy-policy-facts.md) and
[`lib/privacyRetention.ts`](../../lib/privacyRetention.ts).
### 9.1 Unused links and sessions
[`CloudronManifest.json`](../../CloudronManifest.json) registers a
**scheduler** task `prune-expired` at **07:20 UTC** daily. It runs
`node /app/prune-expired.bundle.cjs` (the production bundle of
[`scripts/prune-sessions.ts`](../../scripts/prune-sessions.ts)) and
deletes expired sessions, unused login / save-progress / email-change
tokens and the drafts they carry, and workshop inquiries older than 12
months. It leaves invitation token hashes in place: an expired link cannot
be used to join, and that hash is how the person requests a new link.
Manual runs:
```bash
# inside the container (no token)
cloudron exec --app staging.communityrule.info -- \
node /app/prune-expired.bundle.cjs
# from outside, if OPS_PRUNE_TOKEN is set
curl -sS -X POST \
-H "Authorization: Bearer $OPS_PRUNE_TOKEN" \
https://staging.communityrule.info/api/ops/prune
```
Local:
```bash
node --env-file=.env --import tsx scripts/prune-sessions.ts
```
If the scheduler addon is missing after an image update, add the same
command under the app **Cron** tab (`20 7 * * *`, UTC).
The same scheduler addon also runs **`follow-up-mail`** at **15 minutes
past every hour**. It sends one print follow-up to the address that holds
the pen, 48 hours after the Rule was finished, with PDF and RULE.md
links. Guest Rules with no account are skipped until claimed. Rules
finished more than seven days before this mail existed are closed in the
migration and are not sent. A Rule that is already due stays eligible
until the job claims it, including after a scheduler outage longer than
seven days. The claim is written before the message is sent, and cleared
again if sending fails, so overlapping runs cannot deliver it twice.
Manual run:
```bash
cloudron exec --app staging.communityrule.info -- \
node /app/follow-up-mail.bundle.cjs
# local
node --env-file=.env --import tsx scripts/send-rule-follow-up.ts
```
Named admins can read finished-Rule counts (folded under five) at
`/rule-counts`, or `GET /api/ops/rule-counts` with the same session or
`Authorization: Bearer $OPS_PRUNE_TOKEN`.
### 9.2 Logs
App logs (stdout) and Cloudron request logs are deleted after **14
days**. That is the platform default and the shortest period Cloudron
allows.
Page timing logs (`POST /api/web-vitals` in `external` mode) record
origin + path only. Query strings are stripped before the line is
written, so invitation and login tokens never land in app logs.
Cloudron's reverse-proxy request logs still include the query string.
They follow the same 14-day window. Do not export those logs off the
host for longer than 14 days.
Sign-in mail failures log a request id, not the email address.
### 9.3 Umami (site counts)
Umami is a **second Cloudron app**, not this image. Install the Umami
Cloudron package (or Docker) on a hostname such as
`analytics.communityrule.info`. Create a website for this app’s public
origin. In Umami → website settings, **turn off cookies** (tracker
collects aggregate page views only).
Then on this CommunityRule app:
```bash
cloudron env set --app staging.communityrule.info \
NEXT_PUBLIC_UMAMI_SRC=https://analytics.communityrule.info/script.js
cloudron env set --app staging.communityrule.info \
NEXT_PUBLIC_UMAMI_WEBSITE_ID=<website-id>
```
Unset both to ship no tracker. The snippet sets `data-do-not-track` and
`data-exclude-search` so query strings (share and export tokens) are not
sent to Umami.
### 9.4 Backups
Set the Cloudron backup site to **`30 daily`** (*Backups* → *Schedule &
Retention*). Confirm this on staging and production. Cloudron also keeps
a pre-update snapshot for up to three weeks.
## 10. Related docs
- [`../privacy-policy-facts.md`](../privacy-policy-facts.md) — periods
and hosting facts for the privacy policy.
- [`ops-backend-deploy.md`](ops-backend-deploy.md) — first install, cutover
plan, legacy rules archive, build/push deep dive.
- [`backend-roadmap.md`](backend-roadmap.md) — migrations policy (§8),
rate limiting (§5), environments (§11).
- [`../relaunch-brief.md`](../relaunch-brief.md) — plain-language summary
for MEDLab admin.
- [`../../CONTRIBUTING.md`](../../CONTRIBUTING.md) — local dev setup.