408 lines
18 KiB
Markdown
408 lines
18 KiB
Markdown
# Steady-state operator runbook
|
||
|
||
Day-to-day deploy, rollback, and recovery for CommunityRule on MEDLab
|
||
Cloudron. Assumes staging or production is already installed and smoke-tested.
|
||
|
||
> **First-time install, apex cutover, and legacy decommission** live in
|
||
> [`ops-backend-deploy.md`](ops-backend-deploy.md). Use this doc once an
|
||
> environment is already running.
|
||
|
||
## 1. Quick reference
|
||
|
||
| Item | Value |
|
||
| ---- | ----- |
|
||
| Cloudron dashboard | `https://my.medlab.host` |
|
||
| Cloudron CLI login | `cloudron login my.medlab.host` |
|
||
| Staging app | `staging.communityrule.info` |
|
||
| Production app | `communityrule.info` (after apex cutover) |
|
||
| Container image | `git.medlab.host/communityrule/community-rule:<tag>` |
|
||
| Health check | `GET /api/health` → `200 {"ok":true,"database":"connected"}` |
|
||
| Manifest version | [`CloudronManifest.json`](../../CloudronManifest.json) `version` field (must increase for each release) |
|
||
| Current manifest | `0.1.8` at time of writing — always read the file before deploying |
|
||
|
||
Replace `<app>` below with the Cloudron location (`staging.communityrule.info`
|
||
or `communityrule.info`).
|
||
|
||
## 2. Prerequisites (one-time per operator)
|
||
|
||
1. **Cloudron access** — admin login on `my.medlab.host` and a CLI API token
|
||
(*Profile → API Tokens* on the dashboard). Save the token in 1Password.
|
||
2. **Cloudron CLI** — logged in:
|
||
```bash
|
||
cloudron login my.medlab.host
|
||
```
|
||
3. **Docker + buildx** — Docker Desktop or equivalent with `docker buildx`.
|
||
4. **Gitea registry auth** — personal access token on `git.medlab.host` with
|
||
`read:package` + `write:package`; then:
|
||
```bash
|
||
docker login git.medlab.host
|
||
```
|
||
5. **Repo checkout** — clone
|
||
[`CommunityRule/community-rule`](https://git.medlab.host/CommunityRule/community-rule)
|
||
and work from a clean commit that matches the release you intend to ship.
|
||
|
||
Images are **`linux/amd64` only** (Cloudron host is x86_64). On Apple
|
||
Silicon, the release script still builds amd64 via buildx; a bare
|
||
`docker pull` without `--platform linux/amd64` failing on arm64 is expected.
|
||
|
||
## 3. Deploy a new version
|
||
|
||
Typical release flow: bump manifest → build/push image → `cloudron update`
|
||
→ smoke.
|
||
|
||
### 3.1 Build and push
|
||
|
||
1. Check out the commit to release (`main` or a release branch).
|
||
2. **Bump** [`CloudronManifest.json`](../../CloudronManifest.json) `version`
|
||
(e.g. `0.1.8` → `0.1.9`). Cloudron requires the manifest version to
|
||
**increase** for `cloudron update --image` to be accepted.
|
||
3. From the repo root, build and push (tag should match the manifest version
|
||
for sanity):
|
||
|
||
```bash
|
||
TAG=0.1.9 ./scripts/docker-release.sh
|
||
# equivalent:
|
||
TAG=0.1.9 npm run docker:release
|
||
```
|
||
|
||
Omit `TAG=` to push `git rev-parse --short HEAD` instead — only do that
|
||
for ad-hoc staging experiments; production releases should use semver tags
|
||
aligned with the manifest.
|
||
|
||
4. **Verify anonymous pull** (simulates Cloudron):
|
||
```bash
|
||
docker logout git.medlab.host
|
||
docker pull --platform linux/amd64 \
|
||
git.medlab.host/communityrule/community-rule:0.1.9
|
||
```
|
||
|
||
5. **Commit the manifest bump** in git alongside the code that shipped in
|
||
this build.
|
||
|
||
Registry details and one-time Gitea setup: [`ops-backend-deploy.md` §9](ops-backend-deploy.md#9-build-and-push-image-workflow).
|
||
|
||
### 3.2 Update Cloudron
|
||
|
||
```bash
|
||
cloudron update --app staging.communityrule.info \
|
||
--image git.medlab.host/communityrule/community-rule:0.1.9
|
||
```
|
||
|
||
Use `communityrule.info` for production. Cloudron pulls the image (no registry
|
||
credentials on the host), restarts the container, and runs
|
||
[`scripts/start.sh`](../../scripts/start.sh), which:
|
||
|
||
1. `chown`s `/app/data` (localstorage mount),
|
||
2. runs **`prisma migrate deploy`**,
|
||
3. writes the adopted team Rule (document and visibility together),
|
||
4. execs the Next.js standalone server.
|
||
|
||
Watch the app **Logs** tab in the Cloudron dashboard for a clean migration and
|
||
`Listening on port 3000`.
|
||
|
||
### 3.3 Migrations
|
||
|
||
**Normal case:** migrations apply automatically on container start — no
|
||
separate step.
|
||
|
||
**Manual re-run** (only if debugging a failed deploy or verifying before
|
||
traffic):
|
||
|
||
```bash
|
||
cloudron exec --app staging.communityrule.info -- npm run db:deploy
|
||
```
|
||
|
||
(`npm run db:deploy` → `prisma migrate deploy`.)
|
||
|
||
**Policy:** never run `prisma migrate reset` against staging or production.
|
||
Never edit migration files already applied to a shared database. Fix schema
|
||
drift by adding a **new** migration locally (`prisma migrate dev`) and
|
||
deploying a new image. See [`backend-roadmap.md` §8](backend-roadmap.md#8-prisma-migrations-policy).
|
||
|
||
### 3.4 Seed data (not every deploy)
|
||
|
||
Template + facet seed (`MethodFacet` rows for create-flow “Recommended” tags)
|
||
is **not** applied at boot. The adopted team Rule is the exception: container
|
||
start writes that one row so the public catalog does not keep the old draft
|
||
body. Run the template seed once per environment after first install, or
|
||
when recommendations return all-zero scores:
|
||
|
||
```bash
|
||
cloudron exec --app staging.communityrule.info -- \
|
||
node prisma/seed.bundle.cjs
|
||
```
|
||
|
||
Re-running is safe (idempotent upserts). JSON lives at `/app/seed-data/` in
|
||
the image — not under `/app/data` (Cloudron localstorage overwrites that
|
||
mount).
|
||
|
||
### 3.5 Smoke after deploy
|
||
|
||
**Automated** (from your laptop, repo root):
|
||
|
||
```bash
|
||
./scripts/staging-smoke.sh staging.communityrule.info
|
||
# production:
|
||
./scripts/staging-smoke.sh communityrule.info
|
||
|
||
# optional — exercises magic-link request (check inbox manually):
|
||
EMAIL=you@example.com ./scripts/staging-smoke.sh staging.communityrule.info
|
||
```
|
||
|
||
**Manual** (still required for full acceptance):
|
||
|
||
- Click a magic link → signed in → `GET /api/auth/session` returns a user.
|
||
- Publish a rule end-to-end → public detail page loads.
|
||
- Optional: Save & Exit draft sync; upload with `UPLOAD_ROOT` set.
|
||
|
||
Full checklist and failure table: [`ops-backend-deploy.md` §10](ops-backend-deploy.md).
|
||
|
||
## 4. Roll back (code-only)
|
||
|
||
To revert application code without touching the database:
|
||
|
||
```bash
|
||
cloudron update --app staging.communityrule.info \
|
||
--image git.medlab.host/communityrule/community-rule:<previous-tag>
|
||
```
|
||
|
||
Pick a tag you know was healthy (previous manifest version or git tag recorded
|
||
at last good deploy).
|
||
|
||
**Database implications:**
|
||
|
||
- Rolling back the **image** does **not** undo migrations already applied.
|
||
- If the bad release added a migration, rolling back to an older image may
|
||
leave the DB schema **ahead** of what that code expects — usually safe if
|
||
the migration was additive (new nullable columns, new tables).
|
||
- If the bad release broke because of a **destructive or incompatible**
|
||
migration, do **not** reset production. Restore from a Cloudron backup
|
||
(§5) or fix forward with a corrective migration.
|
||
|
||
**Never** `prisma migrate reset` on staging or production.
|
||
|
||
## 5. Restore drill (quarterly)
|
||
|
||
Verify Cloudron backups are restorable without touching the live app.
|
||
|
||
**Cadence:** at least once per quarter, or after any backup-policy change.
|
||
|
||
**Steps:**
|
||
|
||
1. In the Cloudron dashboard, pick a recent automatic backup of
|
||
`<app>` (*Backups* tab).
|
||
2. **Restore to a scratch location** — e.g.
|
||
`restore-drill-YYYYMMDD.communityrule.info` — not over the live app.
|
||
3. After restore completes, confirm the container starts and migrations are
|
||
current:
|
||
```bash
|
||
curl -sS "https://restore-drill-YYYYMMDD.communityrule.info/api/health"
|
||
```
|
||
Expect `200` with `"database":"connected"`.
|
||
4. Optional: `cloudron exec --app restore-drill-YYYYMMDD.communityrule.info -- npm run db:deploy`
|
||
if logs show pending migrations on an older snapshot.
|
||
5. Run `./scripts/staging-smoke.sh restore-drill-YYYYMMDD.communityrule.info`.
|
||
6. **Uninstall** the scratch app when done.
|
||
|
||
Record the drill date and outcome in your ops notes. Cloudron retains
|
||
automatic backups per platform defaults; confirm retention in the dashboard.
|
||
|
||
## 6. Single-instance limitations
|
||
|
||
The current Cloudron deploy runs **one container per environment**. Do not
|
||
scale to multiple app instances without addressing these per-process limits:
|
||
|
||
### 6.1 In-memory rate limiter
|
||
|
||
[`lib/server/rateLimit.ts`](../../lib/server/rateLimit.ts) stores windows in
|
||
process memory. Limits apply **per container**, not globally across instances.
|
||
|
||
| Route / action | Key | Min interval |
|
||
| -------------- | --- | ------------ |
|
||
| Magic-link request | per email | 60 s |
|
||
| Magic-link request | per IP | 20 s |
|
||
| Email change request | per email / IP / user | 60 s |
|
||
| Organizer inquiry | per email / IP | 60 s / 20 s |
|
||
| Publish with stakeholder invites | per IP | 60 s |
|
||
| Stakeholder add / resend | per IP / invite | 60 s |
|
||
| File upload | per user | 5 s |
|
||
|
||
Before horizontal scale-out, replace with a shared store (e.g. Redis) or edge
|
||
rate limits. See [`backend-roadmap.md` §5](backend-roadmap.md#5-session-and-authentication-v1).
|
||
|
||
### 6.2 Web vitals storage
|
||
|
||
Production defaults to **`external`** mode: vitals are structured log lines, not
|
||
written to Postgres or local files. Setting `WEB_VITALS_STORAGE=local` uses a
|
||
**per-process** file store under `.next/web-vitals` — suitable for dev/admin
|
||
only, not multi-instance. See [`backend-roadmap.md` §7](backend-roadmap.md#7-api-responses-errors-and-observability).
|
||
|
||
## 7. Environment variables (steady-state)
|
||
|
||
Cloudron **auto-injects** addon vars (`CLOUDRON_POSTGRESQL_URL`,
|
||
`CLOUDRON_MAIL_SMTP_*`). Operators set these manually once per app; they
|
||
persist across image updates unless changed:
|
||
|
||
| Variable | Purpose |
|
||
| -------- | ------- |
|
||
| `SESSION_SECRET` | Session cookie signing (≥ 16 chars). Rotating logs everyone out. |
|
||
| `SMTP_FROM` | Visible From on sign-in emails (e.g. `CommunityRule <hello@communityrule.info>`). The app rewrites a leftover `Community Rule` display name to one word. |
|
||
| `ENABLE_PRIVACY_POLICY` | `true` on staging so `/privacy` is reachable. Unset in production until counsel has read the policy (404, no public links). |
|
||
| `NEXT_PUBLIC_ENABLE_BACKEND_SYNC` | `true` in staging/production — Postgres draft persistence. |
|
||
| `UPLOAD_ROOT` | `/app/data/uploads` on Cloudron — required for file uploads. |
|
||
| `OPS_PRUNE_TOKEN` | Bearer token for `POST /api/ops/prune`. Unset hides the route. Optional; the daily scheduler runs the prune inside the container without it. |
|
||
| `WORKSHOP_INQUIRY_PURGE_TOKEN` | Bearer token for `POST /api/workshop-inquiries/purge`. Unset hides that route. |
|
||
| `RULE_ADMIN_EMAILS` | Named admins. Comma-separated. Can open private Rules and `/rule-counts`. |
|
||
| `NEXT_PUBLIC_UMAMI_SRC` | Umami script URL. Unset: no tracker. Pair with `NEXT_PUBLIC_UMAMI_WEBSITE_ID`. |
|
||
| `NEXT_PUBLIC_UMAMI_WEBSITE_ID` | Umami website id for this hostname. |
|
||
|
||
Full detail: [`ops-backend-deploy.md` §3](ops-backend-deploy.md#3-environment-variables).
|
||
|
||
## 8. Troubleshooting
|
||
|
||
| Symptom | Likely cause | Action |
|
||
| ------- | ------------ | ------ |
|
||
| Image pull error on update | Private repo, wrong tag, or amd64 manifest missing | Confirm repo is public; verify pull with `--platform linux/amd64` (§3.1) |
|
||
| Health `503` / `database: disconnected` | Postgres addon or `CLOUDRON_POSTGRESQL_URL` missing | Cloudron app → Environment |
|
||
| Container crash on start | Migration failure | App logs around `prisma migrate deploy`; fix forward with new migration |
|
||
| Magic link not sent | Mail addon, `SMTP_FROM`, or SES DNS | Cloudron mail logs (`CLOUDRON_MAIL_SMTP_*`); inbox/spam; SPF/DKIM for SES (§8.1) |
|
||
| Upload `server_misconfigured` | `UPLOAD_ROOT` unset | `cloudron env set --app <app> UPLOAD_ROOT=/app/data/uploads` |
|
||
| No “Recommended” on method cards | Seed not run | §3.4 — `node prisma/seed.bundle.cjs` |
|
||
| Rate limit too aggressive after deploy | Expected per §6.1 | Single instance only; limits reset on container restart |
|
||
|
||
App logs: Cloudron dashboard → *Logs* tab, or `cloudron logs --app <app> -f`.
|
||
|
||
### 8.1 Mail DNS when SES is the relay
|
||
|
||
`communityrule.info` outbound mail is **Amazon SES SMTP** (`email-smtp.us-east-2.amazonaws.com:587`), not Cloudron's own MTA. Cloudron Mail → domain status therefore **skips SPF and DKIM** ("configure the relay provider") and only checks MX, DMARC (`v=DMARC1; p=reject; pct=100`), and that the SES connection works.
|
||
|
||
That is expected. Recipients still authenticate the visible `From:` (`staging.app@communityrule.info` on staging) against **SES DKIM/SPF**, not `a:my.medlab.host`.
|
||
|
||
**Operator steps (AWS + Cloudron DNS, not app code):**
|
||
|
||
1. In **AWS SES** (us-east-2), open the verified identity for `communityrule.info` (create one if missing). Copy the **DKIM CNAME** records SES shows (three `*._domainkey.communityrule.info` names).
|
||
2. In **Cloudron** → *Domains* → `communityrule.info` → DNS, add those CNAMEs. Cloudron's Namecheap provider publishes them to the registrar. Confirm with `dig +short CNAME <selector>._domainkey.communityrule.info`.
|
||
3. Optional but recommended for SPF alignment: add `include:amazonses.com` to the existing TXT SPF, e.g. `v=spf1 include:amazonses.com a:my.medlab.host ~all`. Do not remove `a:my.medlab.host` until you know nothing still sends directly from the box.
|
||
4. Leave DMARC at `p=reject` once DKIM verifies in SES; if a provider still quarantines after DKIM is live, inspect that provider's headers before relaxing DMARC.
|
||
5. Retest: request a magic link to Gmail **and** a non-Gmail inbox (May First / university). Check spam. Staging From is `CommunityRule <staging.app@communityrule.info>`.
|
||
|
||
`SMTP_FROM` should stay the Cloudron mailbox (`staging.app@communityrule.info` on staging, `hello@communityrule.info` on the apex app). The app falls back to `CLOUDRON_MAIL_FROM` if `SMTP_FROM` is unset.
|
||
|
||
## 9. Retention (logs, backups, unused links)
|
||
|
||
Periods the privacy policy states. Source of truth:
|
||
[`docs/privacy-policy-facts.md`](../privacy-policy-facts.md) and
|
||
[`lib/privacyRetention.ts`](../../lib/privacyRetention.ts).
|
||
|
||
### 9.1 Unused links and sessions
|
||
|
||
[`CloudronManifest.json`](../../CloudronManifest.json) registers a
|
||
**scheduler** task `prune-expired` at **07:20 UTC** daily. It runs
|
||
`node /app/prune-expired.bundle.cjs` (the production bundle of
|
||
[`scripts/prune-sessions.ts`](../../scripts/prune-sessions.ts)) and
|
||
deletes expired sessions, unused login / save-progress / email-change
|
||
tokens and the drafts they carry, and workshop inquiries older than 12
|
||
months. It leaves invitation token hashes in place: an expired link cannot
|
||
be used to join, and that hash is how the person requests a new link.
|
||
|
||
Manual runs:
|
||
|
||
```bash
|
||
# inside the container (no token)
|
||
cloudron exec --app staging.communityrule.info -- \
|
||
node /app/prune-expired.bundle.cjs
|
||
|
||
# from outside, if OPS_PRUNE_TOKEN is set
|
||
curl -sS -X POST \
|
||
-H "Authorization: Bearer $OPS_PRUNE_TOKEN" \
|
||
https://staging.communityrule.info/api/ops/prune
|
||
```
|
||
|
||
Local:
|
||
|
||
```bash
|
||
node --env-file=.env --import tsx scripts/prune-sessions.ts
|
||
```
|
||
|
||
If the scheduler addon is missing after an image update, add the same
|
||
command under the app **Cron** tab (`20 7 * * *`, UTC).
|
||
|
||
The same scheduler addon also runs **`follow-up-mail`** at **15 minutes
|
||
past every hour**. It sends one print follow-up to the address that holds
|
||
the pen, 48 hours after the Rule was finished, with PDF and RULE.md
|
||
links. Guest Rules with no account are skipped until claimed. Rules
|
||
finished more than seven days before this mail existed are closed in the
|
||
migration and are not sent. A Rule that is already due stays eligible
|
||
until the job claims it, including after a scheduler outage longer than
|
||
seven days. The claim is written before the message is sent, and cleared
|
||
again if sending fails, so overlapping runs cannot deliver it twice.
|
||
|
||
Manual run:
|
||
|
||
```bash
|
||
cloudron exec --app staging.communityrule.info -- \
|
||
node /app/follow-up-mail.bundle.cjs
|
||
|
||
# local
|
||
node --env-file=.env --import tsx scripts/send-rule-follow-up.ts
|
||
```
|
||
|
||
Named admins can read finished-Rule counts (folded under five) at
|
||
`/rule-counts`, or `GET /api/ops/rule-counts` with the same session or
|
||
`Authorization: Bearer $OPS_PRUNE_TOKEN`.
|
||
|
||
### 9.2 Logs
|
||
|
||
App logs (stdout) and Cloudron request logs are deleted after **14
|
||
days**. That is the platform default and the shortest period Cloudron
|
||
allows.
|
||
|
||
Page timing logs (`POST /api/web-vitals` in `external` mode) record
|
||
origin + path only. Query strings are stripped before the line is
|
||
written, so invitation and login tokens never land in app logs.
|
||
|
||
Cloudron's reverse-proxy request logs still include the query string.
|
||
They follow the same 14-day window. Do not export those logs off the
|
||
host for longer than 14 days.
|
||
|
||
Sign-in mail failures log a request id, not the email address.
|
||
|
||
### 9.3 Umami (site counts)
|
||
|
||
Umami is a **second Cloudron app**, not this image. Install the Umami
|
||
Cloudron package (or Docker) on a hostname such as
|
||
`analytics.communityrule.info`. Create a website for this app’s public
|
||
origin. In Umami → website settings, **turn off cookies** (tracker
|
||
collects aggregate page views only).
|
||
|
||
Then on this CommunityRule app:
|
||
|
||
```bash
|
||
cloudron env set --app staging.communityrule.info \
|
||
NEXT_PUBLIC_UMAMI_SRC=https://analytics.communityrule.info/script.js
|
||
cloudron env set --app staging.communityrule.info \
|
||
NEXT_PUBLIC_UMAMI_WEBSITE_ID=<website-id>
|
||
```
|
||
|
||
Unset both to ship no tracker. The snippet sets `data-do-not-track` and
|
||
`data-exclude-search` so query strings (share and export tokens) are not
|
||
sent to Umami.
|
||
|
||
### 9.4 Backups
|
||
|
||
Set the Cloudron backup site to **`30 daily`** (*Backups* → *Schedule &
|
||
Retention*). Confirm this on staging and production. Cloudron also keeps
|
||
a pre-update snapshot for up to three weeks.
|
||
|
||
## 10. Related docs
|
||
|
||
- [`../privacy-policy-facts.md`](../privacy-policy-facts.md) — periods
|
||
and hosting facts for the privacy policy.
|
||
- [`ops-backend-deploy.md`](ops-backend-deploy.md) — first install, cutover
|
||
plan, legacy rules archive, build/push deep dive.
|
||
- [`backend-roadmap.md`](backend-roadmap.md) — migrations policy (§8),
|
||
rate limiting (§5), environments (§11).
|
||
- [`../relaunch-brief.md`](../relaunch-brief.md) — plain-language summary
|
||
for MEDLab admin.
|
||
- [`../../CONTRIBUTING.md`](../../CONTRIBUTING.md) — local dev setup.
|