# Steady-state operator runbook Day-to-day deploy, rollback, and recovery for CommunityRule on MEDLab Cloudron. Assumes staging or production is already installed and smoke-tested. > **First-time install, apex cutover, and legacy decommission** live in > [`ops-backend-deploy.md`](ops-backend-deploy.md). Use this doc once an > environment is already running. ## 1. Quick reference | Item | Value | | ---- | ----- | | Cloudron dashboard | `https://my.medlab.host` | | Cloudron CLI login | `cloudron login my.medlab.host` | | Staging app | `staging.communityrule.info` | | Production app | `communityrule.info` (after apex cutover) | | Container image | `git.medlab.host/communityrule/community-rule:` | | Health check | `GET /api/health` → `200 {"ok":true,"database":"connected"}` | | Manifest version | [`CloudronManifest.json`](../../CloudronManifest.json) `version` field (must increase for each release) | | Current manifest | `0.1.8` at time of writing — always read the file before deploying | Replace `` below with the Cloudron location (`staging.communityrule.info` or `communityrule.info`). ## 2. Prerequisites (one-time per operator) 1. **Cloudron access** — admin login on `my.medlab.host` and a CLI API token (*Profile → API Tokens* on the dashboard). Save the token in 1Password. 2. **Cloudron CLI** — logged in: ```bash cloudron login my.medlab.host ``` 3. **Docker + buildx** — Docker Desktop or equivalent with `docker buildx`. 4. **Gitea registry auth** — personal access token on `git.medlab.host` with `read:package` + `write:package`; then: ```bash docker login git.medlab.host ``` 5. **Repo checkout** — clone [`CommunityRule/community-rule`](https://git.medlab.host/CommunityRule/community-rule) and work from a clean commit that matches the release you intend to ship. Images are **`linux/amd64` only** (Cloudron host is x86_64). On Apple Silicon, the release script still builds amd64 via buildx; a bare `docker pull` without `--platform linux/amd64` failing on arm64 is expected. ## 3. Deploy a new version Typical release flow: bump manifest → build/push image → `cloudron update` → smoke. ### 3.1 Build and push 1. Check out the commit to release (`main` or a release branch). 2. **Bump** [`CloudronManifest.json`](../../CloudronManifest.json) `version` (e.g. `0.1.8` → `0.1.9`). Cloudron requires the manifest version to **increase** for `cloudron update --image` to be accepted. 3. From the repo root, build and push (tag should match the manifest version for sanity): ```bash TAG=0.1.9 ./scripts/docker-release.sh # equivalent: TAG=0.1.9 npm run docker:release ``` Omit `TAG=` to push `git rev-parse --short HEAD` instead — only do that for ad-hoc staging experiments; production releases should use semver tags aligned with the manifest. 4. **Verify anonymous pull** (simulates Cloudron): ```bash docker logout git.medlab.host docker pull --platform linux/amd64 \ git.medlab.host/communityrule/community-rule:0.1.9 ``` 5. **Commit the manifest bump** in git alongside the code that shipped in this build. Registry details and one-time Gitea setup: [`ops-backend-deploy.md` §9](ops-backend-deploy.md#9-build-and-push-image-workflow). ### 3.2 Update Cloudron ```bash cloudron update --app staging.communityrule.info \ --image git.medlab.host/communityrule/community-rule:0.1.9 ``` Use `communityrule.info` for production. Cloudron pulls the image (no registry credentials on the host), restarts the container, and runs [`scripts/start.sh`](../../scripts/start.sh), which: 1. `chown`s `/app/data` (localstorage mount), 2. runs **`prisma migrate deploy`**, 3. writes the adopted team Rule (document and visibility together), 4. execs the Next.js standalone server. Watch the app **Logs** tab in the Cloudron dashboard for a clean migration and `Listening on port 3000`. ### 3.3 Migrations **Normal case:** migrations apply automatically on container start — no separate step. **Manual re-run** (only if debugging a failed deploy or verifying before traffic): ```bash cloudron exec --app staging.communityrule.info -- npm run db:deploy ``` (`npm run db:deploy` → `prisma migrate deploy`.) **Policy:** never run `prisma migrate reset` against staging or production. Never edit migration files already applied to a shared database. Fix schema drift by adding a **new** migration locally (`prisma migrate dev`) and deploying a new image. See [`backend-roadmap.md` §8](backend-roadmap.md#8-prisma-migrations-policy). ### 3.4 Seed data (not every deploy) Template + facet seed (`MethodFacet` rows for create-flow “Recommended” tags) is **not** applied at boot. The adopted team Rule is the exception: container start writes that one row so the public catalog does not keep the old draft body. Run the template seed once per environment after first install, or when recommendations return all-zero scores: ```bash cloudron exec --app staging.communityrule.info -- \ node prisma/seed.bundle.cjs ``` Re-running is safe (idempotent upserts). JSON lives at `/app/seed-data/` in the image — not under `/app/data` (Cloudron localstorage overwrites that mount). ### 3.5 Smoke after deploy **Automated** (from your laptop, repo root): ```bash ./scripts/staging-smoke.sh staging.communityrule.info # production: ./scripts/staging-smoke.sh communityrule.info # optional — exercises magic-link request (check inbox manually): EMAIL=you@example.com ./scripts/staging-smoke.sh staging.communityrule.info ``` **Manual** (still required for full acceptance): - Click a magic link → signed in → `GET /api/auth/session` returns a user. - Publish a rule end-to-end → public detail page loads. - Optional: Save & Exit draft sync; upload with `UPLOAD_ROOT` set. Full checklist and failure table: [`ops-backend-deploy.md` §10](ops-backend-deploy.md). ## 4. Roll back (code-only) To revert application code without touching the database: ```bash cloudron update --app staging.communityrule.info \ --image git.medlab.host/communityrule/community-rule: ``` Pick a tag you know was healthy (previous manifest version or git tag recorded at last good deploy). **Database implications:** - Rolling back the **image** does **not** undo migrations already applied. - If the bad release added a migration, rolling back to an older image may leave the DB schema **ahead** of what that code expects — usually safe if the migration was additive (new nullable columns, new tables). - If the bad release broke because of a **destructive or incompatible** migration, do **not** reset production. Restore from a Cloudron backup (§5) or fix forward with a corrective migration. **Never** `prisma migrate reset` on staging or production. ## 5. Restore drill (quarterly) Verify Cloudron backups are restorable without touching the live app. **Cadence:** at least once per quarter, or after any backup-policy change. **Steps:** 1. In the Cloudron dashboard, pick a recent automatic backup of `` (*Backups* tab). 2. **Restore to a scratch location** — e.g. `restore-drill-YYYYMMDD.communityrule.info` — not over the live app. 3. After restore completes, confirm the container starts and migrations are current: ```bash curl -sS "https://restore-drill-YYYYMMDD.communityrule.info/api/health" ``` Expect `200` with `"database":"connected"`. 4. Optional: `cloudron exec --app restore-drill-YYYYMMDD.communityrule.info -- npm run db:deploy` if logs show pending migrations on an older snapshot. 5. Run `./scripts/staging-smoke.sh restore-drill-YYYYMMDD.communityrule.info`. 6. **Uninstall** the scratch app when done. Record the drill date and outcome in your ops notes. Cloudron retains automatic backups per platform defaults; confirm retention in the dashboard. ## 6. Single-instance limitations The current Cloudron deploy runs **one container per environment**. Do not scale to multiple app instances without addressing these per-process limits: ### 6.1 In-memory rate limiter [`lib/server/rateLimit.ts`](../../lib/server/rateLimit.ts) stores windows in process memory. Limits apply **per container**, not globally across instances. | Route / action | Key | Min interval | | -------------- | --- | ------------ | | Magic-link request | per email | 60 s | | Magic-link request | per IP | 20 s | | Email change request | per email / IP / user | 60 s | | Organizer inquiry | per email / IP | 60 s / 20 s | | Publish with stakeholder invites | per IP | 60 s | | Stakeholder add / resend | per IP / invite | 60 s | | File upload | per user | 5 s | Before horizontal scale-out, replace with a shared store (e.g. Redis) or edge rate limits. See [`backend-roadmap.md` §5](backend-roadmap.md#5-session-and-authentication-v1). ### 6.2 Web vitals storage Production defaults to **`external`** mode: vitals are structured log lines, not written to Postgres or local files. Setting `WEB_VITALS_STORAGE=local` uses a **per-process** file store under `.next/web-vitals` — suitable for dev/admin only, not multi-instance. See [`backend-roadmap.md` §7](backend-roadmap.md#7-api-responses-errors-and-observability). ## 7. Environment variables (steady-state) Cloudron **auto-injects** addon vars (`CLOUDRON_POSTGRESQL_URL`, `CLOUDRON_MAIL_SMTP_*`). Operators set these manually once per app; they persist across image updates unless changed: | Variable | Purpose | | -------- | ------- | | `SESSION_SECRET` | Session cookie signing (≥ 16 chars). Rotating logs everyone out. | | `SMTP_FROM` | Visible From on sign-in emails (e.g. `CommunityRule `). The app rewrites a leftover `Community Rule` display name to one word. | | `ENABLE_PRIVACY_POLICY` | `true` on staging so `/privacy` is reachable. Unset in production until counsel has read the policy (404, no public links). | | `NEXT_PUBLIC_ENABLE_BACKEND_SYNC` | `true` in staging/production — Postgres draft persistence. | | `UPLOAD_ROOT` | `/app/data/uploads` on Cloudron — required for file uploads. | | `OPS_PRUNE_TOKEN` | Bearer token for `POST /api/ops/prune`. Unset hides the route. Optional; the daily scheduler runs the prune inside the container without it. | | `WORKSHOP_INQUIRY_PURGE_TOKEN` | Bearer token for `POST /api/workshop-inquiries/purge`. Unset hides that route. | | `RULE_ADMIN_EMAILS` | Named admins. Comma-separated. Can open private Rules and `/rule-counts`. | | `NEXT_PUBLIC_UMAMI_SRC` | Umami script URL. Unset: no tracker. Pair with `NEXT_PUBLIC_UMAMI_WEBSITE_ID`. | | `NEXT_PUBLIC_UMAMI_WEBSITE_ID` | Umami website id for this hostname. | Full detail: [`ops-backend-deploy.md` §3](ops-backend-deploy.md#3-environment-variables). ## 8. Troubleshooting | Symptom | Likely cause | Action | | ------- | ------------ | ------ | | Image pull error on update | Private repo, wrong tag, or amd64 manifest missing | Confirm repo is public; verify pull with `--platform linux/amd64` (§3.1) | | Health `503` / `database: disconnected` | Postgres addon or `CLOUDRON_POSTGRESQL_URL` missing | Cloudron app → Environment | | Container crash on start | Migration failure | App logs around `prisma migrate deploy`; fix forward with new migration | | Magic link not sent | Mail addon, `SMTP_FROM`, or SES DNS | Cloudron mail logs (`CLOUDRON_MAIL_SMTP_*`); inbox/spam; SPF/DKIM for SES (§8.1) | | Upload `server_misconfigured` | `UPLOAD_ROOT` unset | `cloudron env set --app UPLOAD_ROOT=/app/data/uploads` | | No “Recommended” on method cards | Seed not run | §3.4 — `node prisma/seed.bundle.cjs` | | Rate limit too aggressive after deploy | Expected per §6.1 | Single instance only; limits reset on container restart | App logs: Cloudron dashboard → *Logs* tab, or `cloudron logs --app -f`. ### 8.1 Mail DNS when SES is the relay `communityrule.info` outbound mail is **Amazon SES SMTP** (`email-smtp.us-east-2.amazonaws.com:587`), not Cloudron's own MTA. Cloudron Mail → domain status therefore **skips SPF and DKIM** ("configure the relay provider") and only checks MX, DMARC (`v=DMARC1; p=reject; pct=100`), and that the SES connection works. That is expected. Recipients still authenticate the visible `From:` (`staging.app@communityrule.info` on staging) against **SES DKIM/SPF**, not `a:my.medlab.host`. **Operator steps (AWS + Cloudron DNS, not app code):** 1. In **AWS SES** (us-east-2), open the verified identity for `communityrule.info` (create one if missing). Copy the **DKIM CNAME** records SES shows (three `*._domainkey.communityrule.info` names). 2. In **Cloudron** → *Domains* → `communityrule.info` → DNS, add those CNAMEs. Cloudron's Namecheap provider publishes them to the registrar. Confirm with `dig +short CNAME ._domainkey.communityrule.info`. 3. Optional but recommended for SPF alignment: add `include:amazonses.com` to the existing TXT SPF, e.g. `v=spf1 include:amazonses.com a:my.medlab.host ~all`. Do not remove `a:my.medlab.host` until you know nothing still sends directly from the box. 4. Leave DMARC at `p=reject` once DKIM verifies in SES; if a provider still quarantines after DKIM is live, inspect that provider's headers before relaxing DMARC. 5. Retest: request a magic link to Gmail **and** a non-Gmail inbox (May First / university). Check spam. Staging From is `CommunityRule `. `SMTP_FROM` should stay the Cloudron mailbox (`staging.app@communityrule.info` on staging, `hello@communityrule.info` on the apex app). The app falls back to `CLOUDRON_MAIL_FROM` if `SMTP_FROM` is unset. ## 9. Retention (logs, backups, unused links) Periods the privacy policy states. Source of truth: [`docs/privacy-policy-facts.md`](../privacy-policy-facts.md) and [`lib/privacyRetention.ts`](../../lib/privacyRetention.ts). ### 9.1 Unused links and sessions [`CloudronManifest.json`](../../CloudronManifest.json) registers a **scheduler** task `prune-expired` at **07:20 UTC** daily. It runs `node /app/prune-expired.bundle.cjs` (the production bundle of [`scripts/prune-sessions.ts`](../../scripts/prune-sessions.ts)) and deletes expired sessions, unused login / save-progress / email-change tokens and the drafts they carry, and workshop inquiries older than 12 months. It leaves invitation token hashes in place: an expired link cannot be used to join, and that hash is how the person requests a new link. Manual runs: ```bash # inside the container (no token) cloudron exec --app staging.communityrule.info -- \ node /app/prune-expired.bundle.cjs # from outside, if OPS_PRUNE_TOKEN is set curl -sS -X POST \ -H "Authorization: Bearer $OPS_PRUNE_TOKEN" \ https://staging.communityrule.info/api/ops/prune ``` Local: ```bash node --env-file=.env --import tsx scripts/prune-sessions.ts ``` If the scheduler addon is missing after an image update, add the same command under the app **Cron** tab (`20 7 * * *`, UTC). The same scheduler addon also runs **`follow-up-mail`** at **15 minutes past every hour**. It sends one print follow-up to the address that holds the pen, 48 hours after the Rule was finished, with PDF and RULE.md links. Guest Rules with no account are skipped until claimed. Rules finished more than seven days before this mail existed are closed in the migration and are not sent. A Rule that is already due stays eligible until the job claims it, including after a scheduler outage longer than seven days. The claim is written before the message is sent, and cleared again if sending fails, so overlapping runs cannot deliver it twice. Manual run: ```bash cloudron exec --app staging.communityrule.info -- \ node /app/follow-up-mail.bundle.cjs # local node --env-file=.env --import tsx scripts/send-rule-follow-up.ts ``` Named admins can read finished-Rule counts (folded under five) at `/rule-counts`, or `GET /api/ops/rule-counts` with the same session or `Authorization: Bearer $OPS_PRUNE_TOKEN`. ### 9.2 Logs App logs (stdout) and Cloudron request logs are deleted after **14 days**. That is the platform default and the shortest period Cloudron allows. Page timing logs (`POST /api/web-vitals` in `external` mode) record origin + path only. Query strings are stripped before the line is written, so invitation and login tokens never land in app logs. Cloudron's reverse-proxy request logs still include the query string. They follow the same 14-day window. Do not export those logs off the host for longer than 14 days. Sign-in mail failures log a request id, not the email address. ### 9.3 Umami (site counts) Umami is a **second Cloudron app**, not this image. Install the Umami Cloudron package (or Docker) on a hostname such as `analytics.communityrule.info`. Create a website for this app’s public origin. In Umami → website settings, **turn off cookies** (tracker collects aggregate page views only). Then on this CommunityRule app: ```bash cloudron env set --app staging.communityrule.info \ NEXT_PUBLIC_UMAMI_SRC=https://analytics.communityrule.info/script.js cloudron env set --app staging.communityrule.info \ NEXT_PUBLIC_UMAMI_WEBSITE_ID= ``` Unset both to ship no tracker. The snippet sets `data-do-not-track` and `data-exclude-search` so query strings (share and export tokens) are not sent to Umami. ### 9.4 Backups Set the Cloudron backup site to **`30 daily`** (*Backups* → *Schedule & Retention*). Confirm this on staging and production. Cloudron also keeps a pre-update snapshot for up to three weeks. ## 10. Related docs - [`../privacy-policy-facts.md`](../privacy-policy-facts.md) — periods and hosting facts for the privacy policy. - [`ops-backend-deploy.md`](ops-backend-deploy.md) — first install, cutover plan, legacy rules archive, build/push deep dive. - [`backend-roadmap.md`](backend-roadmap.md) — migrations policy (§8), rate limiting (§5), environments (§11). - [`../relaunch-brief.md`](../relaunch-brief.md) — plain-language summary for MEDLab admin. - [`../../CONTRIBUTING.md`](../../CONTRIBUTING.md) — local dev setup.