Files
community-rule/docs/guides/ops-runbook.md
T

17 KiB
Raw Blame History

Steady-state operator runbook

Day-to-day deploy, rollback, and recovery for CommunityRule on MEDLab Cloudron. Assumes staging or production is already installed and smoke-tested.

First-time install, apex cutover, and legacy decommission live in ops-backend-deploy.md. Use this doc once an environment is already running.

1. Quick reference

Item Value
Cloudron dashboard https://my.medlab.host
Cloudron CLI login cloudron login my.medlab.host
Staging app staging.communityrule.info
Production app communityrule.info (after apex cutover)
Container image git.medlab.host/communityrule/community-rule:<tag>
Health check GET /api/health → 200 {"ok":true,"database":"connected"}
Manifest version CloudronManifest.json version field (must increase for each release)
Current manifest 0.1.8 at time of writing — always read the file before deploying

Replace <app> below with the Cloudron location (staging.communityrule.info or communityrule.info).

2. Prerequisites (one-time per operator)

  1. Cloudron access — admin login on my.medlab.host and a CLI API token (Profile → API Tokens on the dashboard). Save the token in 1Password.
  2. Cloudron CLI — logged in:
    cloudron login my.medlab.host
    
  3. Docker + buildx — Docker Desktop or equivalent with docker buildx.
  4. Gitea registry auth — personal access token on git.medlab.host with read:package + write:package; then:
    docker login git.medlab.host
    
  5. Repo checkout — clone CommunityRule/community-rule and work from a clean commit that matches the release you intend to ship.

Images are linux/amd64 only (Cloudron host is x86_64). On Apple Silicon, the release script still builds amd64 via buildx; a bare docker pull without --platform linux/amd64 failing on arm64 is expected.

3. Deploy a new version

Typical release flow: bump manifest → build/push image → cloudron update → smoke.

3.1 Build and push

  1. Check out the commit to release (main or a release branch).

  2. Bump CloudronManifest.json version (e.g. 0.1.8 → 0.1.9). Cloudron requires the manifest version to increase for cloudron update --image to be accepted.

  3. From the repo root, build and push (tag should match the manifest version for sanity):

    TAG=0.1.9 ./scripts/docker-release.sh
    # equivalent:
    TAG=0.1.9 npm run docker:release
    

    Omit TAG= to push git rev-parse --short HEAD instead — only do that for ad-hoc staging experiments; production releases should use semver tags aligned with the manifest.

  4. Verify anonymous pull (simulates Cloudron):

    docker logout git.medlab.host
    docker pull --platform linux/amd64 \
      git.medlab.host/communityrule/community-rule:0.1.9
    
  5. Commit the manifest bump in git alongside the code that shipped in this build.

Registry details and one-time Gitea setup: ops-backend-deploy.md §9.

3.2 Update Cloudron

cloudron update --app staging.communityrule.info \
  --image git.medlab.host/communityrule/community-rule:0.1.9

Use communityrule.info for production. Cloudron pulls the image (no registry credentials on the host), restarts the container, and runs scripts/start.sh, which:

  1. chowns /app/data (localstorage mount),
  2. runs prisma migrate deploy,
  3. writes the adopted team Rule (document and visibility together),
  4. execs the Next.js standalone server.

Watch the app Logs tab in the Cloudron dashboard for a clean migration and Listening on port 3000.

3.3 Migrations

Normal case: migrations apply automatically on container start — no separate step.

Manual re-run (only if debugging a failed deploy or verifying before traffic):

cloudron exec --app staging.communityrule.info -- npm run db:deploy

(npm run db:deploy → prisma migrate deploy.)

Policy: never run prisma migrate reset against staging or production. Never edit migration files already applied to a shared database. Fix schema drift by adding a new migration locally (prisma migrate dev) and deploying a new image. See backend-roadmap.md §8.

3.4 Seed data (not every deploy)

Template + facet seed (MethodFacet rows for create-flow “Recommended” tags) is not applied at boot. The adopted team Rule is the exception: container start writes that one row so the public catalog does not keep the old draft body. Run the template seed once per environment after first install, or when recommendations return all-zero scores:

cloudron exec --app staging.communityrule.info -- \
  node prisma/seed.bundle.cjs

Re-running is safe (idempotent upserts). JSON lives at /app/seed-data/ in the image — not under /app/data (Cloudron localstorage overwrites that mount).

3.5 Smoke after deploy

Automated (from your laptop, repo root):

./scripts/staging-smoke.sh staging.communityrule.info
# production:
./scripts/staging-smoke.sh communityrule.info

# optional — exercises magic-link request (check inbox manually):
EMAIL=you@example.com ./scripts/staging-smoke.sh staging.communityrule.info

Manual (still required for full acceptance):

  • Click a magic link → signed in → GET /api/auth/session returns a user.
  • Publish a rule end-to-end → public detail page loads.
  • Optional: Save & Exit draft sync; upload with UPLOAD_ROOT set.

Full checklist and failure table: ops-backend-deploy.md §10.

4. Roll back (code-only)

To revert application code without touching the database:

cloudron update --app staging.communityrule.info \
  --image git.medlab.host/communityrule/community-rule:<previous-tag>

Pick a tag you know was healthy (previous manifest version or git tag recorded at last good deploy).

Database implications:

  • Rolling back the image does not undo migrations already applied.
  • If the bad release added a migration, rolling back to an older image may leave the DB schema ahead of what that code expects — usually safe if the migration was additive (new nullable columns, new tables).
  • If the bad release broke because of a destructive or incompatible migration, do not reset production. Restore from a Cloudron backup (§5) or fix forward with a corrective migration.

Never prisma migrate reset on staging or production.

5. Restore drill (quarterly)

Verify Cloudron backups are restorable without touching the live app.

Cadence: at least once per quarter, or after any backup-policy change.

Steps:

  1. In the Cloudron dashboard, pick a recent automatic backup of <app> (Backups tab).
  2. Restore to a scratch location — e.g. restore-drill-YYYYMMDD.communityrule.info — not over the live app.
  3. After restore completes, confirm the container starts and migrations are current:
    curl -sS "https://restore-drill-YYYYMMDD.communityrule.info/api/health"
    
    Expect 200 with "database":"connected".
  4. Optional: cloudron exec --app restore-drill-YYYYMMDD.communityrule.info -- npm run db:deploy if logs show pending migrations on an older snapshot.
  5. Run ./scripts/staging-smoke.sh restore-drill-YYYYMMDD.communityrule.info.
  6. Uninstall the scratch app when done.

Record the drill date and outcome in your ops notes. Cloudron retains automatic backups per platform defaults; confirm retention in the dashboard.

6. Single-instance limitations

The current Cloudron deploy runs one container per environment. Do not scale to multiple app instances without addressing these per-process limits:

6.1 In-memory rate limiter

lib/server/rateLimit.ts stores windows in process memory. Limits apply per container, not globally across instances.

Route / action Key Min interval
Magic-link request per email 60 s
Magic-link request per IP 20 s
Email change request per email / IP / user 60 s
Organizer inquiry per email / IP 60 s / 20 s
Publish with stakeholder invites per IP 60 s
Stakeholder add / resend per IP / invite 60 s
File upload per user 5 s

Before horizontal scale-out, replace with a shared store (e.g. Redis) or edge rate limits. See backend-roadmap.md §5.

6.2 Web vitals storage

Production defaults to external mode: vitals are structured log lines, not written to Postgres or local files. Setting WEB_VITALS_STORAGE=local uses a per-process file store under .next/web-vitals — suitable for dev/admin only, not multi-instance. See backend-roadmap.md §7.

7. Environment variables (steady-state)

Cloudron auto-injects addon vars (CLOUDRON_POSTGRESQL_URL, CLOUDRON_MAIL_SMTP_*). Operators set these manually once per app; they persist across image updates unless changed:

Variable Purpose
SESSION_SECRET Session cookie signing (≥ 16 chars). Rotating logs everyone out.
SMTP_FROM Visible From on sign-in emails (e.g. Community Rule <hello@communityrule.info>).
NEXT_PUBLIC_ENABLE_BACKEND_SYNC true in staging/production — Postgres draft persistence.
UPLOAD_ROOT /app/data/uploads on Cloudron — required for file uploads.
OPS_PRUNE_TOKEN Bearer token for POST /api/ops/prune. Unset hides the route. Optional; the daily scheduler runs the prune inside the container without it.
WORKSHOP_INQUIRY_PURGE_TOKEN Bearer token for POST /api/workshop-inquiries/purge. Unset hides that route.
RULE_ADMIN_EMAILS Named admins. Comma-separated. Can open private Rules and /rule-counts.
NEXT_PUBLIC_UMAMI_SRC Umami script URL. Unset: no tracker. Pair with NEXT_PUBLIC_UMAMI_WEBSITE_ID.
NEXT_PUBLIC_UMAMI_WEBSITE_ID Umami website id for this hostname.

Full detail: ops-backend-deploy.md §3.

8. Troubleshooting

Symptom Likely cause Action
Image pull error on update Private repo, wrong tag, or amd64 manifest missing Confirm repo is public; verify pull with --platform linux/amd64 (§3.1)
Health 503 / database: disconnected Postgres addon or CLOUDRON_POSTGRESQL_URL missing Cloudron app → Environment
Container crash on start Migration failure App logs around prisma migrate deploy; fix forward with new migration
Magic link not sent Mail addon, SMTP_FROM, or SES DNS Cloudron mail logs (CLOUDRON_MAIL_SMTP_*); inbox/spam; SPF/DKIM for SES (§8.1)
Upload server_misconfigured UPLOAD_ROOT unset cloudron env set --app <app> UPLOAD_ROOT=/app/data/uploads
No “Recommended” on method cards Seed not run §3.4 — node prisma/seed.bundle.cjs
Rate limit too aggressive after deploy Expected per §6.1 Single instance only; limits reset on container restart

App logs: Cloudron dashboard → Logs tab, or cloudron logs --app <app> -f.

8.1 Mail DNS when SES is the relay

communityrule.info outbound mail is Amazon SES SMTP (email-smtp.us-east-2.amazonaws.com:587), not Cloudron's own MTA. Cloudron Mail → domain status therefore skips SPF and DKIM ("configure the relay provider") and only checks MX, DMARC (v=DMARC1; p=reject; pct=100), and that the SES connection works.

That is expected. Recipients still authenticate the visible From: (staging.app@communityrule.info on staging) against SES DKIM/SPF, not a:my.medlab.host.

Operator steps (AWS + Cloudron DNS, not app code):

  1. In AWS SES (us-east-2), open the verified identity for communityrule.info (create one if missing). Copy the DKIM CNAME records SES shows (three *._domainkey.communityrule.info names).
  2. In Cloudron → Domains → communityrule.info → DNS, add those CNAMEs. Cloudron's Namecheap provider publishes them to the registrar. Confirm with dig +short CNAME <selector>._domainkey.communityrule.info.
  3. Optional but recommended for SPF alignment: add include:amazonses.com to the existing TXT SPF, e.g. v=spf1 include:amazonses.com a:my.medlab.host ~all. Do not remove a:my.medlab.host until you know nothing still sends directly from the box.
  4. Leave DMARC at p=reject once DKIM verifies in SES; if a provider still quarantines after DKIM is live, inspect that provider's headers before relaxing DMARC.
  5. Retest: request a magic link to Gmail and a non-Gmail inbox (May First / university). Check spam. Staging From is Community Rule <staging.app@communityrule.info>.

SMTP_FROM should stay the Cloudron mailbox (staging.app@communityrule.info on staging, hello@communityrule.info on the apex app). The app falls back to CLOUDRON_MAIL_FROM if SMTP_FROM is unset.

Periods the privacy policy states. Source of truth: docs/privacy-policy-facts.md and lib/privacyRetention.ts.

CloudronManifest.json registers a scheduler task prune-expired at 07:20 UTC daily. It runs node /app/prune-expired.bundle.cjs (the production bundle of scripts/prune-sessions.ts) and deletes expired sessions, unused login / save-progress / email-change tokens and the drafts they carry, and workshop inquiries older than 12 months. It leaves invitation token hashes in place: an expired link cannot be used to join, and that hash is how the person requests a new link.

Manual runs:

# inside the container (no token)
cloudron exec --app staging.communityrule.info -- \
  node /app/prune-expired.bundle.cjs

# from outside, if OPS_PRUNE_TOKEN is set
curl -sS -X POST \
  -H "Authorization: Bearer $OPS_PRUNE_TOKEN" \
  https://staging.communityrule.info/api/ops/prune

Local:

node --env-file=.env --import tsx scripts/prune-sessions.ts

If the scheduler addon is missing after an image update, add the same command under the app Cron tab (20 7 * * *, UTC).

The same scheduler addon also runs follow-up-mail at 15 minutes past every hour. It sends one print follow-up to the address that holds the pen, 48 hours after the Rule was finished, with PDF and RULE.md links. Guest Rules with no account are skipped until claimed. Rules finished more than seven days before this mail existed are closed in the migration and are not sent. A Rule that is already due stays eligible until the job claims it, including after a scheduler outage longer than seven days. The claim is written before the message is sent, and cleared again if sending fails, so overlapping runs cannot deliver it twice.

Manual run:

cloudron exec --app staging.communityrule.info -- \
  node /app/follow-up-mail.bundle.cjs

# local
node --env-file=.env --import tsx scripts/send-rule-follow-up.ts

Named admins can read finished-Rule counts (folded under five) at /rule-counts, or GET /api/ops/rule-counts with the same session or Authorization: Bearer $OPS_PRUNE_TOKEN.

9.2 Logs

App logs (stdout) and Cloudron request logs are deleted after 14 days. That is the platform default and the shortest period Cloudron allows.

Page timing logs (POST /api/web-vitals in external mode) record origin + path only. Query strings are stripped before the line is written, so invitation and login tokens never land in app logs.

Cloudron's reverse-proxy request logs still include the query string. They follow the same 14-day window. Do not export those logs off the host for longer than 14 days.

Sign-in mail failures log a request id, not the email address.

9.3 Umami (site counts)

Umami is a second Cloudron app, not this image. Install the Umami Cloudron package (or Docker) on a hostname such as analytics.communityrule.info. Create a website for this app’s public origin. In Umami → website settings, turn off cookies (tracker collects aggregate page views only).

Then on this CommunityRule app:

cloudron env set --app staging.communityrule.info \
  NEXT_PUBLIC_UMAMI_SRC=https://analytics.communityrule.info/script.js
cloudron env set --app staging.communityrule.info \
  NEXT_PUBLIC_UMAMI_WEBSITE_ID=<website-id>

Unset both to ship no tracker. The snippet sets data-do-not-track and data-exclude-search so query strings (share and export tokens) are not sent to Umami.

9.4 Backups

Set the Cloudron backup site to 30 daily (Backups → Schedule & Retention). Confirm this on staging and production. Cloudron also keeps a pre-update snapshot for up to three weeks.