17 KiB
Steady-state operator runbook
Day-to-day deploy, rollback, and recovery for CommunityRule on MEDLab Cloudron. Assumes staging or production is already installed and smoke-tested.
First-time install, apex cutover, and legacy decommission live in
ops-backend-deploy.md. Use this doc once an environment is already running.
1. Quick reference
| Item | Value |
|---|---|
| Cloudron dashboard | https://my.medlab.host |
| Cloudron CLI login | cloudron login my.medlab.host |
| Staging app | staging.communityrule.info |
| Production app | communityrule.info (after apex cutover) |
| Container image | git.medlab.host/communityrule/community-rule:<tag> |
| Health check | GET /api/health → 200 {"ok":true,"database":"connected"} |
| Manifest version | CloudronManifest.json version field (must increase for each release) |
| Current manifest | 0.1.8 at time of writing — always read the file before deploying |
Replace <app> below with the Cloudron location (staging.communityrule.info
or communityrule.info).
2. Prerequisites (one-time per operator)
- Cloudron access — admin login on
my.medlab.hostand a CLI API token (Profile → API Tokens on the dashboard). Save the token in 1Password. - Cloudron CLI — logged in:
cloudron login my.medlab.host - Docker + buildx — Docker Desktop or equivalent with
docker buildx. - Gitea registry auth — personal access token on
git.medlab.hostwithread:package+write:package; then:docker login git.medlab.host - Repo checkout — clone
CommunityRule/community-ruleand work from a clean commit that matches the release you intend to ship.
Images are linux/amd64 only (Cloudron host is x86_64). On Apple
Silicon, the release script still builds amd64 via buildx; a bare
docker pull without --platform linux/amd64 failing on arm64 is expected.
3. Deploy a new version
Typical release flow: bump manifest → build/push image → cloudron update
→ smoke.
3.1 Build and push
-
Check out the commit to release (
mainor a release branch). -
Bump
CloudronManifest.jsonversion(e.g.0.1.8→0.1.9). Cloudron requires the manifest version to increase forcloudron update --imageto be accepted. -
From the repo root, build and push (tag should match the manifest version for sanity):
TAG=0.1.9 ./scripts/docker-release.sh # equivalent: TAG=0.1.9 npm run docker:releaseOmit
TAG=to pushgit rev-parse --short HEADinstead — only do that for ad-hoc staging experiments; production releases should use semver tags aligned with the manifest. -
Verify anonymous pull (simulates Cloudron):
docker logout git.medlab.host docker pull --platform linux/amd64 \ git.medlab.host/communityrule/community-rule:0.1.9 -
Commit the manifest bump in git alongside the code that shipped in this build.
Registry details and one-time Gitea setup: ops-backend-deploy.md §9.
3.2 Update Cloudron
cloudron update --app staging.communityrule.info \
--image git.medlab.host/communityrule/community-rule:0.1.9
Use communityrule.info for production. Cloudron pulls the image (no registry
credentials on the host), restarts the container, and runs
scripts/start.sh, which:
chowns/app/data(localstorage mount),- runs
prisma migrate deploy, - execs the Next.js standalone server.
Watch the app Logs tab in the Cloudron dashboard for a clean migration and
Listening on port 3000.
3.3 Migrations
Normal case: migrations apply automatically on container start — no separate step.
Manual re-run (only if debugging a failed deploy or verifying before traffic):
cloudron exec --app staging.communityrule.info -- npm run db:deploy
(npm run db:deploy → prisma migrate deploy.)
Policy: never run prisma migrate reset against staging or production.
Never edit migration files already applied to a shared database. Fix schema
drift by adding a new migration locally (prisma migrate dev) and
deploying a new image. See backend-roadmap.md §8.
3.4 Seed data (not every deploy)
Template + facet seed (MethodFacet rows for create-flow “Recommended” tags)
is not applied at boot. Run once per environment after first install, or
when recommendations return all-zero scores:
cloudron exec --app staging.communityrule.info -- \
node prisma/seed.bundle.cjs
Re-running is safe (idempotent upserts). JSON lives at /app/seed-data/ in
the image — not under /app/data (Cloudron localstorage overwrites that
mount).
3.5 Smoke after deploy
Automated (from your laptop, repo root):
./scripts/staging-smoke.sh staging.communityrule.info
# production:
./scripts/staging-smoke.sh communityrule.info
# optional — exercises magic-link request (check inbox manually):
EMAIL=you@example.com ./scripts/staging-smoke.sh staging.communityrule.info
Manual (still required for full acceptance):
- Click a magic link → signed in →
GET /api/auth/sessionreturns a user. - Publish a rule end-to-end → public detail page loads.
- Optional: Save & Exit draft sync; upload with
UPLOAD_ROOTset.
Full checklist and failure table: ops-backend-deploy.md §10.
4. Roll back (code-only)
To revert application code without touching the database:
cloudron update --app staging.communityrule.info \
--image git.medlab.host/communityrule/community-rule:<previous-tag>
Pick a tag you know was healthy (previous manifest version or git tag recorded at last good deploy).
Database implications:
- Rolling back the image does not undo migrations already applied.
- If the bad release added a migration, rolling back to an older image may leave the DB schema ahead of what that code expects — usually safe if the migration was additive (new nullable columns, new tables).
- If the bad release broke because of a destructive or incompatible migration, do not reset production. Restore from a Cloudron backup (§5) or fix forward with a corrective migration.
Never prisma migrate reset on staging or production.
5. Restore drill (quarterly)
Verify Cloudron backups are restorable without touching the live app.
Cadence: at least once per quarter, or after any backup-policy change.
Steps:
- In the Cloudron dashboard, pick a recent automatic backup of
<app>(Backups tab). - Restore to a scratch location — e.g.
restore-drill-YYYYMMDD.communityrule.info— not over the live app. - After restore completes, confirm the container starts and migrations are
current:
Expect
curl -sS "https://restore-drill-YYYYMMDD.communityrule.info/api/health"200with"database":"connected". - Optional:
cloudron exec --app restore-drill-YYYYMMDD.communityrule.info -- npm run db:deployif logs show pending migrations on an older snapshot. - Run
./scripts/staging-smoke.sh restore-drill-YYYYMMDD.communityrule.info. - Uninstall the scratch app when done.
Record the drill date and outcome in your ops notes. Cloudron retains automatic backups per platform defaults; confirm retention in the dashboard.
6. Single-instance limitations
The current Cloudron deploy runs one container per environment. Do not scale to multiple app instances without addressing these per-process limits:
6.1 In-memory rate limiter
lib/server/rateLimit.ts stores windows in
process memory. Limits apply per container, not globally across instances.
| Route / action | Key | Min interval |
|---|---|---|
| Magic-link request | per email | 60 s |
| Magic-link request | per IP | 20 s |
| Email change request | per email / IP / user | 60 s |
| Organizer inquiry | per email / IP | 60 s / 20 s |
| Publish with stakeholder invites | per IP | 60 s |
| Stakeholder add / resend | per IP / invite | 60 s |
| File upload | per user | 5 s |
Before horizontal scale-out, replace with a shared store (e.g. Redis) or edge
rate limits. See backend-roadmap.md §5.
6.2 Web vitals storage
Production defaults to external mode: vitals are structured log lines, not
written to Postgres or local files. Setting WEB_VITALS_STORAGE=local uses a
per-process file store under .next/web-vitals — suitable for dev/admin
only, not multi-instance. See backend-roadmap.md §7.
7. Environment variables (steady-state)
Cloudron auto-injects addon vars (CLOUDRON_POSTGRESQL_URL,
CLOUDRON_MAIL_SMTP_*). Operators set these manually once per app; they
persist across image updates unless changed:
| Variable | Purpose |
|---|---|
SESSION_SECRET |
Session cookie signing (≥ 16 chars). Rotating logs everyone out. |
SMTP_FROM |
Visible From on sign-in emails (e.g. Community Rule <hello@communityrule.info>). |
NEXT_PUBLIC_ENABLE_BACKEND_SYNC |
true in staging/production — Postgres draft persistence. |
UPLOAD_ROOT |
/app/data/uploads on Cloudron — required for file uploads. |
OPS_PRUNE_TOKEN |
Bearer token for POST /api/ops/prune. Unset hides the route. Optional; the daily scheduler runs the prune inside the container without it. |
WORKSHOP_INQUIRY_PURGE_TOKEN |
Bearer token for POST /api/workshop-inquiries/purge. Unset hides that route. |
RULE_ADMIN_EMAILS |
Named admins. Comma-separated. Can open private Rules and /rule-counts. |
NEXT_PUBLIC_UMAMI_SRC |
Umami script URL. Unset: no tracker. Pair with NEXT_PUBLIC_UMAMI_WEBSITE_ID. |
NEXT_PUBLIC_UMAMI_WEBSITE_ID |
Umami website id for this hostname. |
Full detail: ops-backend-deploy.md §3.
8. Troubleshooting
| Symptom | Likely cause | Action |
|---|---|---|
| Image pull error on update | Private repo, wrong tag, or amd64 manifest missing | Confirm repo is public; verify pull with --platform linux/amd64 (§3.1) |
Health 503 / database: disconnected |
Postgres addon or CLOUDRON_POSTGRESQL_URL missing |
Cloudron app → Environment |
| Container crash on start | Migration failure | App logs around prisma migrate deploy; fix forward with new migration |
| Magic link not sent | Mail addon, SMTP_FROM, or SES DNS |
Cloudron mail logs (CLOUDRON_MAIL_SMTP_*); inbox/spam; SPF/DKIM for SES (§8.1) |
Upload server_misconfigured |
UPLOAD_ROOT unset |
cloudron env set --app <app> UPLOAD_ROOT=/app/data/uploads |
| No “Recommended” on method cards | Seed not run | §3.4 — node prisma/seed.bundle.cjs |
| Rate limit too aggressive after deploy | Expected per §6.1 | Single instance only; limits reset on container restart |
App logs: Cloudron dashboard → Logs tab, or cloudron logs --app <app> -f.
8.1 Mail DNS when SES is the relay
communityrule.info outbound mail is Amazon SES SMTP (email-smtp.us-east-2.amazonaws.com:587), not Cloudron's own MTA. Cloudron Mail → domain status therefore skips SPF and DKIM ("configure the relay provider") and only checks MX, DMARC (v=DMARC1; p=reject; pct=100), and that the SES connection works.
That is expected. Recipients still authenticate the visible From: (staging.app@communityrule.info on staging) against SES DKIM/SPF, not a:my.medlab.host.
Operator steps (AWS + Cloudron DNS, not app code):
- In AWS SES (us-east-2), open the verified identity for
communityrule.info(create one if missing). Copy the DKIM CNAME records SES shows (three*._domainkey.communityrule.infonames). - In Cloudron → Domains →
communityrule.info→ DNS, add those CNAMEs. Cloudron's Namecheap provider publishes them to the registrar. Confirm withdig +short CNAME <selector>._domainkey.communityrule.info. - Optional but recommended for SPF alignment: add
include:amazonses.comto the existing TXT SPF, e.g.v=spf1 include:amazonses.com a:my.medlab.host ~all. Do not removea:my.medlab.hostuntil you know nothing still sends directly from the box. - Leave DMARC at
p=rejectonce DKIM verifies in SES; if a provider still quarantines after DKIM is live, inspect that provider's headers before relaxing DMARC. - Retest: request a magic link to Gmail and a non-Gmail inbox (May First / university). Check spam. Staging From is
Community Rule <staging.app@communityrule.info>.
SMTP_FROM should stay the Cloudron mailbox (staging.app@communityrule.info on staging, hello@communityrule.info on the apex app). The app falls back to CLOUDRON_MAIL_FROM if SMTP_FROM is unset.
9. Retention (logs, backups, unused links)
Periods the privacy policy states. Source of truth:
docs/privacy-policy-facts.md and
lib/privacyRetention.ts.
9.1 Unused links and sessions
CloudronManifest.json registers a
scheduler task prune-expired at 07:20 UTC daily. It runs
node /app/prune-expired.bundle.cjs (the production bundle of
scripts/prune-sessions.ts) and
deletes expired sessions, unused login / save-progress / email-change
tokens and the drafts they carry, and workshop inquiries older than 12
months. It leaves invitation token hashes in place: an expired link cannot
be used to join, and that hash is how the person requests a new link.
Manual runs:
# inside the container (no token)
cloudron exec --app staging.communityrule.info -- \
node /app/prune-expired.bundle.cjs
# from outside, if OPS_PRUNE_TOKEN is set
curl -sS -X POST \
-H "Authorization: Bearer $OPS_PRUNE_TOKEN" \
https://staging.communityrule.info/api/ops/prune
Local:
node --env-file=.env --import tsx scripts/prune-sessions.ts
If the scheduler addon is missing after an image update, add the same
command under the app Cron tab (20 7 * * *, UTC).
The same scheduler addon also runs follow-up-mail at 15 minutes
past every hour. It sends one print follow-up to the address that holds
the pen, 48 hours after the Rule was finished, with PDF and RULE.md
links. Guest Rules with no account are skipped until claimed. Rules
finished more than seven days before this mail existed are closed in the
migration and are not sent. A Rule that is already due stays eligible
until the job claims it, including after a scheduler outage longer than
seven days. The claim is written before the message is sent, and cleared
again if sending fails, so overlapping runs cannot deliver it twice.
Manual run:
cloudron exec --app staging.communityrule.info -- \
node /app/follow-up-mail.bundle.cjs
# local
node --env-file=.env --import tsx scripts/send-rule-follow-up.ts
Named admins can read finished-Rule counts (folded under five) at
/rule-counts, or GET /api/ops/rule-counts with the same session or
Authorization: Bearer $OPS_PRUNE_TOKEN.
9.2 Logs
App logs (stdout) and Cloudron request logs are deleted after 14 days. That is the platform default and the shortest period Cloudron allows.
Page timing logs (POST /api/web-vitals in external mode) record
origin + path only. Query strings are stripped before the line is
written, so invitation and login tokens never land in app logs.
Cloudron's reverse-proxy request logs still include the query string. They follow the same 14-day window. Do not export those logs off the host for longer than 14 days.
Sign-in mail failures log a request id, not the email address.
9.3 Umami (site counts)
Umami is a second Cloudron app, not this image. Install the Umami
Cloudron package (or Docker) on a hostname such as
analytics.communityrule.info. Create a website for this app’s public
origin. In Umami → website settings, turn off cookies (tracker
collects aggregate page views only).
Then on this CommunityRule app:
cloudron env set --app staging.communityrule.info \
NEXT_PUBLIC_UMAMI_SRC=https://analytics.communityrule.info/script.js
cloudron env set --app staging.communityrule.info \
NEXT_PUBLIC_UMAMI_WEBSITE_ID=<website-id>
Unset both to ship no tracker. The snippet sets data-do-not-track and
data-exclude-search so query strings (share and export tokens) are not
sent to Umami.
9.4 Backups
Set the Cloudron backup site to 30 daily (Backups → Schedule &
Retention). Confirm this on staging and production. Cloudron also keeps
a pre-update snapshot for up to three weeks.
10. Related docs
../privacy-policy-facts.md— periods and hosting facts for the privacy policy.ops-backend-deploy.md— first install, cutover plan, legacy rules archive, build/push deep dive.backend-roadmap.md— migrations policy (§8), rate limiting (§5), environments (§11).../relaunch-brief.md— plain-language summary for MEDLab admin.../../CONTRIBUTING.md— local dev setup.