# Backend roadmap (reference) Temporary working notes for building the backend. Safe to delete once the stack is stable. --- ## 1. Where we are - **Next.js 16** single repo ([`package.json`](package.json)). - **PostgreSQL + Prisma**: schema and migrations under `prisma/`; product APIs under `app/api/*` (health, auth/magic-link, session, drafts, rules, templates, web-vitals). - **Server modules** in `lib/server/` (db, session, mail, rate limiting, etc.). - **Create flow:** **Anonymous** users mirror in-progress state to **`create-flow-anonymous`** in `localStorage`; **Exit** opens the save-progress magic-link modal; after verify, [`PostLoginDraftTransfer`](app/create/PostLoginDraftTransfer.tsx) can **PUT** `/api/drafts/me` when **`NEXT_PUBLIC_ENABLE_BACKEND_SYNC=true`**. **Signed-in** users start a **fresh** in-memory session per “Create rule”; **Save & Exit** (from `select` onward) **PUT**s when sync is on. **Log in** from the marketing header uses the global modal ([`AuthModalProvider`](app/contexts/AuthModalContext.tsx)); **`/login`** remains for verify errors and deep links. - **Web vitals** [`app/api/web-vitals/route.ts`](app/api/web-vitals/route.ts) still use **file-based** storage under `.next` (not suitable for multi-instance production). - **CI:** [`.gitea/workflows/ci.yaml`](.gitea/workflows/ci.yaml) (build, test, lint, `prisma validate`); no in-repo production deploy definition. ### HTTP API (implemented in repo) Mirrors [CONTRIBUTING.md](../CONTRIBUTING.md) **API routes** table; handlers live under `app/api/*/route.ts`. | Method | Path | Purpose | | ---------- | ------------------------------ | --------------------------------------------- | | GET | `/api/health` | Liveness / DB check | | GET | `/api/auth/session` | Current user or null | | POST | `/api/auth/magic-link/request` | Send sign-in link email | | GET | `/api/auth/magic-link/verify` | Validate token, set session cookie, redirect | | POST | `/api/auth/logout` | Clear session | | GET / PUT | `/api/drafts/me` | Load or save create-flow JSON (authenticated) | | GET / POST | `/api/rules` | List or publish rules | | GET | `/api/templates` | List curated templates | **Product sign-in** uses **magic link** (`/api/auth/magic-link/*`). **Also present (not in CONTRIBUTING table):** `POST` / `GET` [`/api/web-vitals`](../app/api/web-vitals/route.ts) — file-based store today; production path TBD (§7). ### HTTP API (profile / account — not implemented yet) Planned for the signed-in profile/dashboard ([Figma profile frame](https://www.figma.com/design/agv0VBLiBlcnSAaiAORgPR/Community-Rule-System?node-id=22143-900069); [docs/backend-linear-tickets.md](backend-linear-tickets.md) Ticket 15; Linear **[CR-86](https://linear.app/community-rule/issue/CR-86/backend-profile-dashboard-account-figma-profile)**): - Authenticated list of **own** `PublishedRule` rows (e.g. `GET /api/rules/me` or a strictly scoped query—**not** the same as public `GET /api/rules`). - Owner-only **delete** and **duplicate** (clone) for published rules. - **Delete account** (authenticated), with an explicit policy for drafts, sessions, and linked rules. **Future (separate ticket):** **Change email** with verification (e.g. magic link to a new address, conflict handling)—**out of scope** for the profile milestone above. --- ## 2. What we’re building **Step 1.** Treat this as **greenfield**: new **PostgreSQL** database and new tables. Do **not** migrate data from the old Community Rule backend. **Step 2.** Keep the backend **inside this Next app**: - HTTP handlers under `app/api/…` - Shared server code under `lib/server/…` **Step 3.** Use the old backend only as a **product hint** (passwordless email sign-in, saving rules, listing rules). Do **not** copy its Express layout or MySQL schema. --- ## 3. Stack choices **Step 1.** Use **PostgreSQL** everywhere (local Docker, staging, production). **Step 2.** Use **Prisma** — `schema.prisma`, `npx prisma migrate dev` / `migrate deploy`. **Step 3.** Add **SMTP** (or Mailhog locally) for **magic-link** sign-in email in deployed environments; when `SMTP_URL` is unset in dev, the app can log the **verify URL** to the console (same pattern as [`lib/server/mail.ts`](lib/server/mail.ts)). **Step 4.** **Redis / queues / Kubernetes** — not required for v1. **Exception:** before running **multiple app instances**, plan a **shared rate-limit store** (often Redis) for **passwordless email (magic-link request)**; the current limiter is in-memory per process ([`lib/server/rateLimit.ts`](lib/server/rateLimit.ts)). --- ## 4. Data to model (first pass) Plain-English entities (names can evolve): | Area | Purpose | | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | **User** | Identified by email after **magic link verification** (primary v1 path). An optional **display name** (or preferred name) could be added later for richer greetings; it does **not** block the profile page—no schema commitment in this roadmap pass alone. | | **Session** | **Custom v1:** HttpOnly cookie; opaque token; **hash** stored in DB ([`lib/server/session.ts`](lib/server/session.ts)). Not NextAuth/Lucia. | | **MagicLinkToken** | Short-lived **hashed** token for email sign-in links; optional `nextPath` for post-login redirect. | | **RuleDraft** | **One** JSON blob per user (create-flow state). Schema already has **`updatedAt`**; no draft **versioning** or **multiple named drafts** in v1. | | **PublishedRule** | Saved rule after publish (title, summary, document JSON). Profile UI badges such as **IN PROGRESS** may be **derived from `document` JSON**, a future `status` column, or UI-only—product decision when implementing Ticket 15. | | **RuleTemplate** | Curated templates (slug, category, ordering, `body` JSON). **v1 API** lists rows for cards / create entry; **not** yet a recommendation engine (see below). | **RuleTemplate — recommendation matrix (after v1 list):** Product may author templates in **spreadsheets** (e.g. one row per governance pattern, columns for **matching dimensions** such as group size, organization type, location, maturity, plus long-form fields for create-flow prefill). That implies: **normalized schema or versioned JSON** for dimensions × template fit (✓/✗, weights, or scores), an **import path** (export `.xlsx` / Sheets → validate → DB or build-time artifact), and **`GET /api/templates` (or a sibling route)** that accepts **user- or wizard-selected facets** and returns a **ranked or filtered** set. **Out of scope for first ship** of Tickets 7–8 (seed + display list); tracked as **Ticket 16** in [docs/backend-linear-tickets.md](backend-linear-tickets.md) and Linear **[CR-88](https://linear.app/community-rule/issue/CR-88/backend-template-recommendation-matrix-xlsx-sheets-ingestion)**. Prefer **batch import** over live Google Sheets API in production unless ops explicitly wants sync. **Session follow-ups to implement or decide:** token **rotation** on sensitive events, whether **new login invalidates other sessions**, and **cleanup** of expired `Session` rows (job or lazy delete). Revisit a small auth library (e.g. Auth.js, Lucia) only if maintaining custom code becomes costly. **RuleDraft future (not v1):** versioning, multiple drafts per user, easier corruption recovery—only if product needs them. Align JSON shapes with `app/create/types.ts` as it matures. --- ## 5. Session and authentication (v1) - **Decision:** **Custom** database-backed sessions + **email magic link**; cookies are **httpOnly**; session and magic-link tokens are hashed at rest. - **Rate limiting (magic-link request):** **In-memory** is acceptable for a **single Node process**. It does **not** coordinate across instances—**add a shared limiter (e.g. Redis)** before horizontal scaling or serious abuse exposure. - Do **not** treat “switch to NextAuth/Lucia” as required for v1; document the custom lifecycle above instead. --- ## 6. Authorization (v1) Match the current API behavior; tighten as product evolves: - **`GET /api/drafts/me` / `PUT /api/drafts/me`:** Authenticated user only; draft is **scoped to that user** (`userId`). - **`POST /api/rules`:** Authenticated user only; rule is stored with **`userId`** (owner). - **`GET /api/rules`:** **Public list** of published rules (metadata: id, title, summary, timestamps)—no auth required today. **Not** a private “my rules” feed unless you add a separate route later (see §1 “profile / account — not implemented yet” and Ticket 15). - **Profile / owner scope (planned):** Authenticated **list own rules**, **delete own rule**, **duplicate own rule**—required for the signed-in dashboard in design; **v1 shipped handlers** may not include these until that work lands. - **Delete account (planned):** Authenticated endpoint + UX to remove the user record per policy (cascade vs orphan `PublishedRule`, drafts, sessions)—Ticket 15. **Change email** is **not** part of that milestone; plan a **future ticket** for verified email updates. - **v1 (shipped today):** No **editing** or **deleting** published rules via API in current handlers; no **sharing** or **collaborative ownership**—treat each rule as **owned by one user** until product defines more. --- ## 7. API responses, errors, and observability **Error JSON (target):** Prefer a stable shape, e.g. `{ "error": { "code": "string", "message": "string" }, "details"?: ... }` for 4xx/5xx, instead of only `{ "error": "string" }`. Validation errors can map into `details`. Implement gradually in route handlers. **Logging:** Use the shared [`lib/logger.ts`](../lib/logger.ts) where possible. Include a **request correlation id** (reuse `x-request-id` if present, else generate) on API routes and log it with errors so support can tie logs together. **Metrics:** No vendor required for v1; optional later: request duration, error counts. **Web vitals:** **Default** is **external** RUM or log drain (e.g. host analytics, Vercel Analytics, OpenTelemetry, SaaS APM)—keep **product Postgres** focused on product entities. Storing vitals in Postgres is an **explicit tradeoff** only if ops strongly wants a single datastore. --- ## 8. Prisma migrations policy - **Never edit** migration files that have **already been applied** to **staging or production** (or any shared database). Fixing schema drift = **add a new migration**. - **Local dev:** `prisma migrate dev` creates migrations; **deployed envs:** `prisma migrate deploy` before serving new code that depends on the schema. --- ## 9. Build order (implementation steps) **Operator / local (always manual):** Steps 1–4 — env file, Docker Postgres, `npm ci`, `prisma migrate dev`, `npm run dev`. **Backend behavior already in the repo:** Steps **5–10** match implemented Route Handlers and middleware (`lib/server/*`). **Step 11** (web vitals) is **not** production-ready (files under `.next`); treat as follow-up work aligned with §7. **Product / frontend still open (not only “backend exists”):** Sign-in UI, wiring publish from the create flow, template seed + UI consumption (flat list first), **spreadsheet-driven template recommendations** (Ticket 16 / [CR-88](https://linear.app/community-rule/issue/CR-88/backend-template-recommendation-matrix-xlsx-sheets-ingestion) — after v1 templates), **profile / my rules dashboard** (Ticket 15)—see §12 and [docs/backend-linear-tickets.md](backend-linear-tickets.md). --- **Step 1.** Copy `.env.example` to `.env`. Set `DATABASE_URL` and secrets (see file comments). **Step 2.** Start Postgres locally: ```bash docker compose up -d postgres ``` **Step 3.** Install dependencies and apply migrations: ```bash npm ci npx prisma migrate dev ``` **Step 4.** Run the app: ```bash npm run dev ``` **Step 5.** Confirm **health**: `GET /api/health` should return JSON. **Step 6.** **Magic-link sign-in** (happy path): 1. `POST /api/auth/magic-link/request` with `{ "email": "you@example.com" }` (optional `"next"` for redirect after verify). 2. Open the link from email, Mailhog, or **server logs** when `SMTP_URL` is unset (dev). 3. Browser hits `GET /api/auth/magic-link/verify?token=...` (and optional `next=...`); response sets the session cookie and redirects. 4. `GET /api/auth/session` should show your user in the same browser. **Before wiring create-flow session UI:** Confirm the same browser that completed verify gets `user` from `GET /api/auth/session` (cookie + same-site). On **staging/production**, magic-link emails embed the app origin—misconfigured **`Host`** or TLS termination can produce broken links; align reverse proxy with the public site URL. **Step 7.** **Drafts**: With a session, `GET /api/drafts/me` and `PUT /api/drafts/me` with `{ "payload": { ... } }` (create flow state object). **Step 8.** **Publish**: `POST /api/rules` with `{ "title", "summary?", "document" }`. **Step 9.** **Templates** (when ready): seed `RuleTemplate` rows; `GET /api/templates` is implemented. **Step 10.** **Frontend draft sync:** Set `NEXT_PUBLIC_ENABLE_BACKEND_SYNC=true` in `.env` so **Save & Exit** and **post-login anonymous → account transfer** can **PUT** `/api/drafts/me`. Without sync, drafts are **not** written to the server (anonymous progress still lives in `localStorage` only). **Step 11.** **Web vitals:** Move off `.next` files—**prefer an external analytics or logging pipeline** (see §7). Use Postgres for vitals only as a deliberate ops choice. --- ## 10. Security checklist - **HTTPS** in staging/production; session cookie **Secure**. - **Rate-limit** magic-link **request** — in-memory OK for one instance; **shared store before multi-instance** (see §5). - **Hash** magic-link tokens and session tokens before storing; short **magic-link** TTL (align with implementation, e.g. 15 minutes). - **Secrets** only in env / secret store — never commit `.env` with real values. - **CORS:** prefer **same-origin** `/api/*`; if cross-origin, configure CORS and CSRF carefully. --- ## 11. Environments | Environment | Purpose | Notes | | -------------- | ------------------------------- | --------------------------------------------------------------------- | | **Local** | Daily development | Docker Compose: Postgres + optional Mailhog (`docker compose up -d`). | | **Staging** | Rehearse deploys and migrations | Match prod as closely as possible; test SMTP. | | **Production** | Users | Backups, monitoring, migration job before traffic. | **Optional QA:** Run automated tests against an **ephemeral** database in CI instead of maintaining a fourth long-lived server. **Admin / infra (coordinate with whoever runs the server):** 1. TLS certificates and hostnames. 2. PostgreSQL backups and restore drill. 3. SMTP DNS (SPF, DKIM). 4. Health check URL for reverse proxy (`/api/health`). 5. Log retention and alerts for 5xx errors. --- ## 12. Frontend hook-up **Step 1.** **Anonymous** create flow: in-progress state is stored in **`create-flow-anonymous`** (`localStorage`). **Signed-in** “Create rule” does **not** auto-load server drafts yet (profile “open draft” is future). **Step 2.** Set `NEXT_PUBLIC_ENABLE_BACKEND_SYNC=true` to enable **PUT** on **Save & Exit** and after **magic-link transfer** from the save-progress exit modal. **Step 3.** Sign-in: **Log in** in the header opens the **modal** ([`AuthModalProvider`](app/contexts/AuthModalContext.tsx)); **`/login`** is still used for verify **error** redirects and bookmarks. Flow: request magic link → open verify URL → session cookie → `GET /api/auth/session` / `/api/drafts/me` as needed. **Step 4.** On publish, call `POST /api/rules` from the completed step when the backend is required (wire when the final review UI is ready). **Step 5.** **Profile / dashboard** (`/profile` or agreed path): signed-in hub for **my rules** (after Ticket 15 APIs exist), **duplicate** / **delete** rule actions, **logout**, **delete account**—aligned with [Figma profile](https://www.figma.com/design/agv0VBLiBlcnSAaiAORgPR/Community-Rule-System?node-id=22143-900069). **Change email** in design is **deferred** (hide, “coming soon,” or backlog) until a future account ticket; greeting copy can stay **static** or use **email local-part in UI only**—no `displayName` field required for MVP. **Step 6.** **Templates:** **Tickets 7–8** — seed `RuleTemplate` and load **`GET /api/templates`** in home / create surfaces (flat list, optional `featured`). **Ticket 16 / [CR-88](https://linear.app/community-rule/issue/CR-88/backend-template-recommendation-matrix-xlsx-sheets-ingestion)** — add **facet-based recommendations** and **spreadsheet ingestion** when product is ready (matrix rows + dimension columns like the decision-making workbook). --- ## 13. Optional later - **Template recommendation matrix** + `.xlsx` / Sheets import pipeline — see **Ticket 16** / **[CR-88](https://linear.app/community-rule/issue/CR-88/backend-template-recommendation-matrix-xlsx-sheets-ingestion)** (also §4 `RuleTemplate` note); not bundled into v1 template list work. - **Session library** spike (Auth.js, Lucia) if custom lifecycle cost grows. - **Redis** (or similar) for **shared magic-link rate limits** and horizontal scale. - **RuleDraft** versioning or multiple drafts per user. - Standalone **API service** (Fastify/Hono) if scaling or workers demand it. - **OpenAPI** if external API clients appear. - **Fourth environment** or stricter rate limiting at the edge. --- ## 14. Useful commands | Command | When | | --------------------------------------- | ------------------------------------------------------------------------------------------ | | `npx prisma studio` | Inspect/edit DB locally. | | `npx prisma migrate dev` | After changing `schema.prisma` in development. | | `npx prisma migrate deploy` | Apply migrations in staging/production. | | `docker compose up -d postgres mailhog` | Local DB + mail UI (http://localhost:8025). | | `docker build -t community-rule .` | Optional production image (Next **standalone** + `node server.js`; see repo `Dockerfile`). | --- ## External reading - [Docker: development best practices](https://docs.docker.com/develop/dev-best-practices/) - [Docker Compose in production](https://docs.docker.com/compose/how-tos/production/)