community-rule/docs/backend-roadmap.md

Backend roadmap (reference)

Temporary working notes for building the backend. Safe to delete once the stack is stable.

---

1. Where we are

• Next.js 16 single repo (package.json).
• PostgreSQL + Prisma: schema and migrations under prisma/; product APIs under app/api/* (health, auth/OTP, session, drafts, rules, templates, web-vitals).
• Server modules in lib/server/ (db, session, mail, rate limiting, etc.).
• Create flow persists in the browser (localStorage); optional server draft sync when NEXT_PUBLIC_ENABLE_BACKEND_SYNC=true and the user is signed in (app/create/context/CreateFlowBackendSync.tsx).
• Web vitals app/api/web-vitals/route.ts still use file-based storage under .next (not suitable for multi-instance production).
• CI: .gitea/workflows/ci.yaml (build, test, lint, prisma validate); no in-repo production deploy definition.

HTTP API (implemented in repo)

Mirrors CONTRIBUTING.md API routes table; handlers live under app/api/*/route.ts.

Method	Path	Purpose
GET	/api/health	Liveness / DB check
GET	/api/auth/session	Current user or null
POST	/api/auth/otp/request	Send email OTP
POST	/api/auth/otp/verify	Verify OTP, set session cookie
POST	/api/auth/logout	Clear session
GET / PUT	/api/drafts/me	Load or save create-flow JSON (authenticated)
GET / POST	/api/rules	List or publish rules
GET	/api/templates	List curated templates

Also present (not in CONTRIBUTING table): POST / GET /api/web-vitals — file-based store today; production path TBD (§7).

---

2. What we’re building

Step 1. Treat this as greenfield: new PostgreSQL database and new tables. Do not migrate data from the old Community Rule backend.

Step 2. Keep the backend inside this Next app:

• HTTP handlers under app/api/…
• Shared server code under lib/server/…

Step 3. Use the old backend only as a product hint (email OTP, saving rules, listing rules). Do not copy its Express layout or MySQL schema.

---

3. Stack choices

Step 1. Use PostgreSQL everywhere (local Docker, staging, production).

Step 2. Use Prisma — schema.prisma, npx prisma migrate dev / migrate deploy.

Step 3. Add SMTP (or Mailhog locally) for email OTP in deployed environments; dev can log OTP to the console when SMTP_URL is unset.

Step 4. Redis / queues / Kubernetes — not required for v1. Exception: before running multiple app instances, plan a shared rate-limit store (often Redis) for OTP endpoints; the current limiter is in-memory per process (lib/server/rateLimit.ts).

---

4. Data to model (first pass)

Plain-English entities (names can evolve):

Area	Purpose
User	Identified by email after OTP verification.
Session	Custom v1: HttpOnly cookie; opaque token; hash stored in DB (lib/server/session.ts). Not NextAuth/Lucia.
OtpChallenge	Short-lived email codes (hashed).
RuleDraft	One JSON blob per user (create-flow state). Schema already has updatedAt; no draft versioning or multiple named drafts in v1.
PublishedRule	Saved rule after publish (title, summary, document JSON).
RuleTemplate	Curated templates (slug, category, ordering).

Session follow-ups to implement or decide: token rotation on sensitive events, whether new login invalidates other sessions, and cleanup of expired Session rows (job or lazy delete). Revisit a small auth library (e.g. Auth.js, Lucia) only if maintaining custom code becomes costly.

RuleDraft future (not v1): versioning, multiple drafts per user, easier corruption recovery—only if product needs them.

Align JSON shapes with app/create/types.ts as it matures.

---

5. Session and authentication (v1)

• Decision: Custom database-backed sessions + email OTP; cookies are httpOnly; tokens are hashed at rest.
• OTP rate limiting: In-memory is acceptable for a single Node process. It does not coordinate across instances—add a shared limiter (e.g. Redis) before horizontal scaling or serious abuse exposure.
• Do not treat “switch to NextAuth/Lucia” as required for v1; document the custom lifecycle above instead.

---

6. Authorization (v1)

Match the current API behavior; tighten as product evolves:

• GET /api/drafts/me / PUT /api/drafts/me: Authenticated user only; draft is scoped to that user (userId).
• POST /api/rules: Authenticated user only; rule is stored with userId (owner).
• GET /api/rules: Public list of published rules (metadata: id, title, summary, timestamps)—no auth required today. Not a private “my rules” feed unless you add a separate route later.
• v1: No editing or deleting published rules via API in the shipped handlers; no sharing or collaborative ownership—treat each rule as owned by one user until product defines more.

---

7. API responses, errors, and observability

Error JSON (target): Prefer a stable shape, e.g. { "error": { "code": "string", "message": "string" }, "details"?: ... } for 4xx/5xx, instead of only { "error": "string" }. Validation errors can map into details. Implement gradually in route handlers.

Logging: Use the shared lib/logger.ts where possible. Include a request correlation id (reuse x-request-id if present, else generate) on API routes and log it with errors so support can tie logs together.

Metrics: No vendor required for v1; optional later: request duration, error counts.

Web vitals: Default is external RUM or log drain (e.g. host analytics, Vercel Analytics, OpenTelemetry, SaaS APM)—keep product Postgres focused on product entities. Storing vitals in Postgres is an explicit tradeoff only if ops strongly wants a single datastore.

---

8. Prisma migrations policy

• Never edit migration files that have already been applied to staging or production (or any shared database). Fixing schema drift = add a new migration.
• Local dev: prisma migrate dev creates migrations; deployed envs: prisma migrate deploy before serving new code that depends on the schema.

---

9. Build order (implementation steps)

Operator / local (always manual): Steps 1–4 — env file, Docker Postgres, npm ci, prisma migrate dev, npm run dev.

Backend behavior already in the repo: Steps 5–10 match implemented Route Handlers and middleware (lib/server/*). Step 11 (web vitals) is not production-ready (files under .next); treat as follow-up work aligned with §7.

Product / frontend still open (not only “backend exists”): Sign-in UI, wiring publish from the create flow, template seed + UI consumption — see §12 and docs/backend-linear-tickets.md.

---

Step 1. Copy .env.example to .env.local. Set DATABASE_URL and secrets (see file comments).

Step 2. Start Postgres locally:

docker compose up -d postgres

Step 3. Install dependencies and apply migrations:

npm ci
npx prisma migrate dev

Step 4. Run the app:

npm run dev

Step 5. Confirm health: GET /api/health should return JSON.

Step 6. OTP login (happy path):

1. POST /api/auth/otp/request with { "email": "you@example.com" }
2. Read the code from your mail catcher or server logs (dev).
3. POST /api/auth/otp/verify with { "email": "...", "code": "..." }
4. GET /api/auth/session should show your user.

Step 7. Drafts: With a session, GET /api/drafts/me and PUT /api/drafts/me with { "payload": { ... } } (create flow state object).

Step 8. Publish: POST /api/rules with { "title", "summary?", "document" }.

Step 9. Templates (when ready): seed RuleTemplate rows; GET /api/templates is implemented.

Step 10. Frontend sync: Set NEXT_PUBLIC_ENABLE_BACKEND_SYNC=true in .env.local for server drafts when logged in; localStorage remains fallback when off or anonymous.

Step 11. Web vitals: Move off .next files—prefer an external analytics or logging pipeline (see §7). Use Postgres for vitals only as a deliberate ops choice.

---

10. Security checklist

• HTTPS in staging/production; session cookie Secure.
• Rate-limit OTP (in-memory OK for one instance; shared store before multi-instance—see §5).
• Hash OTP codes and session tokens before storing; short OTP expiry.
• Secrets only in env / secret store — never commit .env.local.
• CORS: prefer same-origin /api/*; if cross-origin, configure CORS and CSRF carefully.

---

11. Environments

Environment	Purpose	Notes
Local	Daily development	Docker Compose: Postgres + optional Mailhog (docker compose up -d).
Staging	Rehearse deploys and migrations	Match prod as closely as possible; test SMTP.
Production	Users	Backups, monitoring, migration job before traffic.

Optional QA: Run automated tests against an ephemeral database in CI instead of maintaining a fourth long-lived server.

Admin / infra (coordinate with whoever runs the server):

1. TLS certificates and hostnames.
2. PostgreSQL backups and restore drill.
3. SMTP DNS (SPF, DKIM).
4. Health check URL for reverse proxy (/api/health).
5. Log retention and alerts for 5xx errors.

---

12. Frontend hook-up

Step 1. Keep default behavior: no env flag → create flow uses only localStorage (current behavior).

Step 2. Set NEXT_PUBLIC_ENABLE_BACKEND_SYNC=true to opt in to server drafts when logged in.

Step 3. Implement sign-in UI when you are ready: call the OTP routes, then rely on the browser cookie for /api/drafts/me.

Step 4. On publish, call POST /api/rules from the completed step when the backend is required (wire when the final review UI is ready).

---

13. Optional later

• Session library spike (Auth.js, Lucia) if custom lifecycle cost grows.
• Redis (or similar) for shared OTP rate limits and horizontal scale.
• RuleDraft versioning or multiple drafts per user.
• Standalone API service (Fastify/Hono) if scaling or workers demand it.
• OpenAPI if external API clients appear.
• Fourth environment or stricter rate limiting at the edge.

---

14. Useful commands

Command	When
npx prisma studio	Inspect/edit DB locally.
npx prisma migrate dev	After changing schema.prisma in development.
npx prisma migrate deploy	Apply migrations in staging/production.
docker compose up -d postgres mailhog	Local DB + mail UI (http://localhost:8025).
docker build -t community-rule .	Optional production image (Next standalone + node server.js; see repo Dockerfile).

---

External reading

• Docker: development best practices
• Docker Compose in production