From ba12798381c51efbd779d4801bdf218e72ccc763 Mon Sep 17 00:00:00 2001 From: NGPixel Date: Thu, 1 Oct 2026 18:48:02 -0400 Subject: [PATCH] docs: add notifications spec doc [skip ci] --- dev/specs/notifications.md | 888 +++++++++++++++++++++++++++++++++++++ 1 file changed, 888 insertions(+) create mode 100644 dev/specs/notifications.md diff --git a/dev/specs/notifications.md b/dev/specs/notifications.md new file mode 100644 index 000000000..0293f9138 --- /dev/null +++ b/dev/specs/notifications.md @@ -0,0 +1,888 @@ +# Notifications + +**Status:** a proposal. Nothing here is implemented yet. What already exists and is built on (page +watching, the inbox shell, the disabled Profile entry) is listed in [§2](#2-what-exists-today). Table, +column, route and file names are proposals until the first phase lands. +**Covers:** what a user is notified about, how they choose which notifications they get and how, +how an event becomes an inbox entry and an email without slowing the request that caused it, and +how new kinds of notification are added later. + +A wiki already tells you a great deal if you go looking for it: the watch list, the review queue, +the Talk tab. What it does not do is *come and find you*. This document is about that half: being +told that a page you watch has changed, that someone answered you, that a suggestion is waiting for +your review. + +The constraints this is written against: + +- **A notification never tells anyone more than they could have found out for themselves.** Every + recipient is checked against the permission that would let them see the thing they are told + about, *at the time they are told*. +- **The request that caused an event does one write, and nothing else.** Working out who to tell, + checking each person's access, and talking to an SMTP server all happen in worker threads, + however many people end up being notified. +- **A new kind of notification is one file, one emit call and its strings.** The admin-facing ones + that are coming (registrations, failed logins, a failed git sync) must not need a second system. + +--- + +## 1. Goals and non-goals + +**Goals** + +- **Seven categories at launch** ([§4.2](#42-the-launch-categories)), five on by default and two + opt-in. +- **Per-user, per-category choice of channel**: Email, In-App, both or neither, from **Profile → + Notifications**. One set of preferences per user, not one per site. +- **Scales with the instance**: the cost of an event is roughly proportional to the number of + people who actually receive it, not to the number of accounts on the wiki. +- **Coalescing**: a page saved forty times in an afternoon does not produce forty inbox entries. +- **RFC 8058 one-click unsubscribe** on every notification email. This is a hard requirement. +- **A per-site switch** that turns notifications off for a site. +- **Polling now, push later**, without the push work having to change anything polling relies on. + +**Non-goals** + +- **Live push in the first version.** [§9.3](#93-adding-push-later) says how it attaches. +- **Hourly and daily digest emails in the first version.** They are expected to follow, and + [§10.5](#105-leaving-room-for-hourly-and-daily-digests) is what the first version does to make + them additive. +- **Other channels** (browser Web Push, Slack DMs, Matrix). The preference model leaves room for + them ([§5](#5-preferences)), but none is planned. +- **Third-party comment providers.** Disqus, Giscus and the rest keep their discussions in somebody + else's service. Only the built-in provider produces comment events. +- **Notifying guests.** A notification needs an account to belong to. +- **Notifying anyone of their own actions.** The actor of an event is never one of its recipients. + +--- + +## 2. What exists today + +| Piece | Where | State | +| ----- | ----- | ----- | +| Page watching | `db/schema.ts` → `pageWatching`, `models/pageWatching.ts`, `api/watching.ts` | Done. One row per (page, user), carrying the site. The comments in that code already expect "whatever delivers the news later" to read it. **Rows cascade away when the page is deleted**, which matters for [§6.3](#63-deletion-snapshot-the-watchers-first) | +| Watch button and list | the bell in the page header, `pages/InboxWatching.vue` | Done | +| Inbox shell | `layouts/InboxLayout.vue`, `/_inbox/messages` → `pages/InboxMessages.vue` | Placeholder, reading "Nothing here yet" | +| Review deep link | `/_inbox/review/:submissionId?` | Done. Its route comment anticipates a notification linking to it | +| Profile entry | `layouts/ProfileLayout.vue`, key `notifications` | Present but `disabled: true`, and has no route | +| Header button | `components/HeaderNav.vue`, `mdi:inbox-full` → `/_inbox` | Present, with no badge | +| Strings | `locales/en.json` → `profile.notifications`, `inbox.*` | `inbox.title` already reads "Inbox & Notifications" | + +What it is built from: + +| Piece | Where | Used for | +| ----- | ----- | -------- | +| Webhook emit | `models/hooks.ts` → `emit()` | The contract to copy: called from the models, never throws, and does no delivery inline. Every `hooks.emit` call site is a candidate notification emit site | +| Scheduler and workers | `core/scheduler.ts`, `worker.ts`, `tasks/workers/` | Fan-out and mail both run as worker tasks. `dispatch-webhook.ts` is the model for a worker task that imports its own models | +| Page rules | `helpers/pageRules.ts` → `resolvePageRule`, `models/groups.ts` → `checkAccess` | Pure and in memory. The answer depends only on a user's *set of groups*, which is what makes bulk access checks cheap ([§7.2](#72-access-is-checked-per-group-set-not-per-user)) | +| Approval rules | `models/approvals.ts` → `matchesPage`, `reviewerGroups` | Who reviews a page | +| Mentions | `models/comments.ts` → `resolveMentions` | Turns `@handle` text into users with one query | +| Mail | `models/mail.ts`, `locales.translator()` | Templates, two bodies (HTML and text), locale fallback | +| Import scope | `storage.importingFrom()` (AsyncLocalStorage) | Tells an import or a git pull apart from a person's edit ([§6.2](#62-origin)) | +| Form bodies | `@fastify/formbody`, registered in `index.ts` | The one-click unsubscribe POST is `application/x-www-form-urlencoded` | + +--- + +## 3. Architecture + +``` + request path worker: fan-out worker: mail drain + ──────────── ─────────────── ────────────────── + notifications.emit() ──► notificationEvents ──► notifications ──► SMTP + one INSERT, never (outbox; claimed one row per recipient one mail per user + throws; a debounced with SKIP LOCKED) per coalescing key, with per drain, single + job kick in-app and email state or digest + │ + └──► GET /notifications/summary ◄── polling + (push attaches here later, §9.3) +``` + +Three tables and two worker tasks. This is deliberately **not** one scheduler job per recipient, +or even one per event. Every job costs a `jobs` row and a `jobHistory` row. On a busy wiki one job +per page save is thousands of rows a day before anyone has been notified, and one job per recipient +is that multiplied by the audience. Instead, the outbox is drained in batches, and the job is only +the thing that wakes the drain up. + +### 3.1 Where the code lives + +| Path | What | +| ---- | ---- | +| `backend/notifications/index.ts` | The category registry: `NOTIFICATION_CATEGORIES`, the `NotificationCategory` type, lookup by event | +| `backend/notifications/categories/.ts` | One file per category ([§4.1](#41-a-category-is-a-file)) | +| `backend/models/notifications.ts` | `emit()`, preferences, inbox queries, the unsubscribe token, settings | +| `backend/api/notifications.ts` | Inbox, summary, preferences, unsubscribe, admin settings | +| `backend/tasks/workers/dispatch-notifications.ts` | Fan-out ([§7](#7-fan-out)) | +| `backend/tasks/workers/send-notification-mail.ts` | Mail drain ([§10](#10-email)) | +| `backend/tasks/simple/purge-notifications.ts` | Retention ([§12.3](#123-purging)) | + +`notifications/` is a new top-level backend directory rather than a directory under `modules/`: +`refreshFromDisk` expects every directory there to hold a `definition.yml`. A category is code, not +an installable module, and there is nothing to discover from disk. + +--- + +## 4. Categories + +### 4.1 A category is a file + +```ts +export interface NotificationCategory { + key: string // 'watchedPage', also the preference key and the i18n key + events: readonly E[] // which emitted events it reacts to + scope: 'site' | 'instance' // see §11; every launch category is 'site' + origins: readonly EventOrigin[] // which origins it fires for, see §6.2 + defaults: { inApp: boolean; email: boolean } + priority: number // higher wins when one event reaches a user twice, §7.4 + visibleTo(actor: AccessActor): boolean // shown in Profile → Notifications? + recipients(event, ctx): AsyncIterable // candidate user IDs, in batches + accessCheck: string | null // page permission each recipient must hold NOW + groupKey(event): string // the coalescing key, §8.3 + variant(event): string // which message: 'edited', 'moved', 'deleted', ... + snapshot(event): Record // what the entry displays, §8.2 +} +``` + +`NOTIFICATION_CATEGORIES` is a closed `as const` list, the same pattern as `AUDIT_ACTIONS`, so +`npm run typecheck` refuses a key that does not exist. Each key is also its translation prefix +(`notifications.categories..*`), so adding a category means adding its strings too. + +`recipients` returns **candidates**. Removing the actor, checking access, applying preferences and +deduplicating are done once by the fan-out for every category ([§7](#7-fan-out)), so a category +cannot get any of them wrong. + +### 4.2 The launch categories + +| Key | Reacts to | Candidates | Must hold now | Default | Coalesces on | Origins | +| --- | --------- | ---------- | ------------- | ------- | ------------ | ------- | +| `watchedPage` | `page:edit`, `page:rename`, `page:delete` | the page's watchers | `read:pages` | in-app ✓ email ✓ | `watchedPage:` | all | +| `reviewRequested` | `submission:new`, `submission:update` | members of the reviewer groups of every enabled rule matching the page | none ([§4.3](#43-reviewers)) | ✓ ✓ | `review:` | user | +| `watchedPageComment` | `comment:new` | the page's watchers | `read:comments` | ✓ ✓ | `watchedComment:` | user | +| `commentReply` | `comment:new` with a parent | the author of the parent comment, if it has an account | `read:comments` | ✓ ✓ | `reply:` | user | +| `mention` | `comment:new`, `comment:edit` | users whose handle is written in the comment (on an edit, only handles that were not there before) | `read:comments` | ✓ ✓ | `mention:` | user | +| `pageCreated` | `page:create` | users who opted in | `read:pages` | ✗ ✗ | `pageCreated:` | user | +| `pageDeleted` | `page:delete` | users who opted in | `read:pages`, against the page as it was | ✗ ✗ | `pageDeleted:` | user | + +**`watchedPage` variants.** `edited`, `moved` (path or locale changed, which is what `page:rename` +is), `published`, `unpublished`, `scheduled` (the publish window was set or changed), and `deleted`. +A publish-state change is a variant of an edit, worked out by comparing the old and new +`publishState`, `publishStartDate` and `publishEndDate` in `updatePage`. A coalesced entry +([§8.3](#83-coalescing-and-idempotency-are-the-same-index)) records every variant it has absorbed, so +"edited 4 times and moved" can be said. + +**Email defaults apply only where email can be sent.** On an instance where `mail.isConfigured` +is false, every email preference reads as off and the Profile screen says why +([§5.2](#52-the-profile-screen)). + +### 4.3 Reviewers + +Only **members of the groups named in `reviewerGroups`** of the enabled rules that match the page +are notified. Holding `review:pages` at the page, or `manage:system`, does not make anyone a +recipient. On a large wiki, every administrator would otherwise receive every suggestion. An +administrator who wants these notifications is placed in a reviewer group like anyone else. (They +can still see and answer the whole queue, as `getReviewableSubmissions` allows today. Only the +notification is narrower.) + +No page permission is checked on top of that: the rule naming the group *is* the grant, and the +notification says no more than the reviewer's own queue already shows. The submitter is excluded, +like any actor. A guest's suggestion notifies just as a signed-in user's does. + +`submission:update` is the author revising a suggestion that is still open (there is one per author +per page, by the unique index). It bumps the existing unread entry rather than adding a second one. + +`approvals.matchesPage` has to be callable from a worker thread, so if it reads anything off `this` +beyond its arguments, it moves to a pure helper beside `helpers/pageRules.ts`. + +--- + +## 5. Preferences + +### 5.1 Storage + +``` +userNotificationPrefs + userId uuid → users.id, on delete cascade + category varchar(64) + channel varchar(16) -- 'inApp' | 'email' + enabled boolean + updatedAt timestamp + PRIMARY KEY (userId, category, channel) + INDEX (category, channel) WHERE enabled +``` + +**A row is stored only when it differs from the category's default.** No rows means the defaults, +and setting a choice back to its default deletes the row. That keeps the table small, and keeps the +defaults in the registry rather than copied into every account. + +**A table rather than a key in `users.prefs`**, because of the opt-in categories. To fan out +`pageCreated`, the question is "who has turned this on?" across every account on the instance. That +is an index lookup on the partial index above, rather than a scan of a JSONB column. The default-on +categories ask the other way round, "of these watchers, who has turned it off?", which joins the +already-small candidate list against the primary key. + +**One row per channel, not one column per channel**, so a third channel would be a new value +rather than a migration. + +Preferences are **global**: one set per user, covering every site they use. + +### 5.2 The Profile screen + +`pages/ProfileNotifications.vue`, at `/_profile/notifications` (enabling the entry +`ProfileLayout.vue` already has). One row per category the caller can see (`visibleTo`), each with +its title, a one-line description, and two toggles: **In-App** and **Email**. Grouped under +headings: *Pages you watch*, *Discussions*, *Reviews*, *Everything on the wiki*. The last group +holds the two opt-in categories, with a note that they can be busy. + +- With mail not configured, the Email column is disabled, with a hint that the administrator has not + set up outgoing mail. +- On a site where notifications are switched off ([§11](#11-the-site-switch)), a banner says so. + The preferences still apply on the user's other sites. +- A **Stop all email** action at the foot of the screen turns off email for every category in one + go. It is the same thing the unsubscribe page offers. + +`GET /_api/users/me/notifications` answers with every visible category, its effective +`{ inApp, email }`, its defaults, and `emailAvailable`. `PUT` takes the full map back, stores only +the rows that differ from the defaults, and audits `updateNotificationPrefs` (kind `profile`) with +the categories that changed. + +--- + +## 6. Emitting + +### 6.1 The call + +```ts +WIKI.models.notifications.emit('page:edit', { + siteId, actorId, pageId, + snapshot: { title, path, locale, tags, publishState, ... }, + variant: 'edited' +}) +``` + +It writes one `notificationEvents` row and asks for a fan-out run. Like `hooks.emit`, it **never +throws**: a notification problem must not fail the action that caused it. It returns at once, and +it is called only after the action has succeeded. + +It returns without writing anything when: + +- the site has notifications switched off ([§11](#11-the-site-switch)), checked against + `WIKI.sites`, which the request path has and a worker does not, or +- no category listens for this event from this origin. A git pull's `page:create` is dropped here + rather than written and then thrown away. + +**The kick is debounced per instance.** The first emit arms a timer of about one second, and when it +fires, a single `dispatchNotifications` job is added. A burst of 500 saves becomes a handful of jobs. +A `* * * * *` `SYSTEM_SCHEDULE` entry runs the same task as a safety net, so an instance that dies +with a timer armed loses at most a minute. + +Emit sites, beside the existing `hooks.emit` calls: + +| Event | Where | +| ----- | ----- | +| `page:create` | `pages.createPage`, and the restore path that re-creates a page | +| `page:edit` | `pages.updatePage`, including the save `approvals` makes when a suggestion is approved (the actor is the reviewer, and the snapshot carries who suggested it) | +| `page:rename` | `pages.movePage` | +| `page:delete` | `pages.deletePage`, `deletePagesByTag`, `tree.deleteFolder`, and git's removals in `applyIncoming`. Every path that removes a page row has to go through [§6.3](#63-deletion-snapshot-the-watchers-first) | +| `submission:new` / `submission:update` | where `approvals` writes a `pageEditSubmissions` row | +| `comment:new` / `comment:edit` | `api/comments.ts`, beside the webhook emits (comments are only ever created from a route) | + +### 6.2 Origin + +Every event carries an `origin`: `user`, `import` or `bulk`. + +- **`import`**: anything running inside `storage.importingFrom()`, which covers Import Everything + and a git pull's `applyIncoming`. `emit` reads the scope itself, so callers do not pass it. +- **`bulk`**: `deletePagesByTag` and `tree.deleteFolder`. These run their per-page work inside a + `notifications.bulk(work)` AsyncLocalStorage scope of the same shape. +- **`user`**: everything else. + +A category's `origins` decides whether it fires for an origin. **`watchedPage` fires for all +three**, so a watcher hears about a page a git pull rewrote, with the entry saying "via storage +sync" where there is no actor. **`pageCreated` and `pageDeleted` fire for `user` only**, so an +import of 5,000 pages, or a tag deletion, does not send 5,000 notifications to everyone who opted in. + +### 6.3 Deletion: snapshot the watchers first + +`pageWatching` rows are deleted with the page, so by the time a fan-out runs, a deleted page has no +watchers left to find. The `page:delete` emit therefore happens **before** the page row is deleted, +and captures the watchers in the same statement: + +```sql +INSERT INTO "notificationEvents" (…, "recipients") +VALUES (…, ARRAY(SELECT "userId" FROM "pageWatching" WHERE "pageId" = $1)) +``` + +The IDs never pass through Node, and the row is written whether or not the delete then succeeds. A +delete that fails after this point (rare, since the page row is the last thing to go) leaves an +event saying a page was deleted that was not. To avoid that, the fan-out checks that the page row +is really gone before it sends any `deleted` variant, and drops the event if the page is still there. + +Access for a deleted page is checked against the snapshot's path, locale and tags, the page as it +was. + +Restoring a deleted page does not bring its watchers back: the cascade has already removed them. +That is a property of the existing schema, noted here rather than changed. + +--- + +## 7. Fan-out + +`tasks/workers/dispatch-notifications.ts`, in a worker thread. + +### 7.1 Claiming + +It claims up to 50 unprocessed events, oldest first, with `FOR UPDATE SKIP LOCKED`, writing +`claimedAt` and `claimedBy`. Several instances can drain the outbox at once without two of them +taking the same event. A claim older than `scheduler.taskTimeout` plus a margin counts as abandoned +and can be claimed again, the same reasoning as `reapStaleJobs`. When the batch is done, the task +runs again straight away if more events are waiting, and stops otherwise. + +### 7.2 Access is checked per group set, not per user + +`checkAccess` depends only on the actor's groups (plus `manage:system`). Candidates are read +together with their memberships: + +```sql +SELECT u.id, array_agg(ug."groupId" ORDER BY ug."groupId") AS groups, u.prefs->>'locale' … +FROM users u JOIN "userGroups" ug ON ug."userId" = u.id +WHERE u.id = ANY($candidates) AND u."isActive" AND NOT u."isSystem" +GROUP BY u.id +``` + +The check is memoized on the joined group IDs. Ten thousand candidates spread over six group +combinations cost six rule evaluations. The worker has no `WIKI.models.groups`, so it reads +`groups.rules` and `groups.permissions` directly once per run and calls `resolvePageRule` itself. +What it answers is what `checkAccess` would answer, from the same rows. + +Access is checked **when the fan-out runs**, not when the event happened: someone whose access was +removed in between is not told. + +### 7.3 Candidates, in batches + +`recipients()` yields IDs in batches of 1,000, using keyset pagination on `userId`: + +- **Watchers**: `pageWatching` by `pageId`, or `notificationEvents.recipients` for a deletion. +- **Opted in**: `userNotificationPrefs` where `category = $1 AND enabled`, on its partial index. The + users table is never scanned. +- **Reviewers**: `userGroups` where `groupId = ANY($reviewerGroups)`. +- **Mentions and replies**: a handful of IDs, resolved from the comment. + +### 7.4 One entry per person per event + +A single `comment:new` can reach the same person three ways: they watch the page, they wrote the +parent comment, and they are mentioned in it. They get **one** entry, from the category with the +highest `priority` among those that would notify them on some channel: +`mention` (40) > `commentReply` (30) > `watchedPageComment` (10). Likewise `watchedPage` (20) beats +`pageDeleted` (5) for a watcher who also opted into deletions everywhere. + +The dedup runs after preferences have been applied: someone who turned mentions off but watches the +page still gets the watched-page entry. + +### 7.5 Writing + +Rows are written 500 at a time, as one multi-row `INSERT … ON CONFLICT … DO UPDATE` +([§8.3](#83-coalescing-and-idempotency-are-the-same-index)). Each row carries `inApp` from the +preferences, and `emailState = 'pending'` with `emailAfter` set ([§10.2](#102-cadence)) where email is +on, the account is verified and mail is configured. When anything has been written with email +pending, the mail drain is kicked. + +### 7.6 Long fan-outs + +The task checks `signal` between batches. When it is about to time out, it writes the keyset cursor +it has reached onto the event row (`cursor`) and leaves the event claimed but unprocessed. The next +run picks it up from the cursor. This is the same idea as `renderPages` handing the rest of its +queue to a fresh job, so a `pageCreated` going out to a very large opted-in audience is spread over +several runs instead of running into the timeout. + +--- + +## 8. Storage + +### 8.1 `notificationEvents` (the outbox) + +``` +id uuid PK -- the event ID, also the idempotency key +kind varchar(64) -- 'page:edit', 'comment:new', … +origin varchar(16) -- 'user' | 'import' | 'bulk' +siteId uuid null -- null for an instance-scoped event, §11 +actorId uuid null -- no FK: an event outliving its actor is fine +data jsonb -- snapshot, variant, IDs of the page / comment / submission +recipients uuid[] null -- pre-resolved candidates (deletions) +cursor jsonb null -- progress of an interrupted fan-out +claimedAt timestamp null +claimedBy varchar null +processedAt timestamp null +createdAt timestamp +INDEX (createdAt) WHERE processedAt IS NULL +``` + +Processed events are kept for a day, which is enough to debug a notification that did not arrive. +After that they are purged ([§12.3](#123-purging)). + +### 8.2 `notifications` (the inbox and the email queue) + +``` +id uuid PK +userId uuid → users.id, on delete cascade +siteId uuid null → sites.id, on delete cascade -- null: instance-scoped, §11 +category varchar(64) +variant varchar(32) -- the latest variant +groupKey varchar(255) +pageId uuid null → pages.id, on delete set null +commentId uuid null → comments.id, on delete set null +actorId uuid null → users.id, on delete set null +data jsonb -- snapshot: title, path, locale, actor name, excerpt, variants seen, … +count integer default 1 +lastEventId uuid +inApp boolean +emailState varchar(16) default 'none' -- none | pending | sent | failed | skipped +emailAfter timestamp null +emailedAt timestamp null +readAt timestamp null +createdAt timestamp +updatedAt timestamp -- bumped on coalescing, the inbox's sort key + +INDEX (userId, updatedAt DESC) -- the inbox +INDEX (userId) WHERE readAt IS NULL AND inApp -- the badge +INDEX (emailAfter) WHERE emailState = 'pending' -- the mail drain +UNIQUE (userId, groupKey) WHERE readAt IS NULL -- §8.3 +``` + +**The entry carries a snapshot**, because what it is about may be renamed, moved or deleted before +it is read. The link is built from `pageId` where the page still exists, so it follows a move, and +from the snapshot's path otherwise. + +**An excerpt is shown only while its comment exists.** `commentId` is `set null` when the comment +is deleted, and the API leaves the excerpt out of any entry whose `commentId` is null. A moderator +deleting spam therefore removes its text from every inbox it reached. An email that has already +been sent cannot be recalled. + +**One row covers both channels.** An email-only recipient still gets a row, with `inApp = false`, +and the inbox and the badge never show it. That keeps one source of truth, and gives coalescing +([§8.3](#83-coalescing-and-idempotency-are-the-same-index)) and the email cadence +([§10.2](#102-cadence)) one model to work on. An email-only row is closed (`readAt = emailedAt`) once +its email is sent, since there is no inbox in which it could be read. + +### 8.3 Coalescing and idempotency are the same index + +Every category gives each entry a `groupKey`, and a user has at most one **unread** entry per key. +The insert is: + +```sql +INSERT INTO notifications (…) VALUES (…) +ON CONFLICT ("userId", "groupKey") WHERE "readAt" IS NULL +DO UPDATE SET + count = notifications.count + 1, + variant = EXCLUDED.variant, + data = notifications.data || EXCLUDED.data, -- variants seen are unioned in code + actorId = EXCLUDED."actorId", + lastEventId = EXCLUDED."lastEventId", + updatedAt = now(), + emailState = , + emailAfter = +WHERE notifications."lastEventId" IS DISTINCT FROM EXCLUDED."lastEventId" +``` + +- **Coalescing**: forty saves to a page you have not looked at yet are one entry, "edited 40 + times, last by Bob", which moves to the top of the inbox each time. +- **Idempotency**: a fan-out retried after a crash replays the same event ID. The `WHERE` makes the + replay change nothing, so nobody's count is inflated and nobody gets a second row. +- **Reading resets it**: once the entry is read it no longer holds the unique slot, and the next + event starts a new one. + +The keys in [§4.2](#42-the-launch-categories) are chosen so that only things worth merging merge: +two comments on a watched page merge, but two mentions in two comments do not (`mention:`). + +--- + +## 9. In-app delivery + +### 9.1 API + +All routes are the caller's own rows: any logged-in session, no permission, and 401 for a guest. +No route declares `config.permissions`. Each comments `No route-level permissions:` and filters by +`userId` from the session. + +| Route | What | +| ----- | ---- | +| `GET /_api/sites/:siteId/notifications?cursor=&unread=` | Keyset-paginated on `(updatedAt, id)`, 30 per page. Includes the site's rows plus instance-scoped ones (`siteId IS NULL`) | +| `GET /_api/sites/:siteId/notifications/summary` | `{ unread, latestAt }`. `unread` is counted from `SELECT 1 … LIMIT 100` on the badge index and shown as "99+" above 99. Answers with an `ETag` built from the two values, so a poll that finds nothing new is a 304 | +| `PUT /_api/sites/:siteId/notifications/:id/read` | Marks one entry read | +| `PUT /_api/sites/:siteId/notifications/read` | Marks everything read, or `{ pageId }` for one page's entries (mark-read-on-view, [§10.2](#102-cadence)) | +| `DELETE /_api/sites/:siteId/notifications/:id` | Dismisses an entry | + +Marking read and dismissing are **not audited**. They are bookkeeping on one's own inbox, and a +click per notification would bury the audit log, for the same reason page views are not recorded. +Preference changes and the admin settings are audited. + +### 9.2 Polling + +A Pinia store, `stores/notifications.js`, holds `unread`, `latestAt` and the loaded page of entries, +and has one method that matters: `refresh()`, which calls `summary` and loads the list only if +`latestAt` moved. What calls `refresh()` is kept separate from the store itself: + +- app boot, once the session is known to be signed in; +- a route change, throttled to once per 15 s; +- `visibilitychange` to visible; +- an interval of 60 s, **only while the tab is visible**, so a hundred background tabs do not poll. + +The badge in `HeaderNav.vue` reads `unread`. `InboxMessages.vue` becomes the list: grouped by day, +unread entries marked, each entry's text produced **in the browser** from +`notifications.messages..` plus its snapshot, so an entry is read in the +language the interface is drawn in. Each entry links to its target: + +- a page, or the page's Talk tab with the comment anchored; +- `/_inbox/review/:submissionId` for a review; +- for a deleted page, the snapshot's path, which leads to the page-not-found screen and its restore + offer where the reader may restore. + +### 9.3 Adding push later + +Push is designed in now but not built. When it is added, it only ever says **"something changed, +call `refresh()`"**. The summary endpoint stays the single source of truth, so a pushed update and a +polled one leave the store in exactly the same state. Nothing about polling or the endpoint changes. + +When it is built: + +- every write that changes what a user's summary would answer (the fan-out's inserts, the read and + dismiss routes, the purge) sends `NOTIFY notifications` with the affected user IDs, through + `createNotifier` and chunked under Postgres's 8 kB payload limit; +- every instance `LISTEN`s and forwards to its own sockets on a `/_notifications` websocket, beside + the collab and terminal ones; +- the 60 s interval drops to a 5 minute safety net while the socket is open. + +No `NOTIFY` is sent in the polling version: with nothing listening it would be dead code. + +--- + +## 10. Email + +### 10.1 Templates + +`MailTemplateData` gains `notification` (one entry) and `notificationDigest` (several), with strings +under `mail.notification.*`. Both render through the existing `MailContent` → `htmlShell` / +`textBody` pair. The language is the recipient's `prefs.locale`, falling back to the site's primary +locale, which is what `localeFor` already does. + +What a notification email contains: the title, who did it, what it is about (page title and path), +for a comment the excerpt, one action button linking to the target, and a footer with **Manage +notifications** (linking to `/_profile/notifications`) and **Unsubscribe**. No page content is ever +included. + +**The mail model has to run in a worker.** `mail.ts` currently reads `WIKI.sites[siteId]` (hostname, +title, primary locale) and goes through `WIKI.models.locales`, and a worker has neither. `send()` is +changed to take a `MailSiteContext` (`{ baseUrl, title, primaryLocale }`) from its caller. The +existing callers build it from `WIKI.sites`; the drain builds it from the `sites` rows it loads +once per run. The translator is checked to work with the worker's lazily opened database, and +imported directly the way `dispatch-webhook.ts` imports `hooks`. + +### 10.2 Cadence + +**A window, then quiet until read.** An entry is emailed once, a short delay after it first +appears, and then not again until its recipient has read it: + +| | `emailState` / `emailAfter` | +| - | --------------------------- | +| **A new row** | `pending`, `emailAfter = now() + emailDelay` | +| **A coalesce while `pending`** | unchanged. The event merges into the email that is already due | +| **A coalesce after `sent`** | **stays `sent`.** The inbox entry keeps counting, but no further email is sent | +| **Read** | the entry gives up its unique slot ([§8.3](#83-coalescing-and-idempotency-are-the-same-index)), so the next event starts a new entry and a new email | + +`emailAfter` is the only thing the drain looks at. It groups everything due for one user into a +single mail: the `notification` template for one entry, `notificationDigest` for several. + +In practice: Alice watches *Runbook*, Bob saves it twelve times between 10:00 and 10:40, and at +10:05 Carol mentions Alice on another page. Alice receives two emails, "Runbook was edited" at about +10:03 and the mention at about 10:08, and nothing more about Runbook until she opens it. Her inbox +entry reads "edited 12 times". Sending an email per event would have meant thirteen of them, and a +window alone about ten, since Bob's pace keeps opening new windows. + +**This only works if entries get read in the ordinary course of things**, or it becomes "one email, +ever". So **opening a page marks that page's `watchedPage` entries read**, and opening its Talk tab +marks its `watchedPageComment` entries read. The page payload already answers `isWatching`, so it +also answers `hasUnreadNotifications`, and the browser calls `PUT …/notifications/read { pageId }` +only when that is true. A page view therefore writes nothing unless there was something to clear. + +**An email-only row closes when it is emailed** ([§8.2](#82-notifications-the-inbox-and-the-email-queue)), +because it has no inbox to be read in. Its recipient therefore gets one email per window rather than +one ever. + +**`emailAfter` is computed in exactly one place**, `emailAfterFor(user, now)` in the notifications +model, and both the fan-out insert and the coalesce go through it. Today it answers +`now + emailDelay`. It is the seam where hourly and daily digests attach later +([§10.5](#105-leaving-room-for-hourly-and-daily-digests)). + +### 10.3 The drain + +`tasks/workers/send-notification-mail.ts`: + +1. Claims up to `notifications.mailBatchSize` (default 100) **(user, site) pairs** with rows due, + using `SELECT DISTINCT "userId", "siteId" … WHERE "emailState" = 'pending' AND "emailAfter" <= + now() … FOR UPDATE SKIP LOCKED`. +2. For each pair, loads the due rows (capped at 50; above that, the digest says "and N more" and + links to the inbox), drops them as `skipped` if the site has notifications switched off, renders + one mail and sends it. **One mail per site, not one per user**, because a mail names the wiki it + comes from and its links use that site's hostname. A user active on two sites gets two digests, + each of which reads as coming from its own wiki. Instance-scoped entries (`siteId` null) go out + with the site the recipient last signed in on ([§11](#11-the-site-switch)). +3. Marks the rows `sent` with `emailedAt`, or `failed` after the scheduler's retries are used up. A + failure on one user does not stop the batch. +4. Runs itself again while due rows remain. The safety net is the same minute-by-minute schedule as + fan-out. + +Every send goes through the one cached transporter. Turning on nodemailer's `pool` option for this +task is worth measuring once it exists. + +Notification mails also carry `Auto-Submitted: auto-generated`, so out-of-office replies are not +sent back to the wiki. + +### 10.4 One-click unsubscribe (RFC 8058) + +This is a hard requirement. Every notification mail carries: + +``` +List-Unsubscribe: +List-Unsubscribe-Post: List-Unsubscribe=One-Click +``` + +**The token.** `base64url(payload) . base64url(HMAC-SHA256(payload))`, where the payload is +`{ v: 1, u: userId, c: [categories in this mail] }`. It is signed with +**`notifications.unsubscribeSecret`**, a 32-byte value seeded at install beside the other secrets in +`models/settings.ts`. It is deliberately not `auth.secret`: that one is rotated whenever an +administrator invalidates every session, which would break every unsubscribe link already sitting +in somebody's mailbox (the reason `apiKeys.ts` gives for keeping the certificate passphrase apart). +The token **does not expire**. A link in an email from last year still has to work, and the only +thing it can do is turn email off for one person. + +**`POST`** (the one-click): + +- `publicAccess`, no session, no cookie, no CSRF token. The mail client sends none of these. +- Takes `application/x-www-form-urlencoded` with `List-Unsubscribe=One-Click`. +- Verifies the HMAC with `timingSafeEqual`, then turns **email** off for the token's categories, + writing `enabled = false` rows. In-app is left alone: the user only asked to stop receiving mail. +- Idempotent, and answers 200 with no body. An invalid token gets 400, saying nothing about which + part failed. +- Recorded in the audit log as `unsubscribeNotifications` (kind `profile`), with the token's user as + the actor. Like the auth events in `models/users.ts`, this is an action with no session, whose + user is identified only by a credential. + +**`GET`** on the same URL **does nothing** but redirect to the frontend page `/_unsubscribe?t=`. +Mail scanners fetch every link in a message, so a GET that acted would unsubscribe people who never +asked. This is the same principle as the welcome mail's verify link. That page names the categories, +offers **Unsubscribe from these** and **Stop all notification email** (the same POST with +`scope=all`), and links to the Profile screen. The footer link in the mail body points to this page. + +**Deliverability checks, in the implementing PR:** + +- Gmail and Yahoo require the `List-Unsubscribe` headers to be covered by the DKIM signature. + Check which headers nodemailer's DKIM signing covers, and add these two if they are not among + them. +- Gmail honours one-click only for an `https` URL. A site served over plain `http` still gets the + header, which works as an ordinary link elsewhere. Admin → Notifications warns about it + ([§12.2](#122-admin--notifications)). + +### 10.5 Leaving room for hourly and daily digests + +Not built in the first version, but expected to follow: a user choosing **Immediate**, **Hourly** or +**Daily** delivery for their notification email. Nothing built now should have to be reshaped when +it comes. What the design already does for it, and what adding it involves: + +**What is in place from the start** + +- **The drain only looks at `emailAfter`.** It never asks why a row is due, so a digest is nothing + more than a later `emailAfter`. Claiming, grouping, rendering and marking rows sent stay the same. +- **`emailAfterFor` is the only place that computes it** ([§10.2](#102-cadence)). A schedule is a + change to that one function. +- **The digest template already exists.** `notificationDigest` takes any number of entries, + grouped by category, with the "and N more" cap. A daily digest is the same mail with more in it. +- **Quiet-until-read already holds back repeats.** An entry that went out in a digest stays `sent` + however many more events land on it, so the next digest does not repeat it. It reappears only if + it was read and a new entry was started. +- **The unsubscribe token already covers several categories** ([§10.4](#104-one-click-unsubscribe-rfc-8058)), + which a digest needs. +- **One mail per (user, site)** ([§10.3](#103-the-drain)) is the right unit for a digest as well. + +**What adding it involves** + +- **A per-user preference**, global like the rest ([§5](#5-preferences)): `emailSchedule` (`immediate` + | `hourly` | `daily`) and, for daily, the hour to send. It lives in `users.prefs` beside `timezone` + rather than in `userNotificationPrefs`. It belongs to the user, not to a category, and the fan-out + already reads each candidate's row for the locale ([§7.2](#72-access-is-checked-per-group-set-not-per-user)), + so it costs no extra query. It appears as one **Email delivery** selector above the grid on + Profile → Notifications. +- **`emailAfterFor` gains the schedule.** Hourly is the next top of the hour. Daily is the next + occurrence of the chosen hour in the user's `prefs.timezone`, falling back to the site's, then + UTC. That is computed with `Temporal.ZonedDateTime` so a daylight-saving change does not move the + slot by an hour, then converted back to an instant for the column. +- **Some categories skip the digest.** A category definition gains `digest: boolean`, defaulting to + true. A mention, a reply or a review request (and any later admin alert) is something a person is + waiting on, so the proposal is that those stay immediate even for a daily-digest user. The + implementer decides that list when the feature is built. +- **Changing the schedule reschedules what is pending.** Saving the preference recomputes + `emailAfter` for that user's `pending` rows, so switching from daily to immediate does not hold + back what has already queued. +- **Spreading the load.** Thousands of users on the default daily hour would all fall due in the + same minute. `emailAfterFor` therefore adds a fixed offset per user within the hour, derived from + the user ID, so the same user always lands at the same minute and the mail runs see a steady trickle + instead of one spike. `mailBatchSize` and the drain re-running itself while rows remain handle the + rest. +- **Retention.** A daily digest needs entries to survive at least a day before they are sent. + `retentionDays` is already far longer, but the Admin → Notifications screen should refuse a value + under 2 once digests exist. + +--- + +## 11. The site switch + +`features.notifications`, on by default, under **Admin → General → Features**, beside +`features.comments`. + +Turned off for a site: + +- `emit` writes nothing for that site's events ([§6.1](#61-the-call)). +- Pending emails for that site are marked `skipped` by the drain rather than sent. +- The inbox entry, the badge and the bell's "you will be notified" wording are hidden on that site. + The **Watching** list stays, since watching is still a list a user keeps. +- Existing entries are kept, not deleted. Turning the switch back on shows them again. +- Preferences are untouched, since they are global ([§5.2](#52-the-profile-screen)). + +**Instance-scoped categories ignore it.** None ships at launch, but the admin categories planned +([§13](#13-adding-a-category)) include some that belong to no site (a registration, since users are +per instance). Their entries have `siteId = null`, appear in every site's inbox, and their email +links use the site the recipient last signed in on, falling back to the first site. + +--- + +## 12. Administration and housekeeping + +### 12.1 Instance settings + +Settings an administrator sets once for the whole instance. They are not user preferences and not +the per-site switch ([§11](#11-the-site-switch)). + +| Key | Default | What | Where | +| --- | ------- | ---- | ----- | +| `notifications.retentionDays` | 90 | Entries older than this are purged, read or not | Settings blob, edited on Admin → Notifications | +| `notifications.emailDelay` | `3m` | The window of [§10.2](#102-cadence) | Same | +| `notifications.mailBatchSize` | 100 | Users per mail run. Raise it for a mail server that can take more, lower it for one that throttles. The API refuses anything outside 1–1000 | Same | +| `notifications.unsubscribeSecret` | generated at install | Signs unsubscribe tokens ([§10.4](#104-one-click-unsubscribe-rfc-8058)) | Settings blob, never returned by the API | + +### 12.2 Admin → Notifications + +`pages/AdminNotifications.vue` at `/_admin/notifications`, listed in the **System** section of +`AdminLayout.vue`'s sidebar. It goes in alphabetical order between Metrics and Rendering, behind +`manage:system` like its neighbours, with `fluent-topic-push-notification.svg` from `_assets/icons/` as +its icon. Its strings go under `admin.notifications.*`. + +`GET` / `PUT /_api/system/notifications`, behind `manage:system`. A save is audited (kind `admin`, +action `updateNotificationSettings`) with the fields that changed, as the other configuration +routes are. The screen has two cards: + +- **Settings**: retention, the email delay and the mail batch size. +- **Status**, read-only and answered by the same `GET`: + - the outbox backlog (unprocessed events and the age of the oldest one), which is what shows that + fan-out is falling behind; + - emails pending, sent and failed over the last 24 hours; + - whether outgoing mail is configured, linking to Admin → Mail when it is not; + - a warning when a site is served over plain `http`, where Gmail ignores one-click unsubscribe + ([§10.4](#104-one-click-unsubscribe-rfc-8058)). + +This is also where any later instance-wide setting goes, for example turning a category off for the +whole instance. + +### 12.3 Purging + +`tasks/simple/purge-notifications.ts` runs daily off `SYSTEM_SCHEDULE`. It deletes entries past +retention and processed outbox events older than a day, in batches of 10,000, checking `signal` +between batches. + +--- + +## 13. Adding a category + +Worked example: telling the people who look after storage that a git sync failed. + +1. **`notifications/categories/storageSyncFailed.ts`**: `scope: 'site'`, `origins: ['user', + 'import', 'bulk']` (the event comes from a scheduled job, which counts as whatever origin it ran + under, and all are fine here), defaults in-app ✓ email ✓, `accessCheck: null`, + `visibleTo: (actor) => actor.permissions.includes('manage:storage')`, + `recipients`: users in a group whose `permissions` contain `manage:storage`, and + `groupKey: storageSyncFailed:`, so a target failing every five minutes is one entry + until somebody looks at it. +2. Add its key to `NOTIFICATION_CATEGORIES`. +3. Add `WIKI.models.notifications.emit('storage:syncFailed', …)` in `storage.recordState`, **only + on a transition** from healthy to `error` rather than on every failure. +4. Add `notifications.categories.storageSyncFailed.*`, `notifications.messages.storageSyncFailed.*` + and its mail strings to `en.json`. + +Nothing else changes: the Profile screen, fan-out, dedup, coalescing, mail and unsubscribe all come +from the registry. + +**An event anyone can trigger must coalesce, or anyone can fill an administrator's mailbox.** A +failed-login category is the clear case: the login endpoint is open to the world, which is why +failed logins are not audited. It would need a `groupKey` per account, so repeated failures bump a +single entry, which the quiet-until-read cadence ([§10.2](#102-cadence)) then holds to a single email. + +--- + +## 14. Behaviour worth pinning down + +- **The actor is never a recipient**, including when they watch the page they just edited. +- **Inactive and system accounts are skipped**, and so is the guest account. **Unverified accounts + get in-app entries but no email**, because the address has not been confirmed. +- **A comment edited to add a mention** notifies only the handles that were not in the previous + version. Re-saving a comment does not mention everyone in it again. +- **Replies go to the parent's author only**, not to everyone else in the thread. Someone who + wants to follow a whole discussion watches the page. +- **A scheduled page going live at its `publishStartDate` is not an event.** Visibility is worked + out when the page is read, and no job flips anything at that time. Setting or changing the window + is a `scheduled` variant; the moment it takes effect is silent. +- **`pageCreated` fires on creation whatever the publish state**, with the state in the snapshot + ("created as a draft"). That matches what the recipient can see: any signed-in session sees + unpublished pages where it has `read:pages`. It does not fire again when the page is later + published. +- **Password-locked pages** notify like any other, by title and path, which is what the Watching + list already shows. No content is ever part of an entry. +- **A user deleted with entries pending** takes them with them (`on delete cascade`). + +--- + +## 15. Performance + +| Where | Cost | Why it holds | +| ----- | ---- | ------------ | +| The request | One `INSERT` (with a sub-select for a deletion) | No recipient is resolved, no access is checked, no SMTP | +| Jobs | About one `dispatchNotifications` per second per instance under load, and one per minute otherwise | Debounced kick ([§6.1](#61-the-call)) | +| Candidates | Proportional to the audience | Watchers, reviewer groups and mentions are indexed lookups. Opt-ins come from a partial index ([§5.1](#51-storage)). The users table is never scanned | +| Access | One rule evaluation per distinct group set | [§7.2](#72-access-is-checked-per-group-set-not-per-user) | +| Writes | Multi-row inserts of 500 | Coalescing keeps a busy page at one row per watcher | +| Badge | Index-only, capped at 100 rows, 304 when unchanged, no polling from hidden tabs | [§9.1](#91-api), [§9.2](#92-polling) | +| Mail | One mail per user and site per drain | Digesting is part of the drain, not an extra step | +| Timeouts | None to hit | Fan-out resumes from a cursor ([§7.6](#76-long-fan-outs)), and the purge works in batches | +| HA | Any instance drains | `SKIP LOCKED` on both queues. No instance-local state except the debounce timer | + +--- + +## 16. Suggested order + +1. **Foundation**: the three tables and their migration (`npm run db-generate -- + --name=notifications`), the registry with the seven categories, `userNotificationPrefs` with its + API, `ProfileNotifications.vue`, `features.notifications` in General → Features, and the + `notifications` settings blob with `unsubscribeSecret` seeded. +2. **In-app**: `emit()` with origins, the debounce and the emit sites, the deletion snapshot, + `dispatch-notifications.ts`, the inbox API, `stores/notifications.js` with polling, the + `HeaderNav` badge, and `InboxMessages.vue`. Usable on its own: an instance with no mail + configured is complete at this point. +3. **Email**: the `MailSiteContext` refactor, both templates and their strings, + `send-notification-mail.ts` with the quiet-until-read cadence and mark-read-on-view, the + unsubscribe token, routes and `/_unsubscribe` page, the DKIM check, and Admin → Notifications + ([§12.2](#122-admin--notifications)). +4. **Housekeeping**: `purge-notifications.ts` and its `SYSTEM_SCHEDULE` entries. +5. **Later**: push ([§9.3](#93-adding-push-later)), hourly and daily digests + ([§10.5](#105-leaving-room-for-hourly-and-daily-digests)), and the admin categories + ([§13](#13-adding-a-category)). + +Verification for each phase: `npm run typecheck` and `npx oxlint` in `backend/`, `npm run build` in +`frontend/`. Phases 2 and 3 are worth a throwaway instance. Mailpit is the mail server for +development: it receives every message without delivering any, which shows the digest, both bodies +and the `List-Unsubscribe` headers. Script two accounts, one watching a page the other edits ten +times, and check that one entry with a count of ten comes out. Then `curl` the one-click POST with +the token from the mail.