diff --git a/CLAUDE.md b/CLAUDE.md
index 55d1960d0..c955df077 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -86,8 +86,10 @@ path in silence.
`WIKI` global (config + logger + lazy `ensureDb()`) and dynamically imports the task.
- `base.yml` — system defaults for every config key. Do not edit as a user-facing config; it defines
the shape merged with `config.yml` and the db `settings` table.
-- `helpers/` — small pure utilities (`common.ts`, `config.ts`), plus `storageFiles.ts`, which is the
- file-tree half of the storage modules that address content by path (see [Storage targets](#storage-targets)).
+- `helpers/` — small pure utilities (`common.ts`, `config.ts`), plus two that are not: `storageFiles.ts`,
+ the file-tree half of the storage modules that address content by path (see
+ [Storage targets](#storage-targets)), and `appShell.ts`, which describes a page in the HTML document
+ served for it (see [The app shell and SEO](#the-app-shell-and-seo)).
- `types/` — ambient declarations: `global.d.ts` (the `WIKI` global) and `fastify.d.ts` (session +
route-permission augmentations).
- `locales/` — `en.json` source strings (CrowdIn-managed) + `metadata.js` language table (the one
@@ -785,6 +787,89 @@ store; no SVG is ever written into content.
- Picking an icon calls `POST /_api/icons/materialize`, which is what guarantees the wiki can serve it
afterwards without the Iconify API.
+### The app shell and SEO
+
+The compiled SPA is one document for every path on the wiki, served by the `setNotFoundHandler` in
+`index.ts` — which is the fallback rather than a route because a page lives at any path a user cares
+to give it, and the frontend's router is what resolves one.
+
+That document says nothing about the page at its URL until a browser has run it — a `` reading
+`Wiki.js` for every page of every wiki — and **plenty of clients never run it**: a chat client
+building an unfurl card, an AI crawler (GPTBot, ClaudeBot, PerplexityBot and friends fetch raw HTML
+and render nothing), a search engine that does not render, a reader with JavaScript off.
+`helpers/appShell.ts` is what puts something in the document, and `renderAppShell` is the whole of
+the decision.
+
+**There is no server-side rendering here, and none is wanted.** A page's HTML is already a string in
+`pages.render`, produced once in the editor's browser at save time — so describing a page is reading a
+row and two string insertions, not running Vue, a component tree or a second build. Anything that
+proposes rendering on the server is solving a problem this schema does not have.
+
+**Every request gets a head describing the page at its URL.** What separates the two kinds of
+document is whether the client will run the app, and each has its own function:
+
+- **`fragmentsForCrawler`** — for a client that will not. The PUBLIC's view of the page
+ (`pages.describePageForPublic`, which is `listForSitemap`'s question asked of one page): a head, the
+ page's markup appended, and 404 where the public may read nothing there. What goes in is what the
+ GUESTS group may read and nothing else, which is what makes it the same document for whoever asked
+ — and therefore the only half that is cached and handed on.
+- **`fragmentsForBrowser`** — for a browser that will. The head alone, describing the page as THAT
+ requester may see it (`pages.describePageForRequest`), which makes exactly the cut the page route
+ makes: `read:pages` per path, and any signed-in session sees an unpublished page while an API key is
+ answered as the public is. Never cached, because the answer is one reader's. It carries no body (the
+ app is about to draw the page properly, so a copy is bytes on every hard navigation for content the
+ browser discards), never 404s (whether a path this reader may create is empty is the app's own flow
+ to present) and adds no page-specific `noindex`.
+
+ It matters even though the app sets the title itself a moment later: the document's own title is
+ what the tab reads while the bundle loads, and what a bookmark or a history entry made before boot
+ finishes keeps for ever.
+- **The head is columns, not prose**: `<title>` composed exactly as `MainLayout.vue` composes it so
+ the tab does not jump when the app boots, `description`, a canonical link, `hreflang` alternates
+ from the page's locale group, and the `og:`/`twitter:` pair — which is what a link pasted into
+ Slack or Discord reads, and the surface this was most visibly missing.
+- **The body is the stored render with everything that would *run* taken out** — `<script>`,
+ `<style>`, inline handlers (`stripActiveMarkup`). A page whose author holds `write:scripts` keeps
+ those in its render and the app runs them when it draws the page; a copy of the same markup in the
+ document as the browser parses it would run them a second time, before the app exists. Not a
+ sanitizer — the render was sanitized at save time — just the same markup with less in it.
+- **It lands in `#wiki-prerender`, which both sides know about.** `frontend/index.html` hides it and
+ restyles it inside `<noscript>`, so a reader with JavaScript off gets the page as prose; `main.js`
+ removes the element before Vue mounts. The `<noscript>` typography puts back the browser defaults
+ Tailwind's preflight took away and is deliberately not a copy of the content styles, which belong
+ to elements the app draws.
+- **An anonymous request to a page path with nothing public at it answers 404**, where the shell used
+ to answer 200 everywhere — which is what taught a crawler that every URL on the wiki exists. No
+ page, an unpublished one and one the guests group's rules refuse are one answer, since the
+ difference is not something to tell whoever is asking. **The site root is the exception and always
+ answers 200**: there is something to show there whatever the database says — the welcome screen of a
+ wiki with no home page yet, a login form on a wiki that is not public — and a root answering 404
+ would report a working instance as broken to every uptime check pointed at it. Note the consequence
+ on a **private wiki**, where the guests group denies everything by default: every page path answers
+ 404 to an anonymous client, which is the truthful answer and matches the empty sitemap such a site
+ already serves.
+- **Indexing is one header and never a tag.** `robotsTagFor` folds the site's **General → SEO**
+ settings together with the document's own say — a page marked not searchable, a path with nothing
+ public at it, and every URL that is not a page at all (`/_admin`, `/_search`: an interface, not
+ content) are all `noindex`. `X-Robots-Tag` rather than `<meta name="robots">` because it reaches
+ the same clients and cannot fall out of step with itself.
+- **The fragments are cached and the shell is not.** `WIKI.cache` holds the head and body per
+ origin-and-path for ten minutes — the sitemap's figure, for the sitemap's reason — while the shell
+ is re-read per request so that `npm run build` in `frontend/` takes effect immediately. The key
+ carries the request's own host, because a canonical link does; hence the crude
+ `SHELL_CACHE_MAX_ENTRIES` ceiling, without which anybody could grow the cache by inventing
+ hostnames. `invalidateAppShellCache()` is called from every page mutation and from the two places
+ `invalidateSitemaps()` is. Nothing a particular requester holds ever reaches it: the key has no
+ session dimension because the only thing cached has no requester in it.
+- **`groups.actorForPublic()` is the public as an actor**, and is deliberately separate from
+ `actorForRequest`: it is the guests group with no group-wide permissions, asked without a request in
+ hand. Both things that get cached and handed on — the sitemap and the public document — are built
+ from it, and neither may be built from anything one requester happens to hold.
+
+What already existed and is unchanged: `controllers/rootFiles.ts` serves `robots.txt` and
+`sitemap.xml` (with `hreflang` alternates), so **discovery** was never the missing half — the
+document was.
+
### Audit log
Every action a **person** takes is one row in `auditLog` — `userId`, `clientIP`, `ts`, `kind`
diff --git a/backend/controllers/rootFiles.ts b/backend/controllers/rootFiles.ts
index 201d05673..da6b22d21 100644
--- a/backend/controllers/rootFiles.ts
+++ b/backend/controllers/rootFiles.ts
@@ -1,6 +1,7 @@
import { chunk } from 'es-toolkit/array'
-import type { FastifyInstance, FastifyReply, FastifyRequest } from 'fastify'
+import type { FastifyInstance, FastifyReply } from 'fastify'
import type { SitemapPage } from '../models/pages.ts'
+import { originOf } from '../helpers/common.ts'
/**
* The sitemap protocol's own ceiling: 50,000 URLs, and 50MB uncompressed, per file. Past it the
@@ -29,19 +30,14 @@ function xmlEscape(value: string): string {
.replaceAll("'", ''')
}
-/**
- * The origin a `<loc>` — or robots.txt's `Sitemap:` line — is written against.
- *
- * The requester's own, and deliberately not the site's configured hostname: a sitemap may only list
- * URLs on the host it was itself fetched from — a crawler discards the rest as a cross-submission —
- * so the host in the request IS the answer, whether the site is bound to it or is the catch-all `*`.
- * It comes off a header and is therefore whatever the client said, which is why it is escaped before
- * it reaches the XML. It cannot carry a line break into robots.txt: a header value holding one is
- * rejected by the HTTP parser long before this.
- */
-function originOf(req: FastifyRequest): string {
- return `${req.protocol}://${req.host}`
-}
+/*
+ `originOf` (helpers/common.ts) is the origin every `<loc>` — and robots.txt's `Sitemap:` line — is
+ written against: a sitemap may only list URLs on the host it was itself fetched from, since a
+ crawler discards the rest as a cross-submission, so the host in the request IS the answer whether
+ the site is bound to it or is the catch-all `*`. It comes off a header and is therefore whatever the
+ client said, which is why it is escaped before it reaches the XML. It cannot carry a line break into
+ robots.txt: a header value holding one is rejected by the HTTP parser long before this.
+*/
/**
* Where a page is, absolute and ready to be written into an element.
diff --git a/backend/helpers/appShell.ts b/backend/helpers/appShell.ts
new file mode 100644
index 000000000..470a9c210
--- /dev/null
+++ b/backend/helpers/appShell.ts
@@ -0,0 +1,487 @@
+import * as cheerio from 'cheerio'
+import type { FastifyRequest } from 'fastify'
+import type { PageDescription } from '../models/pages.ts'
+import { htmlEscape, isPageUrl, normalizePagePath, originOf, splitLocalePath } from './common.ts'
+
+/**
+ * What the app shell is enriched with before it is handed to a client that will not run it.
+ *
+ * The compiled SPA is one document for every path on the wiki: a `<title>` reading `Wiki.js`, no
+ * description, and an empty `<div id="app">` that only means something once a browser has run the
+ * bundle. Anything that does not — a chat client building an unfurl card, an AI crawler, a search
+ * engine that does not render, a reader with JavaScript off — sees exactly that, for every page.
+ *
+ * This is the cheap half of the fix, and it is cheap because there is nothing to render: a page's
+ * HTML is already a string in `pages.render`, produced once in the editor at save time. So a document
+ * for a crawler is the shell plus that string plus a handful of meta tags, and the server never runs
+ * a renderer, a component tree or a second build to produce one. See the note on caching below for
+ * what a scrape actually costs.
+ *
+ * There are two kinds of document, and which one a request gets turns on whether it will run the app:
+ *
+ * - **A client that will not** — `fragmentsForCrawler` — gets the public's view of the page: a head
+ * describing it, its markup appended, and 404 where the public may read nothing there. What goes
+ * in is what the GUESTS group may read and nothing else, which is what makes it the same document
+ * for whoever asked, and therefore the half that is cached and handed on.
+ * - **A browser that will** — `fragmentsForBrowser` — gets the head alone, describing the page as
+ * THAT reader may see it. It still matters that the title is right: the document's own title is
+ * what the tab reads while the bundle loads and what a bookmark made before it finishes keeps.
+ *
+ * Two further rules hold across both:
+ *
+ * - **Nothing is rendered, and nothing runs.** The injected copy is the stored render with its
+ * scripts and styles taken out — see `stripActiveMarkup`.
+ * - **Nothing a requester holds reaches the cache.** Only the public half is cached, keyed by origin
+ * and path with no session dimension, because there is nothing in it that varies by requester.
+ */
+
+/** Namespaced so the whole lot can be dropped without knowing which hosts or paths are in it. */
+const SHELL_CACHE_PREFIX = 'appShell:'
+
+/**
+ * How long an assembled fragment set is held, in seconds.
+ *
+ * The same figure as the sitemap's, for the same reason: a title or a description a few minutes out
+ * of date misleads nobody, and the alternative is reading a page out of the database for every scrape
+ * of it. What waits for the TTL is an edit showing up in an unfurl card — the wiki itself never shows
+ * a stale page, since everyone who can edit one is logged in and is served the live app.
+ */
+const SHELL_CACHE_TTL = 600
+
+/**
+ * How many fragment sets are held at once.
+ *
+ * A crude ceiling rather than an eviction order: at the limit the whole namespace goes and fills
+ * again. Nothing here is worth the bookkeeping a least-recently-used cache would need — a wiki whose
+ * public traffic fits in this many pages gets a perfect hit rate, and one being crawled end to end
+ * gets little from caching either way, since a crawler fetches each page once.
+ *
+ * Having a ceiling at all is the point, and it is not about a wiki's size: the key carries the
+ * request's own host (see `cacheKeyFor`), so without one anybody could grow this without limit by
+ * asking for the same page under a made-up hostname.
+ */
+const SHELL_CACHE_MAX_ENTRIES = 500
+
+/** The site root, as the path below a locale prefix comes back — `/fr` alone is `/` in French. */
+const ROOT_PATHS = new Set(['', '/'])
+
+/** The page a site's root addresses. Mirrors `normalizePath` in the frontend's page store. */
+const HOME_PATH = 'home'
+
+/** The element the injected copy is wrapped in. `frontend/index.html` styles it; `main.js` removes it. */
+const PRERENDER_ID = 'wiki-prerender'
+
+/** What the shell is enriched with, before it is put into the document. */
+interface ShellFragments {
+ /** Replaces the shell's `<title>` and is appended to its `<head>`. */
+ head: string
+ /** Appended to the shell's `<body>`. Empty where there is no page body to show. */
+ body: string
+ /** 200, or 404 for a page path with nothing a reader without a session may read at it. */
+ status: number
+ /** The `X-Robots-Tag` to answer with, or null where a crawler's default is what the site wants. */
+ robots: string | null
+}
+
+export interface AppShellDocument {
+ html: string
+ status: number
+ robots: string | null
+}
+
+/**
+ * Drop every enriched fragment set.
+ *
+ * Called wherever something that shapes a public document changes: a page (its title, its body, its
+ * indexability, where it sits), a site's settings (its own title and description, whether it wants to
+ * be indexed, how it brackets URLs by locale) or a group's rules (what the public may read at all).
+ * The last of those is a permissions question rather than a freshness one, which is why it cannot be
+ * left to the TTL — the same reasoning as `invalidateSitemaps`, and the same call sites.
+ */
+export function invalidateAppShellCache(): void {
+ WIKI.cache.del(WIKI.cache.keys().filter((key) => key.startsWith(SHELL_CACHE_PREFIX)))
+}
+
+/**
+ * Whether this request will boot the app, or is something that will only ever read the document.
+ *
+ * A verified API key counts as a session for this, as it does everywhere else: a caller holding one
+ * has an identity, and enriching its document with the public's view of a page would be answering a
+ * different question from the one it asked.
+ */
+function isAnonymous(req: FastifyRequest): boolean {
+ return !req.session?.authenticated && !req.apiKey
+}
+
+/**
+ * The cache identity of a document.
+ *
+ * The host is part of it because the document is: a canonical link and an unfurl card carry absolute
+ * URLs, and those are built against the host the request arrived on (`originOf`). A site bound to the
+ * catch-all `*` answers on any number of hostnames, so two requests for the same page are not
+ * necessarily the same document. Hence also `SHELL_CACHE_MAX_ENTRIES`.
+ */
+function cacheKeyFor(origin: string, urlPath: string): string {
+ return `${SHELL_CACHE_PREFIX}${origin}${urlPath}`
+}
+
+/**
+ * What a site tells a crawler about a document it has already fetched, as an `X-Robots-Tag` value.
+ *
+ * The other half of the **General → SEO** settings, and the half that carries what robots.txt cannot:
+ * that file can only say whether to CRAWL a path (see `controllers/rootFiles.ts`), while `noindex`
+ * and `nofollow` are instructions about a document in hand. A search engine reads this header exactly
+ * as it reads a `<meta name="robots">` tag, and — unlike a tag the frontend would set once it booted —
+ * it is there for a crawler that does not run the page's JavaScript. It is the header and not a tag
+ * for that reason, and it is only the header so that the two can never disagree.
+ *
+ * `noindex` is also the setting's only thorough form. `Disallow: /` keeps a crawler off the page, but
+ * a page nobody fetched can still be listed from its inbound links alone; this is what says not to
+ * list it.
+ *
+ * @param indexable False for a document there is no sense in listing whatever the site says: a page
+ * marked as not searchable, one with nothing public at it, and every URL that is not a page at all —
+ * the app's own screens, which are an interface and not content.
+ *
+ * @returns Null when the site wants both and the document allows it, which is every crawler's default
+ * anyway: no header says the same thing as `index, follow`, and a wiki that wants to be found should
+ * not have to repeat it on every response.
+ */
+function robotsTagFor(siteId: string | undefined, indexable: boolean): string | null {
+ // -> A host matching no site at all is still handed the app shell, and is told not to index it:
+ // there is no site here whose settings could say otherwise
+ const robots = siteId ? WIKI.sites[siteId]?.config?.robots : undefined
+ const index = Boolean(robots?.index) && indexable
+ const follow = Boolean(robots?.follow)
+ if (index && follow) {
+ return null
+ }
+ return `${index ? 'index' : 'noindex'}, ${follow ? 'follow' : 'nofollow'}`
+}
+
+/**
+ * Which page a URL addresses, as the database holds it.
+ *
+ * The same reading the frontend router gives it, which is what keeps the injected document and the
+ * app that replaces it about the same page: `splitLocalePath` takes off a locale prefix if the first
+ * segment names one of the site's locales, and what is left is a page path. A path arriving without a
+ * prefix is in the site's primary locale — the request hook in `index.ts` is what redirects one that
+ * should have had a prefix, and it runs before this.
+ */
+function resolvePagePath(
+ siteId: string | undefined,
+ urlPath: string
+): { locale: string; path: string; isRoot: boolean } {
+ const locales = siteId ? WIKI.sites[siteId]?.config?.locales : undefined
+ const split = splitLocalePath(urlPath, WIKI.models.locales.urlPrefixesFor(locales?.active))
+ const below = split?.path ?? urlPath
+ return {
+ locale: split?.locale ?? locales?.primary ?? 'en',
+ // -> The site root is the `home` page rather than an empty path, the same as everywhere else
+ path: normalizePagePath(below) || HOME_PATH,
+ isRoot: ROOT_PATHS.has(below)
+ }
+}
+
+/**
+ * The stored render with everything that would *run* taken out.
+ *
+ * A page whose author holds `write:scripts` or `write:styles` keeps its `<script>` and `<style>` in
+ * the render, and the app runs them when it displays the page. The same markup sitting in the document
+ * as the browser parses it would run them a second time, before the app exists and outside anything
+ * that knows about the page — so what goes into the shell is the prose and nothing else. Inline
+ * handlers, `<template>` and `<link>` go for the same reason.
+ *
+ * Not a security boundary, and not a second sanitizer: the render was sanitized at save time against
+ * what its author was allowed to embed (`models/rendering.ts`), and this is that same markup with
+ * less in it. What this prevents is one page being executed twice.
+ */
+function stripActiveMarkup(html: string): string {
+ const $ = cheerio.load(html, null, false)
+ $('script, style, link, template').remove()
+ for (const el of $('*')) {
+ // -> `$('*')` only ever yields elements, but its element type is the union every node could be
+ for (const attr of Object.keys((el as { attribs?: Record<string, string> }).attribs ?? {})) {
+ if (attr.toLowerCase().startsWith('on')) {
+ $(el).removeAttr(attr)
+ }
+ }
+ }
+ return $.html()
+}
+
+/** One `<meta>`, or nothing at all for a value the site or the page has not filled in. */
+function metaTag(kind: 'name' | 'property', key: string, value: string | null | undefined): string {
+ return value ? `<meta ${kind}="${key}" content="${htmlEscape(value)}">` : ''
+}
+
+/**
+ * The absolute URL of a page, which is what a canonical link and an unfurl card name it by.
+ *
+ * `urlFor` decides the path, the same as for every link the wiki makes of its own pages and for every
+ * `<loc>` in the sitemap — so the two documents a crawler reads agree about where a page is, locale
+ * prefix and all. A page path is held to `[a-zA-Z0-9-_/]` when it is saved, but a locale's short code
+ * is an administrator's to alias, so the result is percent-encoded before it is escaped.
+ */
+function urlOf(origin: string, siteId: string, locale: string, path: string): string {
+ return `${origin}${encodeURI(WIKI.models.pages.urlFor(siteId, locale, path))}`
+}
+
+/**
+ * The head of a document describing one page.
+ *
+ * What a crawler and an unfurl card actually read, and all of it is a column: the title, the
+ * description, when the page last changed, and the locales it exists in. `og:` and `twitter:` are
+ * here because no chat client reads anything else — a link pasted into Slack or Discord is the most
+ * common way a wiki page is shared, and it is the surface this was most visibly missing.
+ *
+ * The title is composed exactly as `MainLayout.vue` composes it, so the tab does not change under a
+ * reader the moment the app boots.
+ *
+ * There is deliberately no `<meta name="robots">`: that is the `X-Robots-Tag` above, which reaches
+ * the same clients and cannot fall out of step with itself.
+ */
+function headForPage(
+ origin: string,
+ siteId: string,
+ siteConfig: any,
+ page: PageDescription
+): string {
+ const siteTitle: string = siteConfig?.title || 'Wiki.js'
+ const url = urlOf(origin, siteId, page.locale, page.path)
+ const description = page.description || siteConfig?.description || ''
+ // -> Only a logo an administrator actually put there: the route falls back to the Wiki.js mark for a
+ // site with none, and a card carrying that says less than a card carrying no image
+ const image =
+ siteConfig?.logoUrl || (siteConfig?.assets?.logo ? `${origin}/_site/current/logo` : '')
+
+ return [
+ `<title>${htmlEscape(`${page.title} - ${siteTitle}`)}`,
+ metaTag('name', 'description', description),
+ ``,
+ ...page.alternates.map(
+ (alt) =>
+ ``
+ ),
+ metaTag('property', 'og:type', 'article'),
+ metaTag('property', 'og:site_name', siteTitle),
+ metaTag('property', 'og:title', page.title),
+ metaTag('property', 'og:description', description),
+ metaTag('property', 'og:url', url),
+ metaTag('property', 'og:image', image),
+ metaTag(
+ 'property',
+ 'article:modified_time',
+ page.updatedAt.toTemporalInstant().toString({
+ smallestUnit: 'second'
+ })
+ ),
+ metaTag('name', 'twitter:card', image ? 'summary' : null)
+ ]
+ .filter(Boolean)
+ .join('\n ')
+}
+
+/**
+ * The head of a document that is not a page: the site itself, and nothing more specific.
+ *
+ * What a URL with no page at it gets, so that a 404 and a wiki whose home page has not been written
+ * yet still carry the site's own name rather than `Wiki.js`.
+ */
+function headForSite(siteConfig: any): string {
+ const siteTitle: string = siteConfig?.title || 'Wiki.js'
+ return [
+ `${htmlEscape(siteTitle)}`,
+ metaTag('name', 'description', siteConfig?.description),
+ metaTag('property', 'og:type', 'website'),
+ metaTag('property', 'og:site_name', siteTitle),
+ metaTag('property', 'og:title', siteTitle)
+ ]
+ .filter(Boolean)
+ .join('\n ')
+}
+
+/**
+ * The page's own markup, for a client that will not run the app.
+ *
+ * Wrapped in one element with a known id, which is what lets both sides deal with it: the stylesheet
+ * in `frontend/index.html` keeps it out of sight of anyone whose browser is about to draw the real
+ * thing and shows it to anyone whose browser is not, and `main.js` takes it out of the document
+ * before Vue mounts, so nothing the app does has to know it was ever there.
+ *
+ * The title is deliberately not repeated above the body as a heading. A page written the ordinary way
+ * opens with its own `# Title`, so a synthesized one is the same words twice — and where a page has
+ * no heading of its own, the title is still in `` and `og:title`, which is where every client
+ * that cares about it looks.
+ */
+function bodyForPage(page: PageDescription): string {
+ if (!page.render) {
+ return ''
+ }
+ return `<div id="${PRERENDER_ID}">${stripActiveMarkup(page.render)}</div>`
+}
+
+/**
+ * The fragments for a client that will never run the app — a crawler, an unfurl card, a reader with
+ * JavaScript off.
+ *
+ * The public's view of the page and nobody else's, which is what makes this the one that is cached
+ * and handed on: it is the same document whoever fetched it. It is also the only kind that carries a
+ * body and the only kind that ever answers 404.
+ */
+async function fragmentsForCrawler(
+ req: FastifyRequest,
+ siteId: string | undefined,
+ urlPath: string
+): Promise<ShellFragments> {
+ const siteConfig = siteId ? WIKI.sites[siteId]?.config : undefined
+
+ /*
+ Everything that is not a page path is the app's own interface — `/_admin`, `/_profile`, `/_edit`,
+ the search screen — and there is nothing in any of it for a search engine to list. Told so
+ explicitly rather than left to a crawler's judgement, and given no description of its own: these
+ screens have no content until the app has run, and a site that wants to be indexed should not have
+ its admin area competing with its pages for the result.
+ */
+ if (!siteId || !isPageUrl(urlPath)) {
+ return {
+ head: headForSite(siteConfig),
+ body: '',
+ status: 200,
+ robots: robotsTagFor(siteId, false)
+ }
+ }
+
+ const { locale, path, isRoot } = resolvePagePath(siteId, urlPath)
+ const page = await WIKI.models.pages.describePageForPublic(siteId, { locale, path })
+
+ if (!page) {
+ /*
+ Nothing here that the public may read — no page, an unpublished one, or one the guests group's
+ rules refuse. All three are one answer, and the answer is 404: the shell answering 200 for every
+ path is what tells a crawler that a mistyped URL is a page, and a wiki can be enumerated
+ indefinitely on the strength of it.
+
+ The site root is the exception and always answers 200. There is something to show there whatever
+ the database says — the welcome screen of a wiki whose home page has not been written yet, or a
+ login form on a wiki that is not public at all — and a root that answered 404 would report a
+ working instance as broken to every uptime check pointed at it.
+ */
+ return {
+ head: headForSite(siteConfig),
+ body: '',
+ status: isRoot ? 200 : 404,
+ robots: robotsTagFor(siteId, false)
+ }
+ }
+
+ return {
+ head: headForPage(originOf(req), siteId, siteConfig, page),
+ body: bodyForPage(page),
+ status: 200,
+ robots: robotsTagFor(siteId, page.isIndexable)
+ }
+}
+
+/**
+ * The fragments for a browser that is about to boot the app.
+ *
+ * The head alone, describing the page as THIS requester may see it — an unpublished page included, a
+ * page the public may not read included, and nothing they could not have opened themselves.
+ * `describePageForRequest` makes exactly the cut the page route makes, so the title in the document
+ * and the page the app then draws can never be about different things.
+ *
+ * It matters even though the app will set the title itself a moment later: the document's own title is
+ * what the tab reads while the bundle loads, what a bookmark or a history entry made before it
+ * finishes boot keeps for ever, and what a `Ctrl+D` on a slow connection saves. `Wiki.js` for every
+ * page of every wiki is simply wrong there.
+ *
+ * Three things it deliberately does not do. No body: the app is about to draw the page properly, so a
+ * copy is bytes on every hard navigation for content the browser will discard. No 404: the difference
+ * between a path with no page and one this reader may create is the app's own flow to present, and it
+ * needs the document to say 200. And no page-specific `noindex` — nothing will index a document
+ * fetched with a session, and a header invented here would be describing that session rather than the
+ * wiki.
+ *
+ * Never cached. The answer depends on who asked, which is exactly what the cached public document may
+ * not do; one read of one row per hard navigation is the price, and the app is about to fetch the same
+ * page over the API regardless.
+ */
+async function fragmentsForBrowser(
+ req: FastifyRequest,
+ siteId: string | undefined,
+ urlPath: string
+): Promise<ShellFragments> {
+ const siteConfig = siteId ? WIKI.sites[siteId]?.config : undefined
+ const robots = robotsTagFor(siteId, true)
+
+ if (!siteId || !isPageUrl(urlPath)) {
+ return { head: headForSite(siteConfig), body: '', status: 200, robots }
+ }
+
+ const { locale, path } = resolvePagePath(siteId, urlPath)
+ const page = await WIKI.models.pages.describePageForRequest(siteId, { locale, path }, req)
+
+ return {
+ head: page ? headForPage(originOf(req), siteId, siteConfig, page) : headForSite(siteConfig),
+ body: '',
+ status: 200,
+ robots
+ }
+}
+
+/**
+ * The document to answer a request for the app shell with.
+ *
+ * Every request gets a head describing the page at its URL; which page that is, and what else travels
+ * with it, is what `fragmentsForCrawler` and `fragmentsForBrowser` differ about.
+ *
+ * Only the public half is cached, and the shell is never cached: the shell is re-read per request so
+ * that `npm run build` in `frontend/` takes effect immediately, which a cached whole document would
+ * have delayed by the TTL, and the two string insertions that combine them are nothing next to a
+ * database read. Both insertions use a replacer function rather than a replacement string — a page
+ * containing `$&` would otherwise rewrite itself as it was inserted.
+ *
+ * @param shell The compiled `assets/index.html`, as read for this request
+ */
+export async function renderAppShell(
+ req: FastifyRequest,
+ siteId: string | undefined,
+ shell: string
+): Promise<AppShellDocument> {
+ const urlPath = req.raw.url!.split('?')[0]!
+ const fragments = isAnonymous(req)
+ ? await publicFragments(req, siteId, urlPath)
+ : await fragmentsForBrowser(req, siteId, urlPath)
+
+ /*
+ The head replaces the shell's own `<title>` where there is one and is appended to its `<head>`
+ where there is not, so that a shell built without one is enriched rather than silently skipped.
+ */
+ const withoutTitle = shell.replace(/[ \t]*<title>[\s\S]*?<\/title>\n?/i, '')
+ const html = withoutTitle
+ .replace('</head>', () => ` ${fragments.head}\n </head>`)
+ .replace('</body>', () => `${fragments.body}</body>`)
+
+ return { html, status: fragments.status, robots: fragments.robots }
+}
+
+/** The cached half: one URL's public fragments, keeping the namespace under its ceiling. */
+async function publicFragments(
+ req: FastifyRequest,
+ siteId: string | undefined,
+ urlPath: string
+): Promise<ShellFragments> {
+ const key = cacheKeyFor(originOf(req), urlPath)
+ const cached = WIKI.cache.get<ShellFragments>(key)
+ if (cached) {
+ return cached
+ }
+ const fragments = await fragmentsForCrawler(req, siteId, urlPath)
+ const held = WIKI.cache.keys().filter((k) => k.startsWith(SHELL_CACHE_PREFIX)).length
+ if (held >= SHELL_CACHE_MAX_ENTRIES) {
+ invalidateAppShellCache()
+ }
+ WIKI.cache.set(key, fragments, SHELL_CACHE_TTL)
+ return fragments
+}
diff --git a/backend/helpers/common.ts b/backend/helpers/common.ts
index 47193aae7..02c2ac88b 100644
--- a/backend/helpers/common.ts
+++ b/backend/helpers/common.ts
@@ -3,7 +3,7 @@ import { startCase } from 'es-toolkit/string'
import crypto from 'node:crypto'
import mime from 'mime'
import fs from 'node:fs'
-import type { FastifyReply } from 'fastify'
+import type { FastifyReply, FastifyRequest } from 'fastify'
export interface Deferred<T = void> {
resolve: (value: T) => void
@@ -73,6 +73,49 @@ export function createDeferred<T = void>(): Deferred<T> {
*/
export const RESERVED_ROOT_FILES = new Set(['favicon.ico', 'robots.txt', 'sitemap.xml'])
+/**
+ * Whether a URL addresses the page tree rather than the server itself.
+ *
+ * Everything the server mounts sits under a leading-underscore segment — `/_api`, `/_assets`,
+ * `/_files`, and the rest registered in `initHTTPServer` — which is what makes the distinction a
+ * prefix test rather than a list to keep in step with the routes.
+ *
+ * Note that the answer is about the URL and not about what is there: a page path with no page at it
+ * is still a page path, which is what lets the app shell answer 404 for one.
+ */
+export function isPageUrl(urlPath: string): boolean {
+ const firstSegment = urlPath.split('/')[1] ?? ''
+ return !firstSegment.startsWith('_') && !RESERVED_ROOT_FILES.has(firstSegment.toLowerCase())
+}
+
+/**
+ * The origin a URL this server writes into a document is built against.
+ *
+ * The requester's own, and deliberately not the site's configured hostname: a site may be bound to
+ * the catch-all `*` and have none, and every document this produces — a sitemap, a canonical link, an
+ * unfurl card — has to name the host it was itself fetched from or be discarded as pointing somewhere
+ * else. It comes off a header and is therefore whatever the client said, which is why everything
+ * built from it is escaped before it reaches a document.
+ */
+export function originOf(req: FastifyRequest): string {
+ return `${req.protocol}://${req.host}`
+}
+
+/**
+ * Escape a string for use as HTML text or inside a double-quoted attribute.
+ *
+ * Both at once, which is why the set is all five: an attribute needs the quotes and text needs the
+ * angle brackets, and a value that is safe in both is one fewer thing to get right per call site.
+ */
+export function htmlEscape(value: string): string {
+ return value
+ .replaceAll('&', '&')
+ .replaceAll('<', '<')
+ .replaceAll('>', '>')
+ .replaceAll('"', '"')
+ .replaceAll("'", ''')
+}
+
/**
* Decode a tree path
*
diff --git a/backend/index.ts b/backend/index.ts
index d606de9b8..4456b7adf 100644
--- a/backend/index.ts
+++ b/backend/index.ts
@@ -34,7 +34,13 @@ import configSvc from './core/config.ts'
import dbManager from './core/db.ts'
import logger from './core/logger.ts'
import scheduler from './core/scheduler.ts'
-import { RESERVED_ROOT_FILES, splitLocalePath, stripPageExtension } from './helpers/common.ts'
+import { renderAppShell } from './helpers/appShell.ts'
+import {
+ isPageUrl,
+ RESERVED_ROOT_FILES,
+ splitLocalePath,
+ stripPageExtension
+} from './helpers/common.ts'
import { corsOrigin, parseCspDirectives } from './helpers/security.ts'
const nanoid = customAlphabet('1234567890abcdef', 10)
@@ -85,65 +91,6 @@ function isServerUrl(urlPath: string): boolean {
return SERVER_ROUTE_SEGMENTS.has(segments[1] ?? '')
}
-/**
- * Whether a URL addresses the page tree rather than the server itself.
- *
- * Everything the server mounts sits under a leading-underscore segment — `/_api`, `/_assets`,
- * `/_files`, and the rest registered in `initHTTPServer` — which is what makes the distinction a
- * prefix test rather than a list to keep in step with the routes.
- */
-function isPageUrl(urlPath: string): boolean {
- const firstSegment = urlPath.split('/')[1] ?? ''
- return !firstSegment.startsWith('_') && !RESERVED_ROOT_FILES.has(firstSegment.toLowerCase())
-}
-
-/**
- * What a site tells a crawler about a document it has already fetched, as an `X-Robots-Tag` value.
- *
- * The other half of the **General → SEO** settings, and the half that carries what robots.txt cannot:
- * that file can only say whether to CRAWL a path (see `controllers/rootFiles.ts`), while `noindex`
- * and `nofollow` are instructions about a document in hand. A search engine reads this header exactly
- * as it reads a `<meta name="robots">` tag, and — unlike a tag the frontend would set once it booted —
- * it is there for a crawler that does not run the page's JavaScript.
- *
- * `noindex` is also the setting's only thorough form. `Disallow: /` keeps a crawler off the page, but
- * a page nobody fetched can still be listed from its inbound links alone; this is what says not to
- * list it.
- *
- * Null when both settings are on, which is every crawler's default anyway: no header says the same
- * thing as `index, follow`, and a wiki that wants to be found should not have to repeat it on every
- * response.
- */
-function robotsTagFor(siteId: string | undefined): string | null {
- // -> A host matching no site at all is still handed the app shell, and is told not to index it:
- // there is no site here whose settings could say otherwise
- const robots = siteId ? WIKI.sites[siteId]?.config?.robots : undefined
- if (robots?.index && robots.follow) {
- return null
- }
- return `${robots?.index ? 'index' : 'noindex'}, ${robots?.follow ? 'follow' : 'nofollow'}`
-}
-
-/**
- * The segments a site's locale-prefixed URLs may start with, mapped to the locale each names.
- *
- * Every code a locale answers to, not only the short one it is addressed by now: an alias an
- * administrator changed leaves the links people have already saved pointing at the old segment, and
- * a wiki that answers 404 to them has broken them. `localeForShortCode` is what knows the set.
- */
-function localePrefixesFor(activeCodes?: string[] | null): Map<string, string> {
- const prefixes = new Map<string, string>()
- for (const code of activeCodes ?? []) {
- const locale = WIKI.cache?.get(`locale:${code}`) as any
- for (const segment of [locale?.displayCode, locale?.derivedCode, code]) {
- if (segment) {
- prefixes.set(segment, code)
- }
- }
- }
- return prefixes
-}
-
if (!semver.satisfies(process.version, '>=26')) {
console.error('ERROR: Node.js 26.x or later required!')
process.exit(1)
@@ -733,7 +680,7 @@ async function initHTTPServer() {
*/
const siteLocales = WIKI.sites[siteId]?.config?.locales
if (siteLocales?.forcePrefix) {
- const prefixes = localePrefixesFor(siteLocales.active)
+ const prefixes = WIKI.models.locales.urlPrefixesFor(siteLocales.active)
if (!splitLocalePath(trimmed, prefixes)) {
const primary = WIKI.models.locales.shortCodeFor(siteLocales.primary)
reply.redirect(withQuery(`/${primary}${trimmed === '/' ? '' : trimmed}`), 302)
@@ -826,20 +773,9 @@ async function initHTTPServer() {
if (!isReadRequest || isSystemPath || isReservedRootFile) {
return reply.notFound()
}
+ let shell: string
try {
- const shell = await readFile(appShellPath, 'utf8')
- /*
- Every HTML document this wiki serves leaves through here — a page and an app route alike — so
- this is the one place the site's indexing settings can be attached to all of them.
-
- Straight off the site caches for the same reason the SEO hook above reads them that way: both
- lookups are what `getSiteByHostname` would do, minus its optional reload.
- */
- const robotsTag = robotsTagFor(WIKI.sitesMappings[req.hostname] || WIKI.sitesMappings['*'])
- if (robotsTag) {
- reply.header('X-Robots-Tag', robotsTag)
- }
- return reply.header('Cache-Control', 'no-store').type('text/html; charset=utf-8').send(shell)
+ shell = await readFile(appShellPath, 'utf8')
} catch (err: any) {
// -> Nothing to serve means the frontend was never built, which is a setup step rather than a
// fault of this request: say which one, since a bare 500 sends people looking in the server
@@ -849,6 +785,38 @@ async function initHTTPServer() {
.type('text/plain; charset=utf-8')
.send('The frontend has not been built yet. Run `npm run build` in frontend/.\n')
}
+
+ /*
+ Every HTML document this wiki serves leaves through here — a page and an app route alike — so this
+ is the one place the site's indexing settings can be attached to all of them, and the one place a
+ document can be made to say something about the page at its URL before the app has run.
+ `renderAppShell` is that: what it does, what it costs and who it does it for are all set out in
+ `helpers/appShell.ts`.
+
+ The site is read straight off the caches for the same reason the SEO hook above reads it that way:
+ both lookups are what `getSiteByHostname` would do, minus its optional reload.
+
+ A failure there is logged and the plain shell goes out instead. Enriching a document is for the
+ benefit of clients that will not run it, and a database that cannot answer for one is no reason to
+ stop serving the app to a reader whose browser would have rendered the page anyway.
+ */
+ const siteId = WIKI.sitesMappings[req.hostname] || WIKI.sitesMappings['*']
+ let doc = { html: shell, status: 200, robots: null as string | null }
+ try {
+ doc = await renderAppShell(req, siteId, shell)
+ } catch (err: any) {
+ WIKI.logger.warn(
+ `Cannot describe ${urlPath} for a client that will not render it: ${err.message}`
+ )
+ }
+ if (doc.robots) {
+ reply.header('X-Robots-Tag', doc.robots)
+ }
+ return reply
+ .code(doc.status)
+ .header('Cache-Control', 'no-store')
+ .type('text/html; charset=utf-8')
+ .send(doc.html)
})
// ----------------------------------------
diff --git a/backend/models/groups.ts b/backend/models/groups.ts
index 6387f6746..866da9a3f 100644
--- a/backend/models/groups.ts
+++ b/backend/models/groups.ts
@@ -1,6 +1,7 @@
import { v4 as uuid } from 'uuid'
import { and, count, eq, ilike, ne, or, sql } from 'drizzle-orm'
import { groups as groupsTable, userGroups, users as usersTable } from '../db/schema.ts'
+import { invalidateAppShellCache } from '../helpers/appShell.ts'
import { CustomError } from '../helpers/common.ts'
import { resolvePageRule, type RulePageRef } from '../helpers/pageRules.ts'
import type { SystemIds } from './types.ts'
@@ -158,11 +159,13 @@ class Groups {
for (const row of rows) {
rulesCache[row.id] = (row.rules ?? []) as GroupRule[]
}
- // -> The sitemap is a list of what the GUESTS group may read, held for minutes at a time. Every
- // other consumer of these rules asks per request and is correct the moment this returns; that
- // one would go on publishing paths a rule had just taken away, which is a permission waiting
- // for a timer rather than a document being a little out of date
+ // -> Both of these are what the GUESTS group may read, held for minutes at a time: an index of
+ // paths, and the page descriptions served to a client that will not run the app. Every other
+ // consumer of these rules asks per request and is correct the moment this returns; those two
+ // would go on publishing what a rule had just taken away, which is a permission waiting for a
+ // timer rather than a document being a little out of date
WIKI.models.pages.invalidateSitemaps()
+ invalidateAppShellCache()
WIKI.logger.info(`Loaded page rules for ${rows.length} groups [ OK ]`)
}
@@ -195,6 +198,23 @@ class Groups {
}
}
+ /**
+ * The actor the PUBLIC speaks for, which is not the same as a request that happens to be anonymous.
+ *
+ * The guests group and nothing else, asked without a request in hand — which is what "whatever
+ * anybody may read" means, and what makes an answer built from it identical for whoever fetched it.
+ * That is the whole reason it exists separately: the sitemap and the app shell's public document are
+ * both cached and handed to the next requester, so neither may be built from anything a particular
+ * requester holds.
+ *
+ * No group-wide permissions rather than the guests group's own: `checkAccess` reads that list only
+ * for `manage:system`, and a wiki that had somehow handed the public an administrator's permission
+ * should not also publish every page it has as a consequence.
+ */
+ actorForPublic(): AccessActor {
+ return { groupIds: [WIKI.data.systemIds.guestsGroupId], permissions: [] }
+ }
+
/**
* Whether this caller may do this to this page.
*
diff --git a/backend/models/locales.ts b/backend/models/locales.ts
index dafd9fd1a..813390603 100644
--- a/backend/models/locales.ts
+++ b/backend/models/locales.ts
@@ -469,6 +469,29 @@ class Locales {
return match?.code ?? segment
}
+ /**
+ * The segments a site's locale-prefixed URLs may start with, mapped to the locale each names.
+ *
+ * Every code a locale answers to, not only the short one it is addressed by now: an alias an
+ * administrator changed leaves the links people have already saved pointing at the old segment, and
+ * a wiki that answers 404 to them has broken them — the same reasoning as `localeForShortCode`,
+ * built as a map because the request hooks that read a page URL do it on every request.
+ *
+ * @param activeCodes The locales the site offers, i.e. `config.locales.active`
+ */
+ urlPrefixesFor(activeCodes?: string[] | null): Map<string, string> {
+ const prefixes = new Map<string, string>()
+ for (const code of activeCodes ?? []) {
+ const locale = WIKI.cache?.get(`locale:${code}`) as any
+ for (const segment of [locale?.displayCode, locale?.derivedCode, code]) {
+ if (segment) {
+ prefixes.set(segment, code)
+ }
+ }
+ }
+ return prefixes
+ }
+
async getLocales({ cache = true }: { cache?: boolean } = {}): Promise<any[]> {
if (!WIKI.cache.has('locales') || !cache) {
const locales = await WIKI.db
diff --git a/backend/models/pages.ts b/backend/models/pages.ts
index 6bb12bb68..63fd540e3 100644
--- a/backend/models/pages.ts
+++ b/backend/models/pages.ts
@@ -6,6 +6,9 @@ import {
normalizePagePath,
timingSafeCompare
} from '../helpers/common.ts'
+import { invalidateAppShellCache } from '../helpers/appShell.ts'
+import type { AccessActor } from './groups.ts'
+import type { FastifyRequest } from 'fastify'
import type { RenderPermissions, TocNode } from './rendering.ts'
import type { DeletedEntry } from './tree.ts'
import type { StoragePageContent, StoragePageRef } from './storage.ts'
@@ -266,6 +269,40 @@ export interface SitemapPage {
localeGroupId: string | null
}
+/**
+ * A page as the HTML document served for it describes itself — see `describePageForPublic` and
+ * `describePageForRequest`.
+ *
+ * Everything here is scoped to whoever it was asked for: a page they may not read is not described at
+ * all, and neither are the translations of one they may not reach.
+ */
+export interface PageDescription {
+ locale: string
+ path: string
+ title: string
+ description: string | null
+ /**
+ * The stored render, or null where this page has no body to show a reader who has not asked for
+ * one: a password-protected page, whose body is exactly what the password covers, and a redirection,
+ * which has nowhere to keep one.
+ *
+ * Only wanted by the caller building a document for a client that will never run the app; a browser
+ * about to boot the SPA is served the head alone.
+ */
+ render: string | null
+ updatedAt: Date
+ /**
+ * Whether a search engine should list it — `isSearchable`, and off as well for the two cases with
+ * nothing to list. The same question the sitemap asks, answered for one page.
+ */
+ isIndexable: boolean
+ /**
+ * Every locale this page exists in and this requester may read, this one included. Empty for a page
+ * that is not part of a translation set, since an alternates list naming one document says nothing.
+ */
+ alternates: { locale: string; path: string }[]
+}
+
export interface PageActor {
id: string
permissions: string[]
@@ -928,10 +965,7 @@ class Pages {
// next fetch as it did on the one that handed the crawler those numbers
.orderBy(pagesTable.locale, pagesTable.path)
- // -> No group-wide permissions rather than the guests group's own: `checkAccess` reads that list
- // only for `manage:system`, and a wiki that had somehow handed the public an administrator's
- // permission should not also publish an index of every page it has as a consequence
- const guests = { groupIds: [WIKI.data.systemIds.guestsGroupId], permissions: [] }
+ const guests = WIKI.models.groups.actorForPublic()
const pages = rows
.filter((row) => WIKI.models.groups.checkAccess(guests, 'read:pages', row))
.map(({ locale, path, updatedAt, localeGroupId }) => ({
@@ -945,6 +979,148 @@ class Pages {
return pages
}
+ /**
+ * What the PUBLIC may see at a path, for the HTML document served to a client that will not run the
+ * app.
+ *
+ * The question `listForSitemap` asks of a whole site, asked of one page: the guests group's rules
+ * decide and nothing about the requester does, which is what makes the answer the same for whoever
+ * fetched it and therefore safe to cache and hand on.
+ *
+ * Published only, as it is everywhere the public is being answered — which is also what lets the app
+ * shell say 404 rather than 200 at a path with nothing published at it.
+ */
+ async describePageForPublic(
+ siteId: string,
+ ref: { locale: string; path: string }
+ ): Promise<PageDescription | null> {
+ return this.describePage(siteId, ref, WIKI.models.groups.actorForPublic(), true)
+ }
+
+ /**
+ * What THIS REQUESTER may see at a path, for the document a browser is about to boot the app from.
+ *
+ * Deliberately the same cut the page route itself makes (`GET /sites/:siteId/pages/:pageIdOrHash`),
+ * so that the title in the document and the page the app then draws can never be about different
+ * things: its `read:pages` check per path, and its `publicOnly: !actorFrom(req)` — which is to say
+ * any signed-in session sees an unpublished page, and an API key is answered as the public is, since
+ * a draft belongs to the people working on it rather than to whatever holds a token.
+ *
+ * Not for caching across requesters: the answer is one reader's.
+ */
+ async describePageForRequest(
+ siteId: string,
+ ref: { locale: string; path: string },
+ req: FastifyRequest
+ ): Promise<PageDescription | null> {
+ const isSession = Boolean(req.session?.authenticated && req.session.user?.id)
+ return this.describePage(siteId, ref, WIKI.models.groups.actorForRequest(req), !isSession)
+ }
+
+ /**
+ * The read behind both, which is the only thing that should call it.
+ *
+ * `actor` and `publicOnly` are paired by the two methods above rather than left to a caller, because
+ * the wrong pairing — the public actor with unpublished pages included — would describe a draft to
+ * whoever asked.
+ */
+ private async describePage(
+ siteId: string,
+ { locale, path }: { locale: string; path: string },
+ actor: AccessActor,
+ publicOnly: boolean
+ ): Promise<PageDescription | null> {
+ const conditions = [
+ eq(pagesTable.siteId, siteId),
+ eq(pagesTable.locale, locale),
+ // -> By path, and not by the hash `getPage` addresses a page with: that hash is 53 bits and is
+ // never checked against the path it stands for, which is a collision a document served to
+ // whoever asked simply need not have — nothing here was handed a hash to look up
+ eq(pagesTable.path, path)
+ ]
+ if (publicOnly) {
+ conditions.push(eq(pagesTable.publishState, 'published'))
+ }
+
+ const rows = await WIKI.db
+ .select({
+ locale: pagesTable.locale,
+ path: pagesTable.path,
+ title: pagesTable.title,
+ description: pagesTable.description,
+ render: pagesTable.render,
+ updatedAt: pagesTable.updatedAt,
+ tags: pagesTable.tags,
+ editor: pagesTable.editor,
+ isSearchable: pagesTable.isSearchable,
+ password: pagesTable.password,
+ localeGroupId: pagesTable.localeGroupId
+ })
+ .from(pagesTable)
+ .where(and(...conditions))
+ .limit(1)
+
+ const row = rows[0]
+ if (!row) {
+ return null
+ }
+ if (!WIKI.models.groups.checkAccess(actor, 'read:pages', row)) {
+ return null
+ }
+
+ // -> A page and its lock are two things: the title and the description are not what a password
+ // covers (the search index treats them the same way), so they travel and the body does not.
+ // Held against the column and not against this requester's unlocks — the public description of
+ // a page is cached and shared, so a body any one requester had earned must not get into it
+ const isProtected = Boolean(row.password)
+ const isRedirect = row.editor === REDIRECT_EDITOR
+ return {
+ locale: row.locale,
+ path: row.path,
+ title: row.title,
+ description: row.description,
+ render: isProtected || isRedirect ? null : row.render,
+ updatedAt: row.updatedAt,
+ isIndexable: row.isSearchable && !isProtected && !isRedirect,
+ alternates: await this.readableAlternates(siteId, row.localeGroupId, actor)
+ }
+ }
+
+ /**
+ * The locales a page can be read in, for the alternates a document names.
+ *
+ * Published and readable by this actor, and including the page itself, which is what an `hreflang`
+ * set calls for. A translation the asker may not read is not a URL to point them at — for the public
+ * that is the same cut the sitemap makes, which is why the two say the same thing about a page.
+ *
+ * Published regardless of who is asking, unlike the page itself: a draft translation is not an
+ * alternate version of a document, it is one that does not exist yet.
+ */
+ private async readableAlternates(
+ siteId: string,
+ localeGroupId: string | null,
+ actor: AccessActor
+ ): Promise<{ locale: string; path: string }[]> {
+ if (!localeGroupId) {
+ return []
+ }
+ const rows = await WIKI.db
+ .select({ locale: pagesTable.locale, path: pagesTable.path, tags: pagesTable.tags })
+ .from(pagesTable)
+ .where(
+ and(
+ eq(pagesTable.siteId, siteId),
+ eq(pagesTable.localeGroupId, localeGroupId),
+ eq(pagesTable.publishState, 'published')
+ )
+ )
+ .orderBy(pagesTable.locale)
+ const readable = rows.filter((row) => WIKI.models.groups.checkAccess(actor, 'read:pages', row))
+ // -> One document is not a set of alternates: a page whose only readable version is itself has
+ // nothing for an annotation to point at
+ return readable.length > 1 ? readable.map(({ locale, path }) => ({ locale, path })) : []
+ }
+
/**
* Where a page lives, as a path.
*
@@ -1244,6 +1420,15 @@ class Pages {
metadata: { title: page.title, description: page.description, editor }
})
+ /*
+ Everything a document served to a client that will not run the app says about a page comes out
+ of the row this just wrote — its title, its description, its body, whether it is published and
+ searchable, and where it sits. Dropping the lot is cheaper than working out which URLs a change
+ reached: a rename moves a page, and joining or leaving a translation set moves its siblings'
+ alternates with it.
+ */
+ invalidateAppShellCache()
+
return { page: (await this.getPage({ siteId, id: page.id })) as Page, versionId }
}
@@ -1414,6 +1599,7 @@ class Pages {
authorId: actor.id,
metadata: { title: updated.title, description: updated.description }
})
+ invalidateAppShellCache()
return { page: updated, versionId }
}
@@ -1547,6 +1733,7 @@ class Pages {
siteId,
authorId: actor.id
})
+ invalidateAppShellCache()
return { page: moved, versionId }
}
@@ -1590,6 +1777,7 @@ class Pages {
siteId,
authorId: actor.id
})
+ invalidateAppShellCache()
return { page, versionId }
}
@@ -1691,6 +1879,7 @@ class Pages {
authorId: actor.id
})
}
+ invalidateAppShellCache()
WIKI.logger.debug(`Deleted ${entries.length} page(s) that went with a deleted folder.`)
}
@@ -2066,6 +2255,8 @@ class Pages {
// -> Nothing was updated when the page went while it sat in the queue
if (updated[0]) {
await WIKI.models.search.indexPage(id, updated[0].locale)
+ // -> This is the column the injected copy of a page IS, so a re-render changes it
+ invalidateAppShellCache()
}
}
diff --git a/backend/models/sites.ts b/backend/models/sites.ts
index fdb28a9bd..3982e3ab1 100644
--- a/backend/models/sites.ts
+++ b/backend/models/sites.ts
@@ -7,6 +7,7 @@ import {
storage as storageTable
} from '../db/schema.ts'
import { and, eq } from 'drizzle-orm'
+import { invalidateAppShellCache } from '../helpers/appShell.ts'
import { detectImageMime, detectSvg, normalizeImage, svgMimeType } from '../helpers/images.ts'
import type { ImageNormalization } from '../helpers/images.ts'
import type { SystemIds } from './types.ts'
@@ -84,13 +85,16 @@ class Sites {
WIKI.sitesMappings[site.hostname] = site.id
}
/*
- Sitemap lists are held per site for minutes at a time, and `WIKI.cache` has no expiry sweeper —
- an entry is only dropped when its own key is next read. So a site that was deleted, or whose
- sitemap was just switched off, would hold its last list for the life of the process: nothing
- will ever ask for that key again. This is the one place every create, update and delete passes
- through, which makes it the place to let them go.
+ Sitemap lists and app shell fragments are held per site for minutes at a time, and `WIKI.cache`
+ has no expiry sweeper — an entry is only dropped when its own key is next read. So a site that
+ was deleted, or whose sitemap was just switched off, would hold its last list for the life of the
+ process: nothing will ever ask for that key again. This is also the one place every create,
+ update and delete of a site's settings passes through, and those settings are in both — the
+ site's own title and description, whether it wants to be indexed, how it brackets URLs by
+ locale. So it is the place to let them go.
*/
WIKI.models.pages.invalidateSitemaps()
+ invalidateAppShellCache()
WIKI.logger.info(`Loaded ${sites.length} site configurations [ OK ]`)
}
diff --git a/frontend/index.html b/frontend/index.html
index 45537b326..3581a1293 100644
--- a/frontend/index.html
+++ b/frontend/index.html
@@ -45,6 +45,19 @@
animation: initspinner .6s linear infinite;
z-index: 2000000000;
}
+
+ /*
+ The copy of the page the server puts into this document for a client that will never run the
+ app -- a crawler, a chat client building an unfurl card, a reader with JavaScript off. See
+ `backend/helpers/appShell.ts` for what goes in it and why.
+
+ Out of sight of anyone whose browser is about to draw the real page, and shown (below) to
+ anyone whose browser is not. `main.js` takes the element out of the document altogether
+ before Vue mounts, so nothing the app does has to know it was ever there.
+ */
+ #wiki-prerender {
+ display: none;
+ }
</style>
<noscript>
@@ -64,6 +77,74 @@
.scroll.relative-position > .absolute {
position: relative !important;
}
+
+ /* With no app coming, the server's copy of the page is the page. */
+ #wiki-prerender {
+ display: block;
+ max-width: 60rem;
+ margin: 0 auto;
+ padding: 2rem 1.5rem;
+ line-height: 1.6;
+ }
+
+ /*
+ The browser defaults Tailwind's preflight takes away, put back. Deliberately not a copy of
+ the content styles in `css/tailwind.css`: those belong to elements the app draws, which is
+ the one thing that is not going to happen here, and a second copy of them would only drift.
+ What is wanted is prose that reads as prose.
+ */
+ #wiki-prerender h1 { font-size: 2em; }
+ #wiki-prerender h2 { font-size: 1.5em; }
+ #wiki-prerender h3 { font-size: 1.17em; }
+ #wiki-prerender h1,
+ #wiki-prerender h2,
+ #wiki-prerender h3,
+ #wiki-prerender h4,
+ #wiki-prerender h5,
+ #wiki-prerender h6 {
+ font-weight: bold;
+ margin: 1.5em 0 .5em;
+ line-height: 1.25;
+ }
+ #wiki-prerender p,
+ #wiki-prerender ul,
+ #wiki-prerender ol,
+ #wiki-prerender pre,
+ #wiki-prerender blockquote,
+ #wiki-prerender table {
+ margin: 0 0 1em;
+ }
+ #wiki-prerender ul,
+ #wiki-prerender ol {
+ padding-left: 2em;
+ list-style: revert;
+ }
+ #wiki-prerender pre,
+ #wiki-prerender code {
+ font-family: monospace;
+ }
+ #wiki-prerender pre {
+ overflow-x: auto;
+ padding: .75em 1em;
+ background: #f4f4f4;
+ }
+ #wiki-prerender blockquote {
+ padding-left: 1em;
+ border-left: 4px solid #ccc;
+ color: #555;
+ }
+ #wiki-prerender img {
+ max-width: 100%;
+ height: auto;
+ }
+ #wiki-prerender table {
+ border-collapse: collapse;
+ }
+ #wiki-prerender th,
+ #wiki-prerender td {
+ padding: .35em .75em;
+ border: 1px solid #ccc;
+ }
</style>
</noscript>
</head>
diff --git a/frontend/src/main.js b/frontend/src/main.js
index 8e34b6262..908c2ab85 100644
--- a/frontend/src/main.js
+++ b/frontend/src/main.js
@@ -32,4 +32,8 @@ initializeEventBus()
initializeIconify()
initializeExternals(router, store)
initializeI18n(app, store)
+// The server's copy of the page, for clients that never get this far -- see
+// `backend/helpers/appShell.ts`. It has done its job by now, and Vue is about to draw the real thing.
+document.getElementById('wiki-prerender')?.remove()
+
app.mount('#app')