diff --git a/CLAUDE.md b/CLAUDE.md index 3e8b69247..4f5337ee8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -71,7 +71,8 @@ scheduler → event emitters), `initHTTPServer()` (Fastify plugins, auth, routes `WIKI` global (config + logger + lazy `ensureDb()`) and dynamically imports the task. - `base.yml` — system defaults for every config key. Do not edit as a user-facing config; it defines the shape merged with `config.yml` and the db `settings` table. -- `helpers/` — small pure utilities (`common.ts`, `config.ts`). +- `helpers/` — small pure utilities (`common.ts`, `config.ts`), plus `storageFiles.ts`, which is the + file-tree half of the storage modules that address content by path (see [Storage targets](#storage-targets)). - `types/` — ambient declarations: `global.d.ts` (the `WIKI` global) and `fastify.d.ts` (session + route-permission augmentations). - `locales/` — `en.json` source strings (Localazy-managed) + `metadata.js` language table (the one @@ -400,15 +401,49 @@ Consequences worth knowing: ### Storage targets -A storage target is one module from `modules/storage//` configured for one site. Two modules -ship, both with a real `storage.ts`: `db`, enabled on every site and impossible to turn off, and -`disk`, which mirrors the wiki's tree at `///` — the configured root is -the site's own folder, since a target belongs to one site, so two sites must not share a path. S3, -Azure, GCS, git and SFTP were removed rather than left as definitions with nothing behind them; a -new module is a directory with a `definition.yml` and a `storage.ts` exporting the `StorageModule` +A storage target is one module from `modules/storage//` configured for one site. Six ship, each +with a real `storage.ts`: + +| Key | What it is | +| --- | ---------- | +| `db` | Enabled on every site and impossible to turn off — bytes in the asset's own row | +| `disk` | The wiki's tree as files at `///` | +| `git` | That same tree in a repository, committed per change and synced with a remote | +| `s3` | Amazon S3 **and anything speaking its API** — R2, Spaces, B2, Wasabi, MinIO | +| `azure` | Azure Blob Storage | +| `gcs` | Google Cloud Storage | +| `sftp` | That tree again, on a remote host over SSH — a copy, never a delivery source | + +A new module is a directory with a `definition.yml` and a `storage.ts` exporting the `StorageModule` contract, and `hasImplementation` gates both dispatch and the admin area's action buttons on the latter existing. +**`disk`, `git` and `sftp` are the same tree in three places**, and share `helpers/storageFiles.ts` +for all of it: the layout, the front matter, what makes a file a page, and `importTree`'s adoption +walk. `sftp` hands `importTree` its own `readFile` — the only thing that differs about a tree on +another machine — and uses `path.posix` throughout, since a wiki on Windows still talks to sshd in +slashes. Its connection is cached per target and serialized, because a single `ssh2-sftp-client` does +not support concurrent operations, and dropped on a connection-shaped failure so the next operation +reconnects. Unlike 2.x's SFTP module it reads as well as writes, so a site can serve from it and +import a tree that was put there from outside. + +**The three object stores are one file of client calls each.** `put`, `get`, `remove` and `copy` — the +`ObjectStoreClient` in `helpers/storageObjects.ts` — and `objectStorageModule` builds the whole +`StorageModule` from them. An object key *is* a path, the same one `disk` would write, so a bucket and +a folder hold a site's content laid out identically and `pathPrefixFor` decides the shape of both. +Object stores have no rename, so `moveObject` copies and then deletes, in that order, and never +deletes on a copy that failed. Credentials are optional on all three: left empty, each SDK falls back +to the machine's own identity (an IAM role, a managed identity, a workload identity), which is how a +deployment keeps a long-lived secret out of the database. Only `exportAll` is offered — there is no +`importAll`, because nothing but the wiki writes into these buckets, which is exactly what makes git +different. + +**Nothing else may live under `modules/storage/`.** `refreshFromDisk` reads every directory there and +expects a `definition.yml` in it; one without takes *every* storage module down with it, since the +read is wrapped in a single try/catch that empties `definitions`. This is why the tree logic shared by +`disk` and `git` — the front matter, the file name a page is filed under, what makes a file a page +rather than an attachment, and the walk an import does — sits in `helpers/storageFiles.ts` instead. + **Content is written to every target that claims it, and read from one.** Those are two separate questions with two separate answers, and conflating them is the way to get this wrong: @@ -420,6 +455,37 @@ questions with two separate answers, and conflating them is the way to get this a target may only be nominated for a type it also stores (`validateTarget` refuses the pair). Pages are never nominated: a page is read from its own row, always. + **Direct access.** An object store can answer instead of being read through: `assetDelivery.mode` + is `streaming` (the default — bytes through the wiki) or `direct`, where both serving routes + (`controllers/files.ts` and the asset content API) answer **302 to a signed URL** and the bytes + never touch the server. `storage.directAccessUrlFor` is the whole decision and both routes call it. + Three things it insists on: the target must be the *nominated* source for that content type + (`deliveryTargetsFor`'s head standing in as the database is not a nomination), the module must + implement `presignAsset`, and the link's lifetime is capped at 7 days because no provider signs for + longer. The redirect is cached for **half** the link's life, so a cached redirect can never outlive + the URL in it. + + **A signed link carries none of the wiki's page rules** — it works for whoever holds it until it + expires. That is what makes the store able to serve without asking the wiki, and it is why the + expiry defaults to `5m`. When signing fails, the site's `storage.directAccessFallback` decides: + `stream` (default) serves the bytes the slow way so a bad credential costs performance rather than + every image on every page, `error` fails the request so it cannot go unnoticed. The target records a + `warning` either way. + + **A custom `baseUrl` is signed *for*, never swapped in afterwards.** S3 and GCS sign the host, so + rewriting it invalidates the signature: S3 builds a second client (`bucketEndpoint` with the URL as + the `Bucket` when the domain *is* the bucket, `forcePathStyle` when the bucket is a path segment), + GCS passes `cname`, and Azure alone can simply swap the origin because a SAS signs the + canonicalized resource and not the host. CloudFront is therefore out of scope — it needs its own + key pair and signing scheme. + + **A module can decline to serve at all.** `assetDelivery.isDeliverySupported` (default true; false + only for `sftp`) keeps a target out of the Content Delivery tab, out of `sourceOptions`, and gets a + nomination refused by `validateTarget` and cleared by `updateTarget`. It is still written to, + exported to and imported from — the point is that every image on every page should not be an SSH + round trip. It does stay in the read fallback list, sorted behind even the database: `offloadUnchecked` + can leave a file whose only copy is there, and a slow answer beats telling a reader it is gone. + **Only an explicit nomination moves delivery.** A type nobody has been nominated for is served from the database (`deliveryTargetsFor`), never from whichever other target happens to be enabled — enabling one says where content is *written*, and a target enabled after an upload holds none of @@ -431,6 +497,23 @@ So neither an asset nor a page records where its bytes went — there is no sing target derives where its own copy sits from the tree, the same way for both. `resolveTargetFor` and `assets.storageInfo` are gone; `writeTargetsFor` and `deliveryTargetsFor` replace them. +**Where a file sits under a target's root is one answer per site too**, and `storage.pathPrefixFor` +is the only thing that gives it: the leading segments of every path any path-based module writes, +with `parseStoredPath` as its exact inverse for reading a folder back in. Two site settings shape it, +both on the **Configuration** tab and both read through `storage.pathLayoutFor`: + +- **`storage.sitePrefix`** (off) files the tree under a folder named after the site. Off because the + configured root already *is* that site's folder; on is what lets two sites share a location, each + ignoring the other's half of it. +- **`storage.localePrefix`** (on) brackets the tree by locale, which is what keeps `guides/logo.png` + from being the same file in every locale. **Off, the site stores its primary locale and no other** + — there is nowhere to put the rest — so `pathPrefixFor` answers null for them and the target is + skipped: a page copy silently (the page is in the database either way), an asset copy after a + `canStore` check, so that the upload still succeeds as long as some *other* target takes the bytes. + +Neither setting moves anything already stored; the disk target's export and import actions are how +content crosses from one layout to the next. + **Which content type a file is, is one answer per site, not per target.** `large` is a category of its own rather than a modifier — that is what lets a target take the 40 MB video without also taking every thumbnail — and the size at which it starts lives in the site's config as @@ -443,7 +526,11 @@ Everything else follows from that: - **A write must succeed everywhere; a read may fall back.** `storage.putAsset` fans out and throws if any target refuses, failing the upload — an asset may have no database copy, so a half-stored one - must not be reported as saved. `getAsset` tries the nominated source and then every other target + must not be reported as saved. **Refusing is not the same as having nowhere to put it**: a target + whose `canStore` says no is never asked, because nothing has gone wrong and the file may well be + storable elsewhere. Only when *nothing* can hold it is the upload failed, and then with a + `CustomError` — a plain `Error` reaches the client as a bare 500, since the error handler in + `index.ts` only forwards a message that came with a `statusCode`. `getAsset` tries the nominated source and then every other target holding the content, database last, because a target enabled after an upload never received it. Pages are gentler still: `mirrorPage` / `removePage` / `relocatePage` log a target that could not keep up and carry on, since the database always has the page. @@ -500,6 +587,56 @@ wins — a later success clears an earlier failure, which is what makes a full d stop reporting itself without anybody dismissing anything. Nothing probes a target proactively, so a misconfigured one reads healthy until something is actually asked of it. +**The git target is the disk target plus history and a remote.** Same tree, same helpers, and then: + +- **Commits are local and immediate; the network is batched.** A page save writes, commits and + returns — `prepareRepo` deliberately never contacts a remote, so an unreachable one cannot make the + wiki slow to edit or fail an upload. `ensureRemote` is the half that does, and only `sync` calls it. +- **`sync` runs on a schedule.** `tasks/simple/sync-storage-targets.ts`, on a `* * * * *` + `jobSchedule` row seeded by `jobs.init()`, walks every enabled target whose module declares a `sync` + handler (`storage.syncableTargets`) and runs it through `executeAction`, so a failure lands on the + target's Status card. The **Force Sync** action is the same call on demand. In an HA set the + scheduler hands the job to one instance, which is the one whose working copy syncs — each instance + keeps its own. +- **How often is `storage.syncInterval`, per site** (the **Configuration** tab, `syncIntervalFor`, + default `5m`). Since it is per site, one cron row cannot express it: the tick is every minute — as + fine as the shortest interval anyone can set — and the task skips the sites whose turn it is not. + Due-ness is `epochMinute % intervalMinutes === 0` rather than a stored last-sync time, so nothing + has to be persisted, two instances agree without coordinating and a restart changes nothing. The + cost is that a missed tick is not caught up, which for something running all day is the right + trade. An interval that will not parse means *never*, not every minute. +- **A pull is authoritative, and that includes deletions.** What comes back is applied to the wiki + with `overwrite`, and a commit that deleted a file deletes the page or asset here too. So push + access to the remote is effectively write access to the wiki. Which of the two a vanished path was + has to come off the tree rather than the file: the stem is looked up first and only counts as the + page if `pages.storageFileNameOf` matches the name that went, so a deleted `readme.pdf` never takes + the page `readme` with it. +- **The diff, not the tree.** Incoming changes come from `git diff --name-status -M -z` between the + commit the branch was on and the one it is on now — a sync runs every few minutes and cannot read + every file each time. `-z` because a path may contain anything, newlines included. +- **Every operation is serialized per target** by `withRepo`: git locks its index for the length of a + write, so two concurrent uploads would otherwise have one fail on `index.lock`. +- **An empty commit is never made.** Most page saves do not change the stored form of the page (a + re-publish, a tag reordered into the same order), and `commitPaths` checks `diff --cached` before + committing so the history says what actually changed. +- Operator git config is inherited on purpose — including `commit.gpgsign`, which will fail every + commit if it is on without a usable key. The failure surfaces as the target's recorded `error`. + +**A change carries who made it, and only git cares.** `StorageAssetRef` and `StoragePageRef` carry an +optional `actorId`, which is the user id the models already had in hand at each dispatch site; +`storage.actorFor` turns it into a name and an email, cached indefinitely because a page save is a hot +path and a stale commit author costs nothing. Absent for a change no one person made — a folder rename +that moved a hundred files, a scheduled sync — and the git target's configured default author stands +in. A scheduled pull creates content, which needs an author, so it uses +`users.getSystemActorId()`: the wiki's longest-standing active administrator. + +Git's **`alwaysUseDefaultAuthor`** makes that stand-in universal, for a repository whose history must +not carry the wiki's accounts. `commitAuthor` then returns the default without calling `actorFor` at +all — not looked up and discarded, so there is nothing to leak by mistake. Note the *committer* is the +default author in every case regardless: it is the repository's own `user.name`/`user.email`, which +`prepareRepo` writes from those same two settings, and which is why they are in +`configFingerprint` — renaming the default author has to re-prepare the repository to take effect. + Not to be confused with `/cache/files`, the serving cache in the assets model. That one is derived and swept; a storage target is where content actually lives. diff --git a/backend/api/assets.ts b/backend/api/assets.ts index 0fe546458..b96546906 100644 --- a/backend/api/assets.ts +++ b/backend/api/assets.ts @@ -226,6 +226,20 @@ async function routes(app: FastifyInstance) { if (!asset || !mayOnAsset(req, 'read:assets', asset)) { return reply.notFound('This asset does not exist.') } + const download = Boolean( + WIKI.config.security?.forceAssetDownload || !INLINE_EXTS.has(asset.fileExt) + ) + + // -> The same redirect `/_files/` makes, for the same reason: where the site has nominated a + // store to serve a content type directly, the bytes have no business coming through here + const directUrl = await WIKI.models.storage.directAccessUrlFor( + { ...asset, siteId: req.params.siteId }, + { contentType: asset.mimeType, ...(download ? { downloadAs: asset.fileName } : {}) } + ) + if (directUrl) { + return reply.redirect(directUrl.url, 302) + } + // -> Through the same local disk cache `/_files/` serves from, since this is the download // button in the file manager rather than an administrative route: anyone who may read a // file may press it @@ -234,7 +248,7 @@ async function routes(app: FastifyInstance) { return reply.notFound('This asset has no content.') } - if (WIKI.config.security?.forceAssetDownload || !INLINE_EXTS.has(asset.fileExt)) { + if (download) { reply.header( 'Content-Disposition', `attachment; filename="${encodeURIComponent(asset.fileName)}"` @@ -347,7 +361,13 @@ async function routes(app: FastifyInstance) { if (!mayOnAsset(req, 'manage:assets', doomed)) { return reply.forbidden('You are not allowed to delete this file.') } - if (!(await WIKI.models.assets.deleteAsset(req.params.siteId, req.params.assetId))) { + if ( + !(await WIKI.models.assets.deleteAsset( + req.params.siteId, + req.params.assetId, + req.session.user?.id + )) + ) { return reply.notFound('This asset does not exist.') } return reply.code(204).send() diff --git a/backend/api/schemas/storage.ts b/backend/api/schemas/storage.ts index ca43f0640..e7c6aee78 100644 --- a/backend/api/schemas/storage.ts +++ b/backend/api/schemas/storage.ts @@ -1,4 +1,8 @@ -import { CONTENT_TYPES, STORAGE_TARGET_STATUSES } from '../../models/storage.ts' +import { + CONTENT_TYPES, + STORAGE_DELIVERY_MODES, + STORAGE_TARGET_STATUSES +} from '../../models/storage.ts' import type { FastifyInstance } from 'fastify' export async function registerSchemas(app: FastifyInstance): Promise { @@ -55,19 +59,33 @@ export async function registerSchemas(app: FastifyInstance): Promise { assetDelivery: { type: 'object', description: - 'How assets reach the user. The `is*Supported` flags come from the module and are read-only.', + 'How assets reach the user. `isDirectAccessSupported` comes from the module and is read-only.', properties: { - isStreamingSupported: { - type: 'boolean' - }, isDirectAccessSupported: { - type: 'boolean' + type: 'boolean', + description: + 'Whether this module can sign a URL a reader fetches the file from directly. True for the object stores and nothing else.' + }, + isDeliverySupported: { + type: 'boolean', + description: + "Whether a site may nominate this target to answer readers' requests. False for SFTP, which is a place to keep a copy of the content rather than one to serve it from - it is still written to, exported to and imported from. A nomination on such a target is refused, and it answers a read only as a last resort, once every target that may serve has been asked and had nothing." }, - streaming: { - type: 'boolean' + mode: { + type: 'string', + enum: [...STORAGE_DELIVERY_MODES], + description: + '`streaming` sends the bytes through the wiki; `direct` answers with a redirect to a URL the store signed. Only consulted on the target nominated for the content type being asked for. Stored as `streaming` on a module that cannot do the other.' }, - directAccess: { - type: 'boolean' + baseUrl: { + type: 'string', + description: + "The origin a direct link is built on, in place of the store's own - a CDN or custom domain in front of the bucket. The signature is made for that host rather than moved onto it afterwards, so it must be a domain that actually fronts the bucket." + }, + linkExpiration: { + type: 'string', + description: + "How long a direct link stays valid, e.g. `5m` or `1h`. Capped at 7 days, which is as far as any of these providers will sign. Short by default: the link carries none of the wiki's page rules, so its lifetime is how long it can be passed on." }, servedTypes: { type: 'array', @@ -172,13 +190,21 @@ export async function registerSchemas(app: FastifyInstance): Promise { }, assetDelivery: { type: 'object', - description: 'A delivery mode the module does not support is stored as off.', + description: 'Direct access asked of a module that cannot do it is stored as streaming.', properties: { - streaming: { - type: 'boolean' + mode: { + type: 'string', + enum: [...STORAGE_DELIVERY_MODES] }, - directAccess: { - type: 'boolean' + baseUrl: { + type: 'string', + maxLength: 1024, + description: "A full http or https origin, or empty for the store's own address." + }, + linkExpiration: { + type: 'string', + maxLength: 32, + description: 'A whole number of minutes or hours, at most 7 days.' }, servedTypes: { type: 'array', diff --git a/backend/api/storage.ts b/backend/api/storage.ts index 4af278536..e01f4dec8 100644 --- a/backend/api/storage.ts +++ b/backend/api/storage.ts @@ -1,5 +1,6 @@ +import { STORAGE_DIRECT_ACCESS_FALLBACKS, STORAGE_TARGET_STATUSES } from '../models/storage.ts' import type { FastifyInstance } from 'fastify' -import type { StorageTargetInput } from '../models/storage.ts' +import type { StorageSiteConfigInput, StorageTargetInput } from '../models/storage.ts' /** * Storage API Routes @@ -39,6 +40,27 @@ async function routes(app: FastifyInstance) { description: 'Size at or above which an asset counts as a large file, e.g. `5MB`. One answer for the whole site: a file has to be the same kind of thing to every target.' }, + sitePrefix: { + type: 'boolean', + description: + 'Whether the paths a target stores content under are filed inside a folder named after the site. Off by default, since a target belongs to one site and its configured root is therefore already the folder of that site; turn it on for two sites sharing a location.' + }, + localePrefix: { + type: 'boolean', + description: + 'Whether the paths a target stores content under are bracketed by locale. On by default. Off, the site stores its primary locale only, directly under the root - content in any other locale has no path and is not written to a path-based target at all.' + }, + syncInterval: { + type: 'string', + description: + 'How often a target with a remote is synchronized, e.g. `5m` or `1h`. Applies to every target of the site that has a remote at all; the others have nothing to synchronize with and ignore it.' + }, + directAccessFallback: { + type: 'string', + enum: [...STORAGE_DIRECT_ACCESS_FALLBACKS], + description: + 'What happens when a target set to hand out direct links cannot sign one. `stream` serves the bytes through the wiki instead, so a signing misconfiguration costs performance rather than breaking every image; `error` fails the request, so it cannot go unnoticed. Either way the target records a warning.' + }, targets: { type: 'array', items: { $ref: 'StorageTarget#' } @@ -53,19 +75,90 @@ async function routes(app: FastifyInstance) { if (!site) { return reply.notFound('Site does not exist.') } + const layout = WIKI.models.storage.pathLayoutFor(req.params.siteId) return { largeThreshold: WIKI.models.storage.largeThresholdFor(req.params.siteId), + sitePrefix: layout.sitePrefix, + localePrefix: layout.localePrefix, + syncInterval: `${WIKI.models.storage.syncIntervalFor(req.params.siteId)}m`, + directAccessFallback: WIKI.models.storage.directAccessFallbackFor(req.params.siteId), targets: await WIKI.models.storage.getSiteTargets(req.params.siteId) } } ) + /** + * GET SITE STORAGE STATUS + */ + app.get<{ Params: { siteId: string } }>( + '/sites/:siteId/storage/status', + { + config: { + // -> Deliberately not `manage:system`, unlike the rest of this file: this answers a status + // light in the admin sidebar, which anybody who can see the storage section at all needs, + // and it carries none of the configuration that makes the rest of these privileged + permissions: ['manage:sites'] + }, + schema: { + summary: "Get the health of a site's storage targets", + description: + 'How each enabled target is behaving, and nothing else - no configuration and no credentials. A target that is disabled is absent rather than reported: it is not being asked to do anything, so what it last recorded is history. Answered from the same cache the upload path resolves through, so it is current without costing a query.', + tags: ['Storage'], + params: { + type: 'object', + properties: { + siteId: { + type: 'string', + format: 'uuid' + } + }, + required: ['siteId'] + }, + response: { + 200: { + description: 'Health of the site storage targets', + type: 'object', + properties: { + targets: { + type: 'array', + items: { + type: 'object', + properties: { + id: { type: 'string', format: 'uuid' }, + title: { type: 'string' }, + isEnabled: { type: 'boolean' }, + state: { + type: 'object', + properties: { + status: { + type: 'string', + enum: [...STORAGE_TARGET_STATUSES] + } + } + } + } + } + } + } + } + } + } + }, + async (req, reply) => { + const site = await WIKI.models.sites.getSiteById({ id: req.params.siteId }) + if (!site) { + return reply.notFound('Site does not exist.') + } + return { targets: await WIKI.models.storage.healthFor(req.params.siteId) } + } + ) + /** * UPDATE SITE STORAGE CONFIGURATION */ app.put<{ Params: { siteId: string } - Body: { largeThreshold?: string; targets?: StorageTargetInput[] } + Body: StorageSiteConfigInput & { targets?: StorageTargetInput[] } }>( '/sites/:siteId/storage', { @@ -75,7 +168,7 @@ async function routes(app: FastifyInstance) { schema: { summary: 'Update the storage configuration of a site', description: - 'Only the targets listed are affected, and within each of them only the fields provided. Everything is validated before any of it is written, so a rejected request changes nothing.', + 'Only the targets listed are affected, and within each of them only the fields provided. Everything is validated before any of it is written, so a rejected request changes nothing. Changing the path layout moves nothing: content already stored stays where the previous layout put it, and a target holding it has export and import actions for putting that right.', tags: ['Storage'], params: { type: 'object', @@ -95,6 +188,26 @@ async function routes(app: FastifyInstance) { maxLength: 32, description: 'A size such as `5MB`. Applies to every target of the site.' }, + sitePrefix: { + type: 'boolean', + description: 'Prepend the site id to every path. Applies to every target of the site.' + }, + localePrefix: { + type: 'boolean', + description: + 'Prepend the locale to every path. Applies to every target of the site. Turning it off leaves every locale but the primary one without a path to be stored under.' + }, + syncInterval: { + type: 'string', + maxLength: 32, + description: + 'A whole number of minutes or hours, such as `5m` or `1h`. The scheduler ticks once a minute, so this is honoured to the minute and cannot be shorter than one.' + }, + directAccessFallback: { + type: 'string', + enum: [...STORAGE_DIRECT_ACCESS_FALLBACKS], + description: 'What to do when a direct link cannot be signed.' + }, targets: { type: 'array', items: { $ref: 'StorageTargetInput#' } diff --git a/backend/api/tree.ts b/backend/api/tree.ts index 4ebdc7fe4..2088ba8ad 100644 --- a/backend/api/tree.ts +++ b/backend/api/tree.ts @@ -598,7 +598,8 @@ async function routes(app: FastifyInstance) { const folder = await WIKI.models.tree.renameFolder({ folderId: req.params.folderId, pathName: req.body.pathName, - title: req.body.title + title: req.body.title, + actorId: req.session.user?.id }) return { ok: true, @@ -653,7 +654,7 @@ async function routes(app: FastifyInstance) { // -> The tree entries are gone; these are the rows behind them, which is where a page and an // asset actually live await WIKI.models.pages.deleteOrphaned(req.params.siteId, removed.pages, actor) - await WIKI.models.assets.deleteOrphaned(req.params.siteId, removed.assets) + await WIKI.models.assets.deleteOrphaned(req.params.siteId, removed.assets, actor.id) return reply.code(204).send() } ) diff --git a/backend/controllers/files.ts b/backend/controllers/files.ts index 809e417d4..eff62e601 100644 --- a/backend/controllers/files.ts +++ b/backend/controllers/files.ts @@ -11,6 +11,17 @@ import type { FastifyInstance } from 'fastify' */ const FILE_CACHE = 'private, max-age=600, must-revalidate' +/** + * How long a browser may keep a redirect to a signed URL. + * + * Half of the link's own lifetime, so that a cached redirect cannot outlive the URL it points at — a + * reader holding one of those has a broken image until their cache gives it up, and no way to force + * the issue. Never longer than an ordinary file would have been cached for anyway. + */ +function directAccessCacheSeconds(expiresInSeconds: number): number { + return Math.max(1, Math.min(600, Math.floor(expiresInSeconds / 2))) +} + /** * _files Routes * @@ -48,6 +59,31 @@ async function routes(app: FastifyInstance) { return reply.notFound('File not found') } + const download = Boolean( + WIKI.config.security?.forceAssetDownload || !INLINE_EXTS.has(asset.fileExt) + ) + + /* + Hand the reader straight to the store where the site has said to, which is the whole point of + keeping content in one: the bytes never touch this server. Resolved before the ETag below, + because the two are answers to different questions — an ETag says "you already have the bytes", + and there are no bytes here to have. + */ + const directUrl = await WIKI.models.storage.directAccessUrlFor( + { ...asset, siteId: site.id }, + { contentType: asset.mimeType, ...(download ? { downloadAs: asset.fileName } : {}) } + ) + if (directUrl) { + // -> Cacheable, but for less than the link lives: a redirect kept past its URL's expiry is a + // reader stuck on a dead link until their cache lets go of it. Half is the simple safe + // fraction, and it still takes most of a page's images off this server on a reload. + reply.header( + 'Cache-Control', + `private, max-age=${directAccessCacheSeconds(directUrl.expiresInSeconds)}` + ) + return reply.redirect(directUrl.url, 302) + } + /* The ID and the timestamp together, because either one alone lies: a file replaced at the same path is a different asset under the same URL, and one edited in place keeps its ID. @@ -69,7 +105,7 @@ async function routes(app: FastifyInstance) { return reply.notFound('File not found') } - if (WIKI.config.security?.forceAssetDownload || !INLINE_EXTS.has(asset.fileExt)) { + if (download) { reply.header( 'Content-Disposition', `attachment; filename="${encodeURIComponent(asset.fileName)}"` diff --git a/backend/helpers/storageFiles.ts b/backend/helpers/storageFiles.ts new file mode 100644 index 000000000..e0a763757 --- /dev/null +++ b/backend/helpers/storageFiles.ts @@ -0,0 +1,510 @@ +import fs from 'node:fs/promises' +import path from 'node:path' +import { dump as dumpYaml, load as loadYaml } from 'js-yaml' +import { pageEditorForExtension, pageFileExtension } from '../models/pages.ts' +import type { StoragePageContent, StoragePageRef, StorageTarget } from '../models/storage.ts' + +/** + * What a storage module needs in order to keep the wiki's content as a tree of ordinary files. + * + * Shared by every target that addresses content by path — the local disk and git today — because the + * two have to agree about it exactly. A page written by one and read back by the other has to come + * back as the same page, so the front matter, the file name a page is filed under, the rule for what + * makes a file a page rather than an attachment and the walk that reads a folder back all live here + * rather than in either module. + * + * What does *not* live here is anything about where the root is or what happens after a file is + * written: the disk target is finished at that point, and git has a commit to make. + * + * Not under `modules/storage/` deliberately. `refreshFromDisk` reads every directory there and + * expects a `definition.yml` in it, and a directory without one takes every storage module down with + * it. + */ + +/** Leading YAML front matter, as `serializePage` writes it. */ +const FRONT_MATTER = /^---\r?\n([\s\S]*?)\r?\n---\r?\n?/ + +/** The extension whose pages are written as one JSON document rather than front matter and a body. */ +export const JSON_EXTENSION = 'json' + +/** + * The editor an imported page falls back to. + * + * Only reached for a file the site reserves as a page extension but which no editor writes — `txt` on + * a default site. Markdown renders plain prose as prose, so it is the least surprising answer. + */ +const DEFAULT_PAGE_EDITOR = 'markdown' + +/** + * What the site's root page is filed as. + * + * A page path may be empty, which is the page a site serves at `/`. It still needs a name of its own + * on disk, and `index` is the one every other tool that writes a tree of documents picks. + */ +export const ROOT_PAGE_NAME = 'index' + +/** + * Names never walked by an import. + * + * Anything starting with a dot, which covers a half-written `.tmp`, the `.DS_Store` a Mac leaves in + * every folder it has looked at, and — the reason this is tested against every segment of the path + * rather than only the file name — the whole of a `.git` directory. None of that is content somebody + * meant to put in their wiki, and a repository's own internals least of all. + */ +const IGNORED_SEGMENT = /^\.|\.tmp$/ + +/** + * A configured root as an absolute path. + * + * A relative setting is resolved from the install directory rather than from the working directory, + * so that `./data/content` means the same folder whichever way the server was started. + */ +export function resolveRoot(configured: string | undefined, fallback: string): string { + return path.resolve(WIKI.ROOTPATH, configured || fallback) +} + +/** + * Where an asset belongs under the root, as a slash-separated relative path. + * + * What brackets the tree — the site, the locale, both or neither — is the site's own answer and is + * `pathPrefixFor`'s to give; everything below it is the tree as the file manager shows it. + * + * @returns Null for content this site's layout has no place for, which a caller reads as "this target + * does not hold that": a secondary locale on a site storing only its primary one + */ +export function assetRelPath( + target: StorageTarget, + ref: { locale: string; folderPath: string; fileName: string } +): string | null { + const prefix = WIKI.models.storage.pathPrefixFor(target.siteId, ref.locale) + if (!prefix) { + return null + } + return [...prefix, ...ref.folderPath.split('/').filter(Boolean), ref.fileName].join('/') +} + +/** + * Where a page's copy belongs, alongside the assets of the same folder. + * + * The extension is the one its editor writes, and is exactly what the wiki reserves against uploads — + * `models/assets.ts` refuses an attachment that would take this name, so the two never meet here. + * + * @returns Null under the same circumstances as `assetRelPath` + */ +export function pageRelPath(target: StorageTarget, ref: StoragePageRef): string | null { + const prefix = WIKI.models.storage.pathPrefixFor(target.siteId, ref.locale) + if (!prefix) { + return null + } + const segments = ref.path.split('/').filter(Boolean) + const fileName = segments.pop() ?? ROOT_PAGE_NAME + return [...prefix, ...segments, `${fileName}.${pageFileExtension(ref.contentType)}`].join('/') +} + +/** + * The absolute path of a stored file, refusing anything that would land outside the root. + * + * Every segment reaching this is either a UUID or a name the tree has already normalized, so this + * catches a stored path that has been tampered with rather than an ordinary mistake — but it is the + * only thing between a `..` in the database and the rest of the file system. + */ +export function absPathIn(root: string, relPath: string): string { + const resolved = path.resolve(root, relPath) + if (resolved !== root && !resolved.startsWith(root + path.sep)) { + throw new Error(`The stored path "${relPath}" resolves outside the storage folder.`) + } + return resolved +} + +/** + * Remove the folders a deleted file leaves behind, stopping at the first one still in use. + * + * Best effort throughout: a folder that turns out not to be empty, or that another request is + * writing into at that moment, is simply left alone. + */ +export async function pruneEmptyDirs(root: string, fromDir: string): Promise { + let dir = fromDir + while (dir !== root && dir.startsWith(root + path.sep)) { + try { + await fs.rmdir(dir) + } catch { + return + } + dir = path.dirname(dir) + } +} + +/** + * Write a file, creating its folder and leaving nothing half-written behind. + * + * Written under a temporary name and renamed, so a reader either finds the previous contents or the + * new ones — never the middle of a write. That matters here more than it does for a cache: this is + * the only copy of an asset once the database target has been purged. + */ +export async function writeFileAtomic(filePath: string, data: Buffer | string): Promise { + const tempPath = `${filePath}.${process.pid}.tmp` + await fs.mkdir(path.dirname(filePath), { recursive: true }) + try { + await fs.writeFile(tempPath, data) + await fs.rename(tempPath, filePath) + } catch (err) { + await fs.rm(tempPath, { force: true }).catch(() => {}) + throw err + } +} + +/** + * Move a file, coping with a root that spans devices and with the file not being there. + * + * @returns Whether anything was moved + */ +export async function moveFile(from: string, to: string): Promise { + await fs.mkdir(path.dirname(to), { recursive: true }) + try { + await fs.rename(from, to) + return true + } catch (err: any) { + if (err.code === 'ENOENT') { + return false + } + if (err.code !== 'EXDEV') { + throw err + } + // -> `rename` cannot cross a mount point + await fs.copyFile(from, to) + await fs.rm(from, { force: true }) + return true + } +} + +/** + * Follow a rename, given where the file was and where it now belongs. + * + * Either end may be nowhere: the layout can have no place for a locale, and a move may cross into or + * out of it. Moving *into* it has nothing to move — the copy was never written — whereas moving out + * of it leaves a file behind at the old path, so that one is a delete. + * + * @returns What actually happened, which a target keeping history needs in order to record it + */ +export async function moveStored( + root: string, + fromRelPath: string | null, + toRelPath: string | null +): Promise<'moved' | 'deleted' | 'nothing'> { + if (!fromRelPath) { + return 'nothing' + } + const from = absPathIn(root, fromRelPath) + if (!toRelPath) { + await fs.rm(from, { force: true }) + await pruneEmptyDirs(root, path.dirname(from)) + return 'deleted' + } + if (await moveFile(from, absPathIn(root, toRelPath))) { + await pruneEmptyDirs(root, path.dirname(from)) + return 'moved' + } + return 'nothing' +} + +/** The metadata every page file carries, whichever of the two forms it is written in. */ +function pageMeta(page: StoragePageContent): Record { + return { + title: page.title, + description: page.description, + published: page.isPublished, + date: page.updatedAt.toISOString(), + tags: page.tags, + // -> The declaration that makes this a page rather than a file that happens to sit here. Nothing + // is imported as a page without it, so it is the one key that must always be written. + editor: page.editor, + dateCreated: page.createdAt.toISOString() + } +} + +/** + * A page as a file that stands on its own, in one of two forms. + * + * A **text** page is YAML front matter and then the source as the author wrote it — the convention + * every static site generator reads and the one Wiki.js 2.x wrote, so the folder is worth something + * to tools that have never heard of this wiki. + * + * A **JSON** page — a redirection today — is a single JSON document with the same metadata at its top + * level and the source under `content`. Front matter would leave a `.json` file that is not JSON, + * which is worth avoiding for the one editor whose source is already structured. + * + * Either way the metadata is the point: a bare body says nothing about whether it was published or + * what it was called, and none of that is recoverable from the prose. + */ +export function serializePage(ref: StoragePageRef, page: StoragePageContent): string { + if (pageFileExtension(ref.contentType) === JSON_EXTENSION) { + let content: any = page.content + try { + content = JSON.parse(page.content) + } catch { + // -> Kept as the string it is. The column is written by the editor and should always parse, + // and a file that says what it holds beats one this refused to write. + } + return `${JSON.stringify({ ...pageMeta(page), content }, null, 2)}\n` + } + return `---\n${dumpYaml(pageMeta(page))}---\n\n${page.content}\n` +} + +/** + * Read a page file back: its declaration, and the source below it. + * + * A page file says it is one by carrying an `editor` — in its front matter, or at the top level of + * the JSON document for the one editor written that way. That declaration is what lets a page and an + * attachment share a folder without a module having to guess which is which from an extension, and + * it is how every file written here comes back. + * + * Returning null does not settle it. A file whose extension the site reserves for pages is a page + * whatever it does or does not declare — see `importTree`, which owns that rule and fills the editor + * in from the extension. This only reports whether the file said so itself. + * + * Beyond the one key it is forgiving: a file may have been hand-written or generated by something + * that has never seen this wiki, so a missing title or date is filled in by the caller, and front + * matter that is not YAML is treated as no declaration rather than as a fault. + * + * @returns The declaration and the source, or null for a file that does not declare itself a page + */ +export function deserializePage( + raw: string, + ext: string +): { meta: Record; content: string } | null { + if (ext === JSON_EXTENSION) { + let parsed: any + try { + parsed = JSON.parse(raw) + } catch { + return null + } + if (!parsed || typeof parsed !== 'object' || typeof parsed.editor !== 'string') { + return null + } + const { content, ...meta } = parsed + return { + meta, + content: typeof content === 'string' ? content : JSON.stringify(content ?? {}) + } + } + + const match = FRONT_MATTER.exec(raw) + if (!match) { + return null + } + let meta: Record + try { + const parsed = loadYaml(match[1]) + if (!parsed || typeof parsed !== 'object') { + return null + } + meta = parsed as Record + } catch { + return null + } + if (typeof meta.editor !== 'string' || !meta.editor) { + return null + } + return { meta, content: raw.slice(match[0].length).trim() } +} + +/** A front matter date, or undefined for one that is missing or not a date at all. */ +export function parseFileDate(value: unknown): Date | undefined { + if (value instanceof Date) { + return value + } + if (typeof value !== 'string') { + return undefined + } + const date = new Date(value) + return Number.isNaN(date.getTime()) ? undefined : date +} + +/** One stored file, as `walkStored` reports it. */ +export interface StoredFile { + /** Absolute. */ + filePath: string + /** Relative to the root, split — the whole path including the file name. */ + segments: string[] +} + +/** + * Every file under a root, with anything hidden — and so a repository's `.git` — left out. + * + * @returns Null when the root does not exist yet, which is not a fault: a target can be configured + * long before anything is written to it + */ +export async function walkStored(root: string): Promise { + let entries + try { + entries = await fs.readdir(root, { recursive: true, withFileTypes: true }) + } catch (err: any) { + if (err.code !== 'ENOENT') { + throw err + } + return null + } + const files: StoredFile[] = [] + for (const entry of entries) { + if (!entry.isFile()) { + continue + } + const filePath = path.join(entry.parentPath, entry.name) + const segments = path.relative(root, filePath).split(path.sep) + if (segments.some((segment) => IGNORED_SEGMENT.test(segment))) { + continue + } + files.push({ filePath, segments }) + } + return files +} + +/** + * How a run of `importTree` went, for the module to report in its own words. + * + * A module says what the run *was* — an import from a folder, a pull from a remote — and this says + * what it did, which is the same either way. + */ +export interface ImportSummary { + pages: number + assets: number + /** Left alone because the wiki already had something at that path and `overwrite` was off. */ + skipped: number + /** Unusable: an empty body, an editor this wiki does not have, a path it cannot address. */ + failed: number +} + +/** + * Take a tree of files into the wiki, either filling in what is missing or letting the files win. + * + * **Two ways a file is a page**, and everything else is an attachment, read where it lies and adopted + * in place rather than copied: + * + * 1. **Its extension is one the site reserves for pages.** Those extensions address a page by URL and + * cannot be uploaded as attachments, so nothing else can be sitting under one — which makes a + * hand-written `.md` dropped into the folder a page, as whoever dropped it meant. + * 2. **It declares an editor**, in its front matter or at the top level of its JSON. This is what + * every file a target writes carries, and it is the only way in for an extension the site does not + * reserve — `.adoc` on a default site, where an attachment could just as well be sitting. + * + * Which editor it belongs to comes from that declaration, and only falls back to the extension for a + * reserved one that made none. + * + * `overwrite` is the only thing separating the safe direction from the authoritative one. Off, a path + * the wiki already has is left alone on both sides, which makes the run repeatable and makes it no + * use for a file edited on both sides — reconciling those is a merge, and a target without history + * cannot do one. On, the file wins: for a restore, or for a target whose remote is the authority. + * + * @param files What to take in, or null to walk the root + * @param readFile How to read one, for a tree that is not on this machine — the SFTP target hands in + * its own and everything else about the walk is the same + */ +export async function importTree({ + target, + root, + actorId, + overwrite, + files, + readFile = (filePath) => fs.readFile(filePath) +}: { + target: StorageTarget + root: string + actorId: string + overwrite: boolean + files?: StoredFile[] | null + readFile?: (filePath: string) => Promise +}): Promise { + const found = files === undefined ? await walkStored(root) : files + if (!found) { + return null + } + + const reserved: string[] = WIKI.sites[target.siteId]?.config?.pageExtensions ?? [] + const summary: ImportSummary = { pages: 0, assets: 0, skipped: 0, failed: 0 } + + for (const { filePath, segments } of found) { + // -> The prefix the site's layout writes, read back off the path. Null for a file that is not + // part of this site's tree: another site's folder, or one sitting outside the layout. + const stored = WIKI.models.storage.parseStoredPath(target.siteId, segments) + if (!stored) { + continue + } + const locale = stored.locale + const rest = stored.segments + const fileName = rest.pop()! + const ext = path.extname(fileName).replace(/^\./, '').toLowerCase() + const folderPath = rest.join('/') + + let raw: Buffer + try { + raw = await readFile(filePath) + } catch (err: any) { + // -> A path a diff named but that is no longer there. Nothing to import and nothing wrong. + if (err.code === 'ENOENT') { + continue + } + throw err + } + const declared = deserializePage(raw.toString('utf8'), ext) + const isReservedExtension = Boolean(ext) && reserved.includes(ext) + + if (!declared && !isReservedExtension) { + const asset = await WIKI.models.assets.adoptStoredFile({ + siteId: target.siteId, + locale, + folderPath, + fileName, + data: raw, + authorId: actorId, + overwrite + }) + if (asset) { + summary.assets++ + } else { + summary.skipped++ + } + continue + } + + // -> A page is addressed without its extension: `guides/setup.md` is the page at `guides/setup` + const name = fileName.slice(0, fileName.length - (ext ? ext.length + 1 : 0)) + const meta = declared?.meta ?? {} + try { + const imported = await WIKI.models.pages.adoptStoredPage({ + siteId: target.siteId, + locale, + // -> The root page is filed under a name of its own, and takes the empty path back + path: [...rest, ...(name === ROOT_PAGE_NAME ? [] : [name])].join('/'), + // -> The declaration first, then the file itself: a page written by hand carries no title, + // and its name is the next best thing + title: typeof meta.title === 'string' && meta.title ? meta.title : name, + description: typeof meta.description === 'string' ? meta.description : '', + editor: + typeof meta.editor === 'string' && meta.editor + ? meta.editor + : (pageEditorForExtension(ext) ?? DEFAULT_PAGE_EDITOR), + tags: Array.isArray(meta.tags) ? meta.tags.map(String) : [], + isPublished: meta.published !== false, + // -> Undeclared means there was no front matter to strip, so the file is all body + content: declared?.content ?? raw.toString('utf8').trim(), + createdAt: parseFileDate(meta.dateCreated), + updatedAt: parseFileDate(meta.date), + authorId: actorId, + overwrite + }) + if (imported) { + summary.pages++ + } else { + summary.skipped++ + } + } catch (err: any) { + // -> One unusable file must not stop the rest of the tree from being imported + summary.failed++ + WIKI.logger.warn(`Could not import the page at ${filePath} [ SKIPPED ]`) + WIKI.logger.warn(err.message) + } + } + + return summary +} diff --git a/backend/helpers/storageObjects.ts b/backend/helpers/storageObjects.ts new file mode 100644 index 000000000..c6616ccbc --- /dev/null +++ b/backend/helpers/storageObjects.ts @@ -0,0 +1,268 @@ +import mime from 'mime' +import { assetRelPath, pageRelPath, serializePage } from './storageFiles.ts' +import type { StorageModule, StorageTarget } from '../models/storage.ts' + +/** + * The shared half of every object-store target — S3, Azure Blob Storage, Google Cloud Storage. + * + * All three answer the same four questions (put, get, remove, copy) against a flat namespace of keys, + * and everything above that is identical between them: which key a page or an asset takes, how a + * rename is done where there is no rename, what a bulk export walks. That part lives here, so a + * module is its client and nothing else. + * + * **A key is a path**, the same one the disk target would write — `pathPrefixFor` decides what + * brackets it, and pages and assets sit beside each other in it exactly as they do in a folder. An + * object store has no directories, so the slashes are just characters in a name, which is why there is + * nothing here about creating or pruning them. + * + * Not under `modules/storage/`, for the reason `storageFiles.ts` gives: a directory there without a + * `definition.yml` takes every storage module down with it. + */ + +/** + * A direct-access URL as the shared layer asks for one. + * + * `key` rather than a ref, because signing is about an object and not about the wiki: the store has + * to know what to declare the response as and whether to make the browser save it, both of which the + * wiki knows and the object may not have been stored with. + */ +export interface PresignRequest { + key: string + expiresInSeconds: number + contentType: string + /** The file name to save as, when the browser should save rather than display. */ + downloadAs?: string +} + +/** + * The origin a signed URL should be built on, or null for the store's own. + * + * Normalized to no trailing slash so that a module can always join it to a key with one, however the + * administrator typed it. + */ +export function signingBaseUrl(target: StorageTarget): string | null { + const configured = target.assetDelivery.baseUrl?.trim() + return configured ? configured.replace(/\/+$/, '') : null +} + +/** What a store has to be able to do for `objectStorageModule` to build a target out of it. */ +export interface ObjectStoreClient { + /** Write an object, replacing whatever was at that key. */ + put: (target: StorageTarget, key: string, data: Buffer, contentType: string) => Promise + /** Read one back, or null when the store does not have it. Must not throw for a missing key. */ + get: (target: StorageTarget, key: string) => Promise + /** Drop one. Must not throw for a key that is already gone. */ + remove: (target: StorageTarget, key: string) => Promise + /** + * Copy one key onto another, server-side where the store can. + * + * @returns Whether there was anything at the source. False rather than a throw, because a target + * enabled after an upload legitimately has no copy of the file being moved. + */ + copy: (target: StorageTarget, fromKey: string, toKey: string) => Promise + /** + * Sign a URL a reader can fetch the object from without going through the wiki. + * + * Optional only in the type: all three object stores implement it, and a store that could not + * would declare `isDirectAccessSupported: false` and never be asked. + */ + presign?: (target: StorageTarget, request: PresignRequest) => Promise +} + +/** + * What to declare an object as, so that a store handing it straight to a browser says the right thing. + * + * Guessed from the name rather than taken from the asset, because the reference a target is given + * carries the file's size and kind but not its type — and the name is what the wiki itself resolves + * the served type from, so guessing the same way keeps the two in step. + */ +function contentTypeOf(fileName: string): string { + return mime.getType(fileName) ?? 'application/octet-stream' +} + +/** + * Turn a client into a storage module. + * + * The eight contract methods plus `exportAll`, which is the one action all three declare. A module + * spreads the result and adds nothing, unless its store can do something the others cannot. + */ +export function objectStorageModule(client: ObjectStoreClient): StorageModule { + const module: StorageModule = { + canStore(target, ref) { + return WIKI.models.storage.pathPrefixFor(target.siteId, ref.locale) !== null + }, + + async putAsset(target, ref, data) { + const key = assetRelPath(target, ref) + // -> Guarded rather than skipped: the model asks `canStore` before dispatching a write, so + // reaching this means somebody wrote without asking, and an asset's bytes may exist nowhere + // else + if (!key) { + throw new Error( + `${target.title} has no path for ${ref.locale} content, so ${ref.fileName} cannot be stored there.` + ) + } + await client.put(target, key, data, contentTypeOf(ref.fileName)) + }, + + async getAsset(target, ref) { + const key = assetRelPath(target, ref) + return key ? client.get(target, key) : null + }, + + async deleteAsset(target, ref) { + const key = assetRelPath(target, ref) + if (key) { + await client.remove(target, key) + } + }, + + async moveAsset(target, ref, previous) { + await moveObject( + client, + target, + assetRelPath(target, { ...ref, ...previous }), + assetRelPath(target, ref) + ) + }, + + async putPage(target, ref, page) { + const key = pageRelPath(target, ref) + // -> Unlike an asset, a page with no place here is not worth failing over: it is in the + // database, which is where a page always is, and this copy is the thing the site declined + if (!key) { + return + } + await client.put( + target, + key, + Buffer.from(serializePage(ref, page), 'utf8'), + contentTypeOf(key) + ) + }, + + async deletePage(target, ref) { + const key = pageRelPath(target, ref) + if (key) { + await client.remove(target, key) + } + }, + + async movePage(target, ref, previousPath) { + await moveObject( + client, + target, + pageRelPath(target, { ...ref, path: previousPath }), + pageRelPath(target, ref) + ) + }, + + ...(client.presign + ? { + async presignAsset(target, ref, options) { + const key = assetRelPath(target, ref) + if (!key) { + return null + } + return client.presign!(target, { key, ...options }) + } + } + : {}), + + /** + * Write a copy of everything this target is configured to hold into the store. + * + * How content that predates the target being enabled gets into it: an upload only ever goes to + * the targets enabled at the time, so a store turned on today holds nothing from yesterday. A + * plain copy and nothing more — no database row is touched, nothing is repointed, and running it + * twice does the same work to the same effect. + */ + async exportAll(target: StorageTarget): Promise { + let assets = 0 + let unreadable = 0 + let unstored = 0 + + for (const asset of await WIKI.models.assets.listStoredAssets(target.siteId)) { + const contentType = WIKI.models.storage.contentTypeFor( + target.siteId, + asset.kind, + asset.fileSize + ) + if (!target.contentTypes.activeTypes.includes(contentType)) { + continue + } + if (!assetRelPath(target, asset)) { + unstored++ + continue + } + const data = await WIKI.models.storage.getAsset(asset) + if (!data) { + unreadable++ + continue + } + await module.putAsset(target, asset, data) + assets++ + } + + let pages = 0 + if (target.contentTypes.activeTypes.includes('pages')) { + for (const { ref, content } of await WIKI.models.pages.listForStorage(target.siteId)) { + if (!pageRelPath(target, ref)) { + unstored++ + continue + } + await module.putPage(target, ref, content) + pages++ + } + } + + WIKI.logger.info(`Exported ${assets} asset(s) and ${pages} page(s) to ${target.title} [ OK ]`) + const parts = [] + if (assets > 0 || pages > 0) { + parts.push(`Exported ${pages} page(s) and ${assets} asset(s).`) + } else { + parts.push('There was nothing to export.') + } + if (unreadable > 0) { + parts.push(`${unreadable} asset(s) could not be read and were skipped.`) + } + if (unstored > 0) { + const { primaryLocale } = WIKI.models.storage.pathLayoutFor(target.siteId) + parts.push( + `${unstored} item(s) are not in the ${primaryLocale} locale, which is the only one this site stores.` + ) + } + return parts.join(' ') + } + } + + return module +} + +/** + * Follow a rename, which in an object store is a copy and a delete. + * + * Either end may be nowhere, as on disk: the layout can have no path for a locale, and a move may + * cross into or out of it. Moving *into* it has nothing to copy from; moving out of it leaves an + * object behind at the old key, so that one is a delete. + * + * The delete only happens once the copy has reported success, so a store that fails halfway leaves + * the file at its old key rather than nowhere. + */ +async function moveObject( + client: ObjectStoreClient, + target: StorageTarget, + fromKey: string | null, + toKey: string | null +): Promise { + if (!fromKey) { + return + } + if (!toKey) { + await client.remove(target, fromKey) + return + } + if (await client.copy(target, fromKey, toKey)) { + await client.remove(target, fromKey) + } +} diff --git a/backend/locales/en.json b/backend/locales/en.json index 0f2825795..e6af813f5 100644 --- a/backend/locales/en.json +++ b/backend/locales/en.json @@ -856,21 +856,41 @@ "admin.storage.contentTypes": "Content Types", "admin.storage.contentTypesHint": "Select the type of content that is written to this storage target. The same type can be stored on several targets at once.", "admin.storage.delivery": "Content Delivery", + "admin.storage.deliveryBaseUrl": "Custom Base URL", + "admin.storage.deliveryBaseUrlHint": "Serve links from your own domain or CDN instead of the provider's, e.g. https://files.example.com. The link is signed for that host, so it must be a domain that actually fronts the bucket - a CDN that changes the host on the way through will not work. Leave empty to use the provider's own address.", + "admin.storage.deliveryConfig": "Content Delivery Configuration", + "admin.storage.deliveryConfigHint": "How a reader's request for a file stored here is answered. This only takes effect for the content types this target is selected for under the Content Delivery tab; everything else is served from whichever target is.", + "admin.storage.deliveryDirectWarn": "A direct link works for anyone holding it until it expires, without the wiki's page rules behind it. Keep the expiration short so a link to restricted content cannot usefully be passed on.", + "admin.storage.deliveryExpiration": "Link Expiration", + "admin.storage.deliveryExpirationHint": "How long a direct link stays valid, such as 5m or 1h. At most 7 days, which is as far as these providers will sign.", "admin.storage.deliveryHint": "Choose which storage target each kind of content is served from when a reader requests a file.", + "admin.storage.deliveryModeDirect": "Direct Access via Presigned URLs", + "admin.storage.deliveryModeDirectHint": "Recommended for best performance. The reader is redirected to a signed link and fetches the file straight from the provider, so the bytes never pass through this server.", + "admin.storage.deliveryModeStreaming": "Asset Streaming", + "admin.storage.deliveryModeStreamingHint": "Slower. Every file is read from the provider by this server and sent on to the reader, so all of the traffic goes through the wiki.", "admin.storage.deliveryNoTarget": "No enabled storage target is configured to store this content type.", "admin.storage.deliveryPagesHint": "Pages are always served from the database.", "admin.storage.deliveryRelationHint": "A target can only be chosen here for a content type it is also configured to store, under Targets.", + "admin.storage.directAccessFallback": "Failed Direct Links", + "admin.storage.directAccessFallbackError": "Fail with an error", + "admin.storage.directAccessFallbackHint": "What happens when a target set to hand out direct links cannot sign one, because of expired credentials or a provider outage. Either way the target reports it under Status.", + "admin.storage.directAccessFallbackStream": "Fallback to asset streaming", "admin.storage.inactiveTarget": "Inactive", "admin.storage.largeThreshold": "Large File Size Threshold", "admin.storage.largeThresholdHint": "Assets of this size or larger count as large files, whatever their type, such as videos, zip archives and binaries. Pages never fall into this category, irrespective of their size.", "admin.storage.loadFailed": "Failed to load storage configuration.", + "admin.storage.localePrefix": "Add Locale Prefix", + "admin.storage.localePrefixHint": "Prepend the locale to all pages and assets paths. Otherwise only the primary locale will be stored and directly at the root.", "admin.storage.noActions": "This storage target has no actions that you can execute.", "admin.storage.noConfigOption": "This storage target has no configuration options you can modify.", "admin.storage.notConfigured": "Not Configured", "admin.storage.pagesAndAssets": "Pages and Assets", "admin.storage.pagesOnly": "Pages Only", + "admin.storage.pathLayoutHint": "Changing the path layout moves nothing that is already stored: content stays where the previous layout put it. Use a target's export and import actions to bring it across.", "admin.storage.saveFailed": "Failed to save storage configuration.", "admin.storage.saveSuccess": "Storage configuration saved successfully.", + "admin.storage.sitePrefix": "Add Site ID Prefix", + "admin.storage.sitePrefixHint": "Prepend the site ID to all pages and assets paths. Useful when several sites are stored in the same location.", "admin.storage.stateActive": "Healthy", "admin.storage.stateError": "Error", "admin.storage.stateInactive": "Not in use", @@ -878,6 +898,9 @@ "admin.storage.stateWarning": "Degraded", "admin.storage.status": "Status", "admin.storage.subtitle": "Choose where the content of your wiki is stored and served from", + "admin.storage.syncInterval": "Sync Interval", + "admin.storage.syncIntervalHint": "How often storage targets with a remote are synchronized, such as 5m or 1h. Targets with nothing to synchronize with are unaffected. A sync can always be triggered immediately from a target's Actions.", + "admin.storage.targetConfig": "Target Configuration", "admin.storage.targets": "Targets", "admin.storage.title": "Storage", "admin.system.browser": "Browser", diff --git a/backend/models/assets.ts b/backend/models/assets.ts index abed50261..ee5c6b179 100644 --- a/backend/models/assets.ts +++ b/backend/models/assets.ts @@ -92,6 +92,13 @@ export interface Asset { fileSize: number /** Slash-separated, without a leading or trailing slash. Empty at the site root. */ folderPath: string + /** + * Which locale's tree it sits in. + * + * Part of where the file *is*, so anything addressing the stored copy needs it — a storage target + * brackets its tree by locale unless the site says otherwise. + */ + locale: string title: string hasPreview: boolean createdAt: Date @@ -99,12 +106,9 @@ export interface Asset { } /** - * An asset found by its path, which is the one lookup that has to say which locale it landed on: the - * URL in a page carries none, and the permission rules may be written against one. + * An asset found by its path, rather than by its ID. */ -export interface AssetAtPath extends Asset { - locale: string -} +export interface AssetAtPath extends Asset {} /** * Reduce whatever a client called the file to something safe to store, address and serve. @@ -399,6 +403,7 @@ class Assets { { id: entry.id, siteId, + actorId: authorId, locale, folderPath, fileName: storedName, @@ -431,6 +436,7 @@ class Assets { mimeType: resolvedMime, fileSize: data.length, folderPath, + locale, title: entry.title, hasPreview: Boolean(preview), createdAt: entry.createdAt, @@ -482,7 +488,7 @@ class Assets { authorId: string }): Promise { await WIKI.models.storage.putAsset( - { id, siteId, locale, folderPath, fileName, kind, fileSize: data.length }, + { id, siteId, actorId: authorId, locale, folderPath, fileName, kind, fileSize: data.length }, data ) await WIKI.db @@ -531,6 +537,7 @@ class Assets { mimeType, fileSize: data.length, folderPath, + locale, title, hasPreview: Boolean(preview), createdAt: new Date(), @@ -554,6 +561,7 @@ class Assets { createdAt: assetsTable.createdAt, updatedAt: assetsTable.updatedAt, folderPath: treeTable.folderPath, + locale: treeTable.locale, title: treeTable.title, // -> Only whether there is one: the preview itself can be megabytes, and no caller of this // wants it inlined @@ -682,7 +690,11 @@ class Assets { /** * Where each of these assets sits, as a storage target addresses one. */ - async getStorageRefs(siteId: string, ids: string[]): Promise { + async getStorageRefs( + siteId: string, + ids: string[], + actorId?: string + ): Promise { if (ids.length < 1) { return [] } @@ -702,6 +714,7 @@ class Assets { return rows.map((row) => ({ id: row.id, siteId, + actorId, locale: row.locale, folderPath: decodeTreePath(row.folderPath ?? '') ?? '', fileName: row.fileName, @@ -761,11 +774,13 @@ class Assets { */ async relocateAssets( siteId: string, - moves: { id: string; previous: { locale: string; folderPath: string; fileName: string } }[] + moves: { id: string; previous: { locale: string; folderPath: string; fileName: string } }[], + actorId?: string ): Promise { const refs = await this.getStorageRefs( siteId, - moves.map((move) => move.id) + moves.map((move) => move.id), + actorId ) for (const ref of refs) { const previous = moves.find((move) => move.id === ref.id)?.previous @@ -915,6 +930,7 @@ class Assets { mimeType, fileSize: data.length, folderPath: importedFolderPath, + locale, title: entry.title, hasPreview: Boolean(preview), createdAt: entry.createdAt, @@ -1278,14 +1294,14 @@ class Assets { * * @returns Whether an asset was deleted */ - async deleteAsset(siteId: string, id: string): Promise { + async deleteAsset(siteId: string, id: string, actorId?: string): Promise { const asset = await this.getAsset(siteId, id) if (!asset) { return false } // -> Read before the rows go: where an asset sits is the tree's to say, and the tree row is // about to be deleted along with it - const [ref] = await this.getStorageRefs(siteId, [id]) + const [ref] = await this.getStorageRefs(siteId, [id], actorId) await WIKI.db.delete(assetsTable).where(eq(assetsTable.id, id)) await WIKI.models.tree.deleteEntry(id) if (ref) { @@ -1308,7 +1324,7 @@ class Assets { /** * Delete the assets left behind by a folder deletion, which removed their tree entries already. */ - async deleteOrphaned(siteId: string, entries: DeletedEntry[]): Promise { + async deleteOrphaned(siteId: string, entries: DeletedEntry[], actorId?: string): Promise { if (entries.length < 1) { return } @@ -1335,6 +1351,7 @@ class Assets { await WIKI.models.storage.removeAsset({ id: entry.id, siteId, + actorId, locale: entry.locale, folderPath: entry.folderPath, fileName: entry.fileName, diff --git a/backend/models/jobs.ts b/backend/models/jobs.ts index fb677160c..11e129da4 100644 --- a/backend/models/jobs.ts +++ b/backend/models/jobs.ts @@ -55,6 +55,13 @@ class Jobs { task: 'updateLocales', cron: '0 0 * * *', type: 'system' + }, + { + // -> Every minute, and the task decides which sites are actually due: the interval is a + // per-site setting, so the tick has to be as fine as the shortest one anybody can ask for + task: 'syncStorageTargets', + cron: '* * * * *', + type: 'system' } ]) diff --git a/backend/models/pages.ts b/backend/models/pages.ts index b20cee19f..f21ddd9ab 100644 --- a/backend/models/pages.ts +++ b/backend/models/pages.ts @@ -608,7 +608,7 @@ class Pages { reason: input.reasonForChange }) - const stored = this.toStoragePage(siteId, page, page.content ?? '') + const stored = this.toStoragePage(siteId, actor.id, page, page.content ?? '') await WIKI.models.storage.mirrorPage(stored.ref, stored.content) await WIKI.models.search.indexPage(page.id, locale) @@ -761,7 +761,12 @@ class Pages { // -> The source is whatever this save set it to, else whatever it already was: a save that only // changed the title still rewrites the copy, since the title is in its front matter - const stored = this.toStoragePage(siteId, updated, values.content ?? existing.content ?? '') + const stored = this.toStoragePage( + siteId, + actor.id, + updated, + values.content ?? existing.content ?? '' + ) await WIKI.models.storage.mirrorPage(stored.ref, stored.content) await WIKI.models.search.indexPage(id, updated.locale) @@ -867,7 +872,7 @@ class Pages { // -> Moved and then rewritten, rather than deleted and written afresh: the move is what keeps a // versioned target's history of the file attached to it, and the rewrite is because a move may // carry a new title and always carries a new modification time, both of which are in the copy - const stored = this.toStoragePage(siteId, moved, existingContent ?? '') + const stored = this.toStoragePage(siteId, actor.id, moved, existingContent ?? '') await WIKI.models.storage.relocatePage(stored.ref, page.path) await WIKI.models.storage.mirrorPage(stored.ref, stored.content) @@ -908,6 +913,7 @@ class Pages { await WIKI.models.storage.removePage({ id, siteId, + actorId: actor.id, locale: page.locale, path: page.path, contentType: page.contentType @@ -978,6 +984,7 @@ class Pages { await WIKI.models.storage.removePage({ id: entry.id, siteId, + actorId: actor.id, locale: entry.locale, path, contentType @@ -1083,6 +1090,7 @@ class Pages { */ private toStoragePage( siteId: string, + actorId: string | undefined, page: { id: string locale: string @@ -1102,6 +1110,7 @@ class Pages { ref: { id: page.id, siteId, + actorId, locale: page.locale, path: page.path, contentType: page.contentType @@ -1148,7 +1157,7 @@ class Pages { .from(pagesTable) .where(eq(pagesTable.siteId, siteId)) - return rows.map((row) => this.toStoragePage(siteId, row, row.content ?? '')) + return rows.map((row) => this.toStoragePage(siteId, undefined, row, row.content ?? '')) } /** @@ -1284,6 +1293,7 @@ class Pages { // it had for the moment it existed with the wrong ones const restored = this.toStoragePage( siteId, + actor.id, { ...page, createdAt: createdAt ?? page.createdAt, updatedAt: updatedAt ?? page.updatedAt }, content ) diff --git a/backend/models/sites.ts b/backend/models/sites.ts index fe4a11e43..c40e4b1f5 100644 --- a/backend/models/sites.ts +++ b/backend/models/sites.ts @@ -195,7 +195,11 @@ class Sites { conflictBehavior: 'overwrite' }, storage: { - largeThreshold: '25MB' + largeThreshold: '25MB', + sitePrefix: false, + localePrefix: true, + syncInterval: '5m', + directAccessFallback: 'stream' } }, config @@ -453,7 +457,11 @@ class Sites { conflictBehavior: 'overwrite' }, storage: { - largeThreshold: '25MB' + largeThreshold: '25MB', + sitePrefix: false, + localePrefix: true, + syncInterval: '5m', + directAccessFallback: 'stream' } } }) diff --git a/backend/models/storage.ts b/backend/models/storage.ts index 473963eb1..3193da610 100644 --- a/backend/models/storage.ts +++ b/backend/models/storage.ts @@ -2,7 +2,7 @@ import fs from 'node:fs/promises' import path from 'node:path' import { load } from 'js-yaml' import { and, eq, inArray } from 'drizzle-orm' -import { parseModuleProps } from '../helpers/common.ts' +import { CustomError, parseModuleProps } from '../helpers/common.ts' import { sites as sitesTable, storage as storageTable } from '../db/schema.ts' import type { ModuleProp } from '../helpers/common.ts' import type { AssetKind } from './assets.ts' @@ -25,6 +25,15 @@ const CONTENT_TYPE_BY_KIND: Record = { other: 'others' } +/** What each content type is called in something an uploader reads. */ +const CONTENT_TYPE_LABELS: Record = { + pages: 'pages', + images: 'images', + documents: 'documents', + others: 'other files', + large: 'large files' +} + /** * What counts as a large file on a site that has never said. * @@ -34,6 +43,36 @@ const CONTENT_TYPE_BY_KIND: Record = { */ const DEFAULT_LARGE_THRESHOLD = '25MB' +/** + * Whether a site's tree is filed under a folder named after the site. + * + * Off, because a target belongs to exactly one site: the folder an administrator configured IS this + * site's folder, and a level for the site inside it would be a folder that never has a sibling. It is + * turned on for the one case where that stops being true — two sites pointed at the same place. + */ +const DEFAULT_SITE_PREFIX = false + +/** + * Whether a site's tree is bracketed by locale. + * + * On, because the tree repeats itself across locales: `guides/logo.png` can exist once in each, and + * without the folder all of them would be the same file. A single-locale wiki has no such collision + * and can turn it off to be rid of an `en` folder that never has a sibling either. + */ +const DEFAULT_LOCALE_PREFIX = true + +/** + * How often a target with a remote is synced, on a site that has never said. + * + * Five minutes, which is what the schedule was fixed at before it was configurable. Site-wide rather + * than per target for the same reason as everything else here: the schedule is a property of how + * often this site's content should reach the outside, not of any one place it reaches. + */ +const DEFAULT_SYNC_INTERVAL = '5m' + +/** A sync interval, as it is written: a whole number of minutes or hours. */ +const SYNC_INTERVAL = /^(\d+)\s?(m|h)$/i + /** Bytes in each unit a size threshold may be written with. See `parseSize`. */ const SIZE_UNIT_BYTES: Record = { b: 1, @@ -63,6 +102,36 @@ const TARGET_CACHE_TTL_MS = 30_000 * wiki carries on in that case, which is exactly why it needs saying somewhere an administrator will * see it. */ +export const STORAGE_DELIVERY_MODES = ['streaming', 'direct'] as const + +export type StorageDeliveryMode = (typeof STORAGE_DELIVERY_MODES)[number] + +/** + * How long a direct-access URL lasts on a target that has never said. + * + * Deliberately short. The link works for whoever holds it until it expires, with none of the wiki's + * page rules behind it, so its lifetime is the window in which a reader can pass a restricted file to + * somebody who could not have asked for it. Long enough to load a page and its images; not long + * enough to be worth sharing. + */ +const DEFAULT_LINK_EXPIRATION = '5m' + +/** + * The longest a direct-access URL may be asked to last. + * + * Seven days, which is not this wiki's opinion but every provider's limit: SigV4, a Google V4 + * signature and an Azure user delegation SAS all refuse to sign for longer. + */ +const MAX_LINK_EXPIRATION_MINUTES = 7 * 24 * 60 + +/** What a site does when a target that should sign a URL cannot. See `directAccessUrlFor`. */ +export const STORAGE_DIRECT_ACCESS_FALLBACKS = ['stream', 'error'] as const + +export type StorageDirectAccessFallback = (typeof STORAGE_DIRECT_ACCESS_FALLBACKS)[number] + +/** What a site does about a failed signature when it has never said. */ +const DEFAULT_DIRECT_ACCESS_FALLBACK: StorageDirectAccessFallback = 'stream' + export const STORAGE_TARGET_STATUSES = ['healthy', 'warning', 'error'] as const export type StorageTargetStatus = (typeof STORAGE_TARGET_STATUSES)[number] @@ -101,10 +170,24 @@ export interface StorageDefinition { defaultTypesEnabled: string[] } assetDelivery: { - isStreamingSupported: boolean + /** + * Whether this module can hand a reader a URL to fetch the file from directly. + * + * The object stores can — a presigned URL or a shared access signature — and nothing else does: + * a file on this machine's disk or in a git working copy has no address of its own that a + * browser could reach. There is no `isStreamingSupported` beside it, because every target can be + * read through the wiki; streaming is what a target does when it does not do this. + */ isDirectAccessSupported: boolean - defaultStreamingEnabled: boolean - defaultDirectAccessEnabled: boolean + /** + * Whether a site may nominate this module to answer readers' requests at all. + * + * True for everything but SFTP, which is a place to *put* a copy of the wiki rather than a place + * to read one from: every image on every page would be an SSH round trip. A target like that is + * still written to, still exported to and still imported from — it is only the Content Delivery + * tab it stays out of. + */ + isDeliverySupported: boolean } props: Record actions: StorageAction[] @@ -134,10 +217,29 @@ export interface StorageTarget { activeTypes: string[] } assetDelivery: { - isStreamingSupported: boolean isDirectAccessSupported: boolean - streaming: boolean - directAccess: boolean + /** Whether this target may be nominated to serve content. See the definition's own field. */ + isDeliverySupported: boolean + /** + * How a reader's request for a file is answered from this target. + * + * `streaming` reads the bytes and sends them through the wiki, which every target can do. + * `direct` answers with a redirect to a URL the store signed, so the bytes never pass through + * this server at all — faster, and much of the reason to put content in an object store. Only + * consulted on the target a site has nominated for the content type; see `servedTypes`. + */ + mode: StorageDeliveryMode + /** + * The origin a direct-access URL is built on, in place of the store's own. + * + * A CDN or a custom domain in front of the bucket. The URL becomes `/?` + * whichever store it is, and the signature is made *for that host* rather than translated onto + * it afterwards — see each module's `presignAsset`, because S3 and GCS sign the host and Azure + * does not. Empty means the store's own address. + */ + baseUrl: string + /** How long a direct-access URL stays valid, as `5m` or `2h`. See `parseInterval`. */ + linkExpiration: string /** * The content types this target is the site's delivery source for, i.e. the ones a reader's * request for a file is answered from here. @@ -166,13 +268,59 @@ export interface StorageTargetInput { activeTypes?: string[] } assetDelivery?: { - streaming?: boolean - directAccess?: boolean + mode?: StorageDeliveryMode + baseUrl?: string + linkExpiration?: string servedTypes?: string[] } config?: Record } +/** + * Who a change is attributed to, for a target that records authorship. + * + * Only git has any use for this today, as the author of the commit it makes. Resolved from a user id + * by `actorFor` rather than carried around as a name and an address, so that the models dispatching + * content pass what they already have and only a target that actually needs the identity pays for + * looking it up. + */ +export interface StorageActor { + name: string + email: string +} + +/** + * How a target lays its tree out, which is the same answer for every target of a site. + * + * Two settings and the locale they are read against, resolved together because neither of them means + * anything on its own — see `pathPrefixFor`, which is the only thing that should be applying them, + * and `parseStoredPath`, which reads a path back the same way. Site-wide rather than per target for + * the same reason `largeThreshold` is: a file has to be at the same place in every target's tree, or + * a target enabled later would be looking for content under a layout the one before it never wrote. + */ +export interface StoragePathLayout { + /** Whether the tree is filed under a folder named after the site. */ + sitePrefix: boolean + /** + * Whether the tree is bracketed by locale. + * + * Off means the site stores its primary locale only, at the root: there is nowhere else to put the + * others without a folder to tell them apart. + */ + localePrefix: boolean + /** The locale that is stored at the root when `localePrefix` is off. */ + primaryLocale: string +} + +/** The site-wide half of the storage configuration, i.e. everything that is not on a target. */ +export interface StorageSiteConfigInput { + largeThreshold?: string + sitePrefix?: boolean + localePrefix?: boolean + syncInterval?: string + directAccessFallback?: StorageDirectAccessFallback +} + /** * Where an asset sits, which is all a target needs in order to find its copy of one. * @@ -184,6 +332,13 @@ export interface StorageTargetInput { export interface StorageAssetRef { id: string siteId: string + /** + * Who is making this change, for a target that records authorship — see `actorFor`. + * + * Absent for a change no one person made: a bulk action, or a folder rename that moved a hundred + * files. A target that cares falls back to whatever it is configured to attribute those to. + */ + actorId?: string locale: string /** Slash-separated, without the file name. Empty at the site root. */ folderPath: string @@ -203,6 +358,8 @@ export interface StorageAssetLocation { export interface StoragePageRef { id: string siteId: string + /** Who is making this change. As on `StorageAssetRef`. */ + actorId?: string locale: string /** Slash-separated, with no leading slash and no file extension. */ path: string @@ -240,6 +397,18 @@ export interface StoragePageContent { * database at all, so every target claiming it has to accept the write or the upload fails. */ export interface StorageModule { + /** + * Whether this target has anywhere to put content in that locale, as the site is laid out now. + * + * Only a module addressing content by path has an answer worth giving, which is why it is optional: + * a target that stores by id — the database — holds every locale whatever the layout says. Asked + * *before* any of the fan-out is written, so that a target with no path for a file is skipped + * rather than failing an upload the rest of the site is perfectly able to store. Nothing has gone + * wrong when this answers false: it is the configuration saying so, so it costs the target no + * recorded state and the uploader no error — unless it is the last target left, which is + * `putAsset`'s to report. + */ + canStore?: (target: StorageTarget, ref: { locale: string }) => boolean /** Store this target's copy of an asset's bytes, replacing any copy it already had. */ putAsset: (target: StorageTarget, ref: StorageAssetRef, data: Buffer) => Promise /** Read its copy back. Null when this target does not have one. */ @@ -258,6 +427,30 @@ export interface StorageModule { deletePage: (target: StorageTarget, ref: StoragePageRef) => Promise /** Follow a move, `ref` being where the page now is. */ movePage: (target: StorageTarget, ref: StoragePageRef, previousPath: string) => Promise + /** + * A URL a reader can fetch this asset from directly, signed by the store. + * + * Only the object stores implement it, and only they declare `isDirectAccessSupported`. What comes + * back is handed to the reader as a redirect, so it has to carry everything the response needs — + * the type to serve it as, and whether the browser should display it or save it — because after + * the redirect the wiki is no longer in the conversation. + * + * @param expiresInSeconds How long the URL must stay valid. Never longer than seven days, which is + * as far as any of these providers will sign. + * @returns Null when this target cannot sign for this file, which the caller treats the same way as + * a failure — see `directAccessUrlFor` + */ + presignAsset?: ( + target: StorageTarget, + ref: StorageAssetRef, + options: { + expiresInSeconds: number + /** What to declare the response as, i.e. the asset's own mime type. */ + contentType: string + /** Set when the browser should save the file rather than display it. The file name to save as. */ + downloadAs?: string + } + ) => Promise /** Handlers named by the definition's actions. */ [handler: string]: any } @@ -276,6 +469,21 @@ export function parseSize(value: string): number { return Number(match[1]) * SIZE_UNIT_BYTES[match[2].toLowerCase()] } +/** + * A sync interval as whole minutes. + * + * @returns 0 for anything unparseable, which the scheduled task reads as "never" — a site whose + * interval nobody can make sense of is left alone rather than synced every tick + */ +export function parseInterval(value: string): number { + const match = SYNC_INTERVAL.exec(String(value ?? '').trim()) + if (!match) { + return 0 + } + const amount = Number(match[1]) + return match[2].toLowerCase() === 'h' ? amount * 60 : amount +} + /** * Storage model * @@ -303,6 +511,9 @@ class Storage { /** Configured targets, keyed by site. See `TARGET_CACHE_TTL_MS` for what keeps this honest. */ targetCache = new Map() + /** Resolved authors, keyed by user id. See `actorFor`. */ + actorCache = new Map() + /** * Load the storage module definitions from disk. */ @@ -391,8 +602,11 @@ class Storage { activeTypes: definition.contentTypes?.defaultTypesEnabled ?? [] }, assetDelivery: { - streaming: definition.assetDelivery?.defaultStreamingEnabled ?? false, - directAccess: definition.assetDelivery?.defaultDirectAccessEnabled ?? false, + // -> Streaming whatever the module can do: reading through the wiki is the behaviour that + // needs no configuration and leaks no links, so direct access is opted into + mode: 'streaming' satisfies StorageDeliveryMode, + baseUrl: '', + linkExpiration: DEFAULT_LINK_EXPIRATION, // -> A new target serves nothing until a site says so: content already uploaded is not on // it, so nominating it as a source on its own would answer 404 for every existing file servedTypes: definition.key === DB_MODULE ? [...CONTENT_TYPES] : [] @@ -476,10 +690,14 @@ class Storage { activeTypes: contentTypes.activeTypes ?? [] }, assetDelivery: { - isStreamingSupported: definition.assetDelivery?.isStreamingSupported ?? false, isDirectAccessSupported: definition.assetDelivery?.isDirectAccessSupported ?? false, - streaming: assetDelivery.streaming ?? false, - directAccess: assetDelivery.directAccess ?? false, + // -> Serving is the ordinary thing for a target to do, so a module has to opt *out* + isDeliverySupported: definition.assetDelivery?.isDeliverySupported ?? true, + mode: (STORAGE_DELIVERY_MODES as readonly string[]).includes(assetDelivery.mode) + ? (assetDelivery.mode as StorageDeliveryMode) + : 'streaming', + baseUrl: assetDelivery.baseUrl ?? '', + linkExpiration: assetDelivery.linkExpiration || DEFAULT_LINK_EXPIRATION, servedTypes: assetDelivery.servedTypes ?? [] }, props: definition.props, @@ -627,6 +845,40 @@ class Storage { return `${definition.title} cannot serve ${unstored}, as it is not configured to store them.` } } + if ( + servedTypes && + servedTypes.length > 0 && + definition.assetDelivery.isDeliverySupported === false + ) { + return `${definition.title} cannot be a delivery source: it is a place to keep a copy of this site's content, not one to serve it from.` + } + const delivery = patch.assetDelivery + if (delivery?.mode && !(STORAGE_DELIVERY_MODES as readonly string[]).includes(delivery.mode)) { + return `"${delivery.mode}" is not a valid delivery mode.` + } + if (delivery?.mode === 'direct' && !definition.assetDelivery.isDirectAccessSupported) { + return `${definition.title} cannot hand out direct links, so its content has to be streamed.` + } + if (delivery?.linkExpiration !== undefined) { + const minutes = parseInterval(delivery.linkExpiration) + if (minutes < 1) { + return `"${delivery.linkExpiration}" is not a valid link expiration. Use a whole number of minutes or hours, such as "5m" or "1h".` + } + if (minutes > MAX_LINK_EXPIRATION_MINUTES) { + return 'A direct link cannot be valid for more than 7 days, which is the longest any of these providers will sign for.' + } + } + if (delivery?.baseUrl) { + let parsed: URL + try { + parsed = new URL(delivery.baseUrl) + } catch { + return `"${delivery.baseUrl}" is not a valid Custom Base URL. Give the full origin, such as "https://files.example.com".` + } + if (!['http:', 'https:'].includes(parsed.protocol)) { + return 'The Custom Base URL must be an http or https address.' + } + } return this.validateConfig(target.module, patch.config) } @@ -670,17 +922,19 @@ class Storage { nomination too, and a patch that nominates types while turning it off must not win. */ const willBeEnabled = patch.isEnabled ?? target.isEnabled - const servedTypes = willBeEnabled - ? (patch.assetDelivery?.servedTypes ?? target.assetDelivery.servedTypes) - : [] + const servedTypes = + willBeEnabled && definition.assetDelivery.isDeliverySupported + ? (patch.assetDelivery?.servedTypes ?? target.assetDelivery.servedTypes) + : [] if (patch.assetDelivery || (!willBeEnabled && target.assetDelivery.servedTypes.length > 0)) { + const mode = patch.assetDelivery?.mode ?? target.assetDelivery.mode values.assetDelivery = { - streaming: - definition.assetDelivery.isStreamingSupported && - (patch.assetDelivery?.streaming ?? target.assetDelivery.streaming), - directAccess: - definition.assetDelivery.isDirectAccessSupported && - (patch.assetDelivery?.directAccess ?? target.assetDelivery.directAccess), + // -> A module that cannot sign a URL is stored as streaming whatever was asked for, the same + // way an unsupported capability has always been handled here: it is the module's answer + // to give, not the client's + mode: definition.assetDelivery.isDirectAccessSupported ? mode : 'streaming', + baseUrl: patch.assetDelivery?.baseUrl ?? target.assetDelivery.baseUrl, + linkExpiration: patch.assetDelivery?.linkExpiration ?? target.assetDelivery.linkExpiration, servedTypes } } @@ -712,35 +966,147 @@ class Storage { } /** - * Check a site-wide storage setting. + * How this site's targets lay their tree out. See `StoragePathLayout`. + */ + pathLayoutFor(siteId: string): StoragePathLayout { + const config = WIKI.sites[siteId]?.config + return { + sitePrefix: config?.storage?.sitePrefix ?? DEFAULT_SITE_PREFIX, + localePrefix: config?.storage?.localePrefix ?? DEFAULT_LOCALE_PREFIX, + primaryLocale: config?.locales?.primary ?? 'en' + } + } + + /** + * What this site does when a direct link cannot be signed. See `directAccessFailed`. + */ + directAccessFallbackFor(siteId: string): StorageDirectAccessFallback { + const configured = WIKI.sites[siteId]?.config?.storage?.directAccessFallback + return (STORAGE_DIRECT_ACCESS_FALLBACKS as readonly string[]).includes(configured) + ? (configured as StorageDirectAccessFallback) + : DEFAULT_DIRECT_ACCESS_FALLBACK + } + + /** + * How often this site's targets are synced, in whole minutes. * - * @returns The reason it is invalid, or null when it is fine + * Read by the scheduled task rather than by any module: a module is asked to sync and does, and how + * often that happens is not its business. */ - validateSiteConfig(patch: { largeThreshold?: string }): string | null { + syncIntervalFor(siteId: string): number { + return parseInterval(WIKI.sites[siteId]?.config?.storage?.syncInterval ?? DEFAULT_SYNC_INTERVAL) + } + + /** + * The leading segments of every path a target writes for this site. + * + * The one place the layout is applied, so that a page and an asset of the same folder land beside + * each other whatever it is set to, and so that a module reading a file back looks where the module + * that wrote it put it. What follows is the target's own business: the folders of the tree, and then + * a file name each kind of content decides for itself. + * + * @returns Null for content the layout has no place for — a secondary locale on a site storing only + * its primary one. Not an error: it is what the site asked for, and each operation decides what + * that means for it. A write is the one that cannot shrug (`putAsset` in the disk module). + */ + pathPrefixFor(siteId: string, locale: string): string[] | null { + const layout = this.pathLayoutFor(siteId) + const prefix = layout.sitePrefix ? [siteId] : [] + if (layout.localePrefix) { + return [...prefix, locale] + } + return locale === layout.primaryLocale ? prefix : null + } + + /** + * Read a stored path back: which locale it belongs to, and where it sits under the prefix. + * + * The exact inverse of `pathPrefixFor`, and here rather than in the module that walks a folder + * because the two have to agree — an import that reads a path differently from the way it was + * written takes content in under the wrong name. + * + * @param segments The path relative to the target's root, already split, file name included + * @returns The remaining segments and the locale they are in, or null for a path that is not part + * of this site's tree: another site's folder, or a file sitting where no locale can be read off it + */ + parseStoredPath( + siteId: string, + segments: string[] + ): { locale: string; segments: string[] } | null { + const layout = this.pathLayoutFor(siteId) + let rest = segments + if (layout.sitePrefix) { + // -> Which is what makes two sites able to share a folder: each ignores the other's half of it + if (rest[0] !== siteId) { + return null + } + rest = rest.slice(1) + } + let locale = layout.primaryLocale + if (layout.localePrefix) { + // -> A file straight in the root is outside the layout and belongs to no locale + if (rest.length < 2) { + return null + } + locale = rest[0] + rest = rest.slice(1) + } + return rest.length > 0 ? { locale, segments: rest } : null + } + + /** + * Check the site-wide storage settings. + * + * @returns The reason they are invalid, or null when they are fine + */ + validateSiteConfig(patch: StorageSiteConfigInput): string | null { if ( patch.largeThreshold !== undefined && !/^\d+(\.\d+)?\s?(B|KB|MB|GB|TB)$/i.test(patch.largeThreshold) ) { return `"${patch.largeThreshold}" is not a valid size threshold. Use a size such as "5MB".` } + if (patch.syncInterval !== undefined && parseInterval(patch.syncInterval) < 1) { + return `"${patch.syncInterval}" is not a valid sync interval. Use a whole number of minutes or hours, such as "5m" or "1h".` + } + if ( + patch.directAccessFallback !== undefined && + !(STORAGE_DIRECT_ACCESS_FALLBACKS as readonly string[]).includes(patch.directAccessFallback) + ) { + return `"${patch.directAccessFallback}" is not a valid answer for a failed direct link.` + } return null } /** - * Write a site-wide storage setting. + * Write the site-wide storage settings. * * Goes through the sites model rather than the table, so that the sites cache — which is where - * `largeThresholdFor` reads it back from — is reloaded with it. + * `largeThresholdFor` and `pathLayoutFor` read them back from — is reloaded with them. + * + * Nothing is moved. The layout settings decide where content is written and looked for from now on, + * and every file already stored stays where the previous layout put it — which is what the disk + * target's export and import actions are for. * * @returns Whether anything was written */ - async updateSiteConfig(siteId: string, patch: { largeThreshold?: string }): Promise { - if (patch.largeThreshold === undefined) { + async updateSiteConfig(siteId: string, patch: StorageSiteConfigInput): Promise { + const config: Record = {} + for (const key of [ + 'largeThreshold', + 'sitePrefix', + 'localePrefix', + 'syncInterval', + 'directAccessFallback' + ] as const) { + if (patch[key] !== undefined) { + config[key] = patch[key] + } + } + if (Object.keys(config).length < 1) { return false } - const updated = await WIKI.models.sites.updateSite(siteId, { - config: { storage: { largeThreshold: patch.largeThreshold } } - }) + const updated = await WIKI.models.sites.updateSite(siteId, { config: { storage: config } }) // -> A target's content type depends on the threshold, so the resolved list is now stale this.targetCache.delete(siteId) return updated @@ -816,8 +1182,23 @@ class Storage { kind: AssetKind, fileSize: number ): Promise { - const targets = await this.writeTargetsFor(siteId, kind, fileSize) const contentType = this.contentTypeFor(siteId, kind, fileSize) + /* + A target that may not be nominated goes to the very back, behind even the database. + + It is still in the list, and deliberately: `offloadUnchecked` can leave a file whose only copy + is on one of these, and answering a reader that their image is gone when it is sitting on the + other end of an SSH connection would be worse than the round trip. Last resort is the whole of + the role — reached only once every target that is allowed to serve has been asked and had + nothing. + */ + const targets = (await this.writeTargetsFor(siteId, kind, fileSize)).sort((a, b) => + a.assetDelivery.isDeliverySupported === b.assetDelivery.isDeliverySupported + ? 0 + : a.assetDelivery.isDeliverySupported + ? -1 + : 1 + ) const source = targets.find((t) => (t.assetDelivery.servedTypes ?? []).includes(contentType)) ?? targets.find((t) => t.module === DB_MODULE) @@ -825,24 +1206,55 @@ class Storage { } /** - * Write an asset's bytes to every target that holds its kind. + * Write an asset's bytes to every target that holds its kind, and that has somewhere to put them. * - * @throws When any of them refuses, which fails the upload. Unlike a page, an asset may have no - * copy in the database to fall back on, so a half-stored asset is not something to report as - * success — the caller undoes the rest of the upload. + * Two different things can go wrong here and they are not the same failure. A target that + * **cannot** hold this file — the site's layout gives it no path for the locale, per `canStore` — + * is simply not asked: nothing is broken, and as long as one other target takes the bytes the + * upload succeeds and the asset is stored. A target that **refuses** the write it was given is a + * fault, and it fails the upload. + * + * @throws A `CustomError` when nothing can hold the file, which is a configuration the uploader + * cannot be expected to work out from a failure — see `unstorableAssetError`. Also whatever a + * target throws on a write: unlike a page, an asset may have no copy in the database to fall + * back on, so a half-stored asset is not something to report as success and the caller undoes + * the rest of the upload. */ async putAsset(ref: StorageAssetRef, data: Buffer): Promise { + const contentType = this.contentTypeFor(ref.siteId, ref.kind, ref.fileSize) const targets = await this.writeTargetsFor(ref.siteId, ref.kind, ref.fileSize) if (targets.length < 1) { - throw new Error( - `This site has no storage target configured to hold ${this.contentTypeFor(ref.siteId, ref.kind, ref.fileSize)}.` + throw new CustomError( + 'assetNoStorageTarget', + `This site has no storage target configured to hold ${CONTENT_TYPE_LABELS[contentType]}. Enable one under Storage > Targets.`, + 409 ) } + + const accepting: { target: StorageTarget; mod: StorageModule }[] = [] + const declining: StorageTarget[] = [] for (const target of targets) { const mod = await this.ensureModule(target.module) if (!mod) { throw new Error(`The ${target.title} storage module has no implementation installed.`) } + if (mod.canStore && !mod.canStore(target, ref)) { + declining.push(target) + continue + } + accepting.push({ target, mod }) + } + if (accepting.length < 1) { + throw this.unstorableAssetError(ref, contentType, declining) + } + if (declining.length > 0) { + // -> Not a warning on the target: it did what the site configured it to do + WIKI.logger.debug( + `No path for ${ref.locale} content in ${declining.map((t) => t.title).join(', ')}, so ${ref.fileName} was not written there.` + ) + } + + for (const { target, mod } of accepting) { try { await mod.putAsset(target, ref, data) } catch (err: any) { @@ -854,6 +1266,42 @@ class Storage { } } + /** + * Why an asset has nowhere to go, as something the person uploading it can act on. + * + * Worth this much prose because nothing has failed: every target is healthy, the file is a + * perfectly ordinary one, and the site is simply configured so that no target will hold it. A + * generic "upload failed" leaves an administrator with nothing to look at, and the state cards + * with nothing to show — so the message has to name the setting that closed the door and the two + * ways of opening it. + * + * The locale is the reason today and `canStore` is only implemented for it, but the fallback is + * not decoration: a later module declining for a reason of its own would otherwise be reported as + * a locale problem it has nothing to do with. + */ + private unstorableAssetError( + ref: StorageAssetRef, + contentType: ContentType, + declining: StorageTarget[] + ): CustomError { + const label = CONTENT_TYPE_LABELS[contentType] + const names = declining.map((t) => t.title).join(', ') + const them = declining.length > 1 ? 'those targets hold' : 'that target holds' + if (this.pathPrefixFor(ref.siteId, ref.locale) === null) { + const { primaryLocale } = this.pathLayoutFor(ref.siteId) + return new CustomError( + 'assetLocaleNotStored', + `This site stores ${label} in ${names} only, and with "Add Locale Prefix" turned off ${them} the ${primaryLocale} locale and no other - so there is nowhere to put a file in ${ref.locale}. Turn "Add Locale Prefix" back on under Storage > Configuration, or have the database store ${label} as well.`, + 409 + ) + } + return new CustomError( + 'assetNoStorageTarget', + `No storage target on this site can hold ${ref.fileName}: ${names} store ${label} but declined it.`, + 409 + ) + } + /** * Read an asset's bytes, from the target the site serves that content type from. * @@ -888,6 +1336,99 @@ class Storage { return null } + /** + * A URL to send the reader to instead of the bytes, or null to stream them as usual. + * + * The whole of the direct-access decision, in one place because two routes need exactly the same + * answer — the public `/_files/` path every page image goes through, and the file manager's + * download button. + * + * **Only the nominated delivery source counts.** `deliveryTargetsFor` answers with a fallback list + * whose head may be the database standing in for a type nobody nominated, and standing in is not + * the same as being chosen: a target only hands out links for a content type the site explicitly + * pointed at it, which is what the Content Delivery tab sets. + * + * A link works for whoever holds it until it expires, with none of the wiki's page rules behind it. + * That is inherent to the feature rather than an oversight — it is what makes the store able to + * serve the file without asking the wiki — and it is why the expiry defaults to minutes. + * + * @returns The URL and how long it lasts — the caller needs the second to keep a cached redirect + * from outliving it — or null when the caller should stream the file itself + * @throws When signing failed and the site is configured to fail rather than fall back + */ + async directAccessUrlFor( + ref: StorageAssetRef, + options: { contentType: string; downloadAs?: string } + ): Promise<{ url: string; expiresInSeconds: number } | null> { + const [source] = await this.deliveryTargetsFor(ref.siteId, ref.kind, ref.fileSize) + if (!source || source.assetDelivery.mode !== 'direct') { + return null + } + const contentType = this.contentTypeFor(ref.siteId, ref.kind, ref.fileSize) + if (!source.assetDelivery.servedTypes.includes(contentType)) { + return null + } + const mod = await this.ensureModule(source.module) + if (typeof mod?.presignAsset !== 'function') { + return this.directAccessFailed( + source, + ref, + `${source.title} is set to hand out direct links but cannot sign one.` + ) + } + + // -> Clamped rather than trusted: the stored value is validated on the way in, but a provider + // refusing the whole request over an out-of-range expiry is a poor way to find that out + const minutes = Math.min( + Math.max(parseInterval(source.assetDelivery.linkExpiration), 1), + MAX_LINK_EXPIRATION_MINUTES + ) + const expiresInSeconds = minutes * 60 + try { + const url = await mod.presignAsset(source, ref, { expiresInSeconds, ...options }) + if (!url) { + return this.directAccessFailed( + source, + ref, + `${source.title} could not sign a link for ${ref.fileName}.` + ) + } + await this.recordState(source, 'healthy') + return { url, expiresInSeconds } + } catch (err: any) { + return this.directAccessFailed( + source, + ref, + `Could not sign a link for ${ref.fileName}: ${err.message}` + ) + } + } + + /** + * What a site does when the target that should have signed a link did not. + * + * Either answer is defensible, which is why it is a setting rather than a decision made here. + * Falling back keeps every image on every page loading while somebody fixes the credentials, at + * the cost of quietly serving everything the slow way; failing makes the misconfiguration + * impossible to miss, at the cost of a visibly broken wiki. The target records a warning + * regardless, so the Status card says so under either. + * + * @returns Null, meaning stream it + * @throws When the site asked to be told + */ + private async directAccessFailed( + target: StorageTarget, + ref: StorageAssetRef, + message: string + ): Promise { + WIKI.logger.warn(`${message} [ ${this.directAccessFallbackFor(ref.siteId).toUpperCase()} ]`) + await this.recordState(target, 'warning', message) + if (this.directAccessFallbackFor(ref.siteId) === 'error') { + throw new Error(message) + } + return null + } + /** * Drop every target's copy of an asset. Never throws over bytes that are already gone. */ @@ -1041,6 +1582,33 @@ class Storage { ) } + /** + * Who to attribute a change to, for the one kind of target that records it. + * + * Held indefinitely once looked up, which is the whole reason this is a method and not a join: a + * page save is a hot path, a name and an address change about never, and the cost of being a day + * out of date on a commit author is nothing at all. + * + * @returns Null for a change nothing attributed, or a user who has since been deleted. The caller + * decides what to put in its place — for git, the target's configured default author. + */ + async actorFor(actorId?: string | null): Promise { + if (!actorId) { + return null + } + const cached = this.actorCache.get(actorId) + if (cached) { + return cached + } + const user = await WIKI.models.users.getById(actorId) + if (!user) { + return null + } + const actor: StorageActor = { name: user.name, email: user.email } + this.actorCache.set(actorId, actor) + return actor + } + /** * Ensure a module's implementation is loaded * @@ -1100,6 +1668,54 @@ class Storage { await WIKI.db.update(storageTable).set({ state }).where(eq(storageTable.id, target.id)) } + /** + * How this site's storage is behaving, as little as a status indicator needs to know. + * + * Answered from the target cache rather than the table on purpose: `recordState` patches the cached + * object in place as it writes the row, so the cache is current for exactly the field this reads, + * and this is the one storage call something outside the storage page makes. + * + * **Only enabled targets count.** A target that is off is not being asked to do anything, so what + * it last recorded is history — and disabling a target is a perfectly ordinary way to deal with one + * that is broken, which must not leave the wiki reporting itself as degraded forever. + */ + async healthFor( + siteId: string + ): Promise<{ id: string; title: string; isEnabled: boolean; state: { status: string } }[]> { + return (await this.getCachedSiteTargets(siteId)) + .filter((t) => t.isEnabled) + .map((t) => ({ + id: t.id, + title: t.title, + isEnabled: t.isEnabled, + state: { status: t.state.status } + })) + } + + /** + * Every target across every site that has a `sync` to run, for the scheduled sync task. + * + * Driven off the sites cache rather than a query over the table, so that a site removed from this + * instance's view is not synced by it. A target whose module declares no `sync` handler — the + * database, the local disk — is not a target with a remote to fall out of step with, and is simply + * not in the list. + */ + async syncableTargets(): Promise { + const syncable: StorageTarget[] = [] + for (const siteId of Object.keys(WIKI.sites ?? {})) { + for (const target of await this.getCachedSiteTargets(siteId)) { + if (!target.isEnabled) { + continue + } + const mod = await this.ensureModule(target.module) + if (typeof mod?.sync === 'function') { + syncable.push(target) + } + } + } + return syncable + } + /** * Run one of the actions a module declares. * diff --git a/backend/models/tree.ts b/backend/models/tree.ts index b434b4dd5..12d9f9a98 100644 --- a/backend/models/tree.ts +++ b/backend/models/tree.ts @@ -843,11 +843,14 @@ class Tree { async renameFolder({ folderId, pathName, - title + title, + actorId }: { folderId: string pathName: string title: string + /** Who is renaming it, for a target that records who moved a file. */ + actorId?: string }): Promise { const folder = await this.getFolderById(folderId) if (!folder) { @@ -936,6 +939,7 @@ class Tree { { id: page.id, siteId: folder.siteId, + actorId, locale: page.locale, path: page.path, contentType: page.contentType @@ -975,7 +979,8 @@ class Tree { ), fileName: row.fileName } - })) + })), + actorId ) // -> Every asset under it is served from a different path now, and nothing about the assets diff --git a/backend/models/users.ts b/backend/models/users.ts index 1faf22635..9b7219af8 100644 --- a/backend/models/users.ts +++ b/backend/models/users.ts @@ -238,6 +238,34 @@ class Users { return res?.[0] ?? null } + /** + * Who unattended work is recorded as having done it. + * + * Content arriving without a person behind it still has to name an author — a page pulled in by a + * scheduled storage sync lands in the wiki as an ordinary page, and an ordinary page has an author. + * The wiki's own longest-standing administrator is the least surprising answer: it is an account + * that exists on every instance, and one whose owner is entitled to have created the content. + * + * @returns The user id, or null on an instance with no active administrator at all + */ + async getSystemActorId(): Promise { + const rows = await WIKI.db + .select({ id: usersTable.id }) + .from(usersTable) + .innerJoin(userGroups, eq(userGroups.userId, usersTable.id)) + .innerJoin(groupsTable, eq(groupsTable.id, userGroups.groupId)) + .where( + and( + eq(usersTable.isActive, true), + eq(usersTable.isSystem, false), + sql`${groupsTable.permissions} @> '["manage:system"]'::jsonb` + ) + ) + .orderBy(usersTable.createdAt) + .limit(1) + return rows[0]?.id ?? null + } + async getById(id: string) { const res = await WIKI.db.select().from(usersTable).where(eq(usersTable.id, id)).limit(1) return res?.[0] ?? null diff --git a/backend/modules/storage/azure/definition.yml b/backend/modules/storage/azure/definition.yml new file mode 100644 index 000000000..6d639a7a9 --- /dev/null +++ b/backend/modules/storage/azure/definition.yml @@ -0,0 +1,49 @@ +key: azure +title: Azure Blob Storage +icon: '/_assets/icons/ultraviolet-azure.svg' +banner: '/_assets/storage/azure.jpg' +description: Azure Blob Storage is Microsoft's object storage solution for the cloud, optimized for storing large amounts of unstructured data. +assetDelivery: + isDirectAccessSupported: true +contentTypes: + defaultTypesEnabled: ['images', 'documents', 'others', 'large'] +props: + accountName: + type: String + title: Account Name + default: '' + hint: Your unique storage account name. + icon: 3d-touch + order: 1 + accountKey: + type: String + title: Account Access Key + default: '' + hint: Either key 1 or key 2 from the storage account. Leave empty to use the credentials the machine already has, such as a managed identity. + icon: key + sensitive: true + order: 2 + containerName: + type: String + title: Container Name + default: wiki + hint: The container to store content in. It is created on first use if it does not exist yet. + icon: shipping-container + order: 3 + accessTier: + type: String + title: Access Tier + default: Cool + hint: What new blobs are stored as. Cool costs less to keep and more to read, which suits content that is served from another target and kept here as a copy. + icon: scan-stock + order: 4 + enum: + - Hot|Hot + - Cool|Cool + - Cold|Cold + - Archive|Archive +actions: + exportAll: + label: Export Everything + hint: Write a copy of every page and asset this target is configured to hold to the container, overwriting whatever is already there. Nothing in the database is changed and nothing is moved, so this is how content created before the target was enabled gets onto it. + icon: this-way-up diff --git a/backend/modules/storage/azure/storage.ts b/backend/modules/storage/azure/storage.ts new file mode 100644 index 000000000..efc55b279 --- /dev/null +++ b/backend/modules/storage/azure/storage.ts @@ -0,0 +1,171 @@ +import { + BlobSASPermissions, + BlobServiceClient, + StorageSharedKeyCredential, + generateBlobSASQueryParameters +} from '@azure/storage-blob' +import { DefaultAzureCredential } from '@azure/identity' +import { objectStorageModule, signingBaseUrl } from '../../../helpers/storageObjects.ts' +import type { ContainerClient } from '@azure/storage-blob' +import type { ObjectStoreClient } from '../../../helpers/storageObjects.ts' +import type { StorageTarget } from '../../../models/storage.ts' + +/** Live container clients, keyed by target, plus whether the container has been ensured. */ +const containers = new Map() + +/** The settings a client is built from — a change to any of them needs a new one. */ +function configFingerprint(target: StorageTarget): string { + const c = target.config + return JSON.stringify([c.accountName, c.accountKey, c.containerName]) +} + +/** + * The container client for this target, created once and kept. + * + * The container is created on the way, which is the one place these three modules differ in what they + * will do for you: a container is namespaced under the storage account and costs nothing to make, + * whereas an S3 bucket is a global name and a GCS bucket is billable, so both of those are the + * administrator's to create. + * + * **The account key is optional.** Left empty, `DefaultAzureCredential` is used instead — a managed + * identity on an Azure VM or container app, or the standard `AZURE_*` environment variables — which is + * how a deployment avoids putting a long-lived key in the database at all. + */ +async function containerFor(target: StorageTarget): Promise { + const fingerprint = configFingerprint(target) + const cached = containers.get(target.id) + if (cached && cached.fingerprint === fingerprint) { + return cached.container + } + const { accountName, accountKey, containerName } = target.config + const url = `https://${accountName}.blob.core.windows.net` + const service = accountKey + ? new BlobServiceClient(url, new StorageSharedKeyCredential(accountName, accountKey)) + : new BlobServiceClient(url, new DefaultAzureCredential()) + const container = service.getContainerClient(containerName || 'wiki') + await container.createIfNotExists() + containers.set(target.id, { container, fingerprint }) + return container +} + +/** + * A user delegation key, for an account authenticating as itself rather than with a shared key. + * + * The managed-identity path: with no account key there is nothing to sign a SAS with, so Azure is + * asked for a short-lived key to sign with instead. It needs the **Storage Blob Delegator** role on + * the account, and it is what makes direct access work without a long-lived secret in the database. + * + * Cached until shortly before it expires, since fetching one is a round trip and every image on every + * page would otherwise pay for it. + */ +const delegationKeys = new Map() + +/** How long a delegation key is asked for, and how much of that is left unused as a safety margin. */ +const DELEGATION_KEY_MINUTES = 60 +const DELEGATION_KEY_MARGIN_MS = 5 * 60_000 + +async function delegationKeyFor(target: StorageTarget): Promise { + const cached = delegationKeys.get(target.id) + const now = Date.now() + if (cached && cached.expiresAt - DELEGATION_KEY_MARGIN_MS > now) { + return cached.key + } + const service = new BlobServiceClient( + `https://${target.config.accountName}.blob.core.windows.net`, + new DefaultAzureCredential() + ) + const expiresAt = now + DELEGATION_KEY_MINUTES * 60_000 + const key = await service.getUserDelegationKey(new Date(now), new Date(expiresAt)) + delegationKeys.set(target.id, { key, expiresAt }) + return key +} + +/** Whether the service is telling us the blob simply is not there. */ +function isNotFound(err: any): boolean { + return err?.statusCode === 404 || err?.details?.errorCode === 'BlobNotFound' +} + +const azureClient: ObjectStoreClient = { + async put(target, key, data, contentType) { + const blob = (await containerFor(target)).getBlockBlobClient(key) + await blob.uploadData(data, { + blobHTTPHeaders: { blobContentType: contentType }, + ...(target.config.accessTier ? { tier: target.config.accessTier } : {}) + }) + }, + + async get(target, key) { + try { + return await (await containerFor(target)).getBlockBlobClient(key).downloadToBuffer() + } catch (err: any) { + if (isNotFound(err)) { + // -> This target does not have the file: enabled after the upload, or removed from outside + // the wiki. Not a fault — the caller asks the next target. + return null + } + throw err + } + }, + + async remove(target, key) { + await (await containerFor(target)).getBlockBlobClient(key).deleteIfExists() + }, + + async copy(target, fromKey, toKey) { + const container = await containerFor(target) + const source = container.getBlockBlobClient(fromKey) + if (!(await source.exists())) { + return false + } + // -> Server-side, and awaited: the destination has to be complete before the caller deletes the + // source, and `beginCopyFromURL` is only a promise that the copy has *started* + const copy = await container.getBlockBlobClient(toKey).beginCopyFromURL(source.url) + await copy.pollUntilDone() + return true + }, + + async presign(target, { key, expiresInSeconds, contentType, downloadAs }) { + const { accountName, accountKey, containerName } = target.config + const container = containerName || 'wiki' + const now = Date.now() + const values = { + containerName: container, + blobName: key, + permissions: BlobSASPermissions.parse('r'), + // -> A minute of slack at the front, because the reader's clock and Azure's need not agree and + // a SAS that is not valid yet fails exactly as hard as one that has expired + startsOn: new Date(now - 60_000), + expiresOn: new Date(now + expiresInSeconds * 1000), + contentType, + ...(downloadAs + ? { contentDisposition: `attachment; filename="${encodeURIComponent(downloadAs)}"` } + : {}) + } + + const sas = accountKey + ? generateBlobSASQueryParameters( + values, + new StorageSharedKeyCredential(accountName, accountKey) + ) + : generateBlobSASQueryParameters(values, await delegationKeyFor(target), accountName) + + /* + Azure signs the canonicalized resource — the account, the container and the blob — and not the + host, which is the one thing that makes this simpler than S3 and GCS: a CDN or Front Door + endpoint in front of the container can be put in front of a signature made for the account, and + Azure still validates it when the request reaches the origin. + */ + const base = signingBaseUrl(target) + const origin = base ?? `https://${accountName}.blob.core.windows.net/${container}` + return `${origin}/${key.split('/').map(encodeURIComponent).join('/')}?${sas.toString()}` + } +} + +/** + * Azure Blob Storage module + * + * Blob names are the same paths the disk target writes, so a container and a folder hold the wiki's + * content laid out identically, and the shape of both is the site's `pathPrefixFor` to decide. See + * `helpers/storageObjects.ts` for everything above the four calls below. + */ +export default objectStorageModule(azureClient) diff --git a/backend/modules/storage/db/definition.yml b/backend/modules/storage/db/definition.yml index a201c9b0d..7bc1a9d73 100644 --- a/backend/modules/storage/db/definition.yml +++ b/backend/modules/storage/db/definition.yml @@ -4,10 +4,7 @@ icon: '/_assets/icons/ultraviolet-database.svg' banner: '/_assets/storage/database.jpg' description: 'The local PostgreSQL database can store any assets. It is however not recommended to store large files directly in the database as this can cause performance issues.' assetDelivery: - isStreamingSupported: true isDirectAccessSupported: false - defaultStreamingEnabled: true - defaultDirectAccessEnabled: false contentTypes: defaultTypesEnabled: ['pages', 'images', 'documents', 'others', 'large'] props: {} diff --git a/backend/modules/storage/disk/definition.yml b/backend/modules/storage/disk/definition.yml index 69d573a07..286532558 100644 --- a/backend/modules/storage/disk/definition.yml +++ b/backend/modules/storage/disk/definition.yml @@ -4,10 +4,7 @@ icon: '/_assets/icons/ultraviolet-hdd.svg' banner: '/_assets/storage/disk.jpg' description: Store files on the local file system or over network attached storage. Note that you must use replicated storage if using high-availability instances. assetDelivery: - isStreamingSupported: true isDirectAccessSupported: false - defaultStreamingEnabled: true - defaultDirectAccessEnabled: false contentTypes: defaultTypesEnabled: ['pages', 'images', 'documents', 'others', 'large'] props: diff --git a/backend/modules/storage/disk/storage.ts b/backend/modules/storage/disk/storage.ts index 6ed759042..f5e6e7c61 100644 --- a/backend/modules/storage/disk/storage.ts +++ b/backend/modules/storage/disk/storage.ts @@ -1,411 +1,57 @@ import fs from 'node:fs/promises' import path from 'node:path' -import { dump as dumpYaml, load as loadYaml } from 'js-yaml' -import { pageEditorForExtension, pageFileExtension } from '../../../models/pages.ts' -import type { - StorageModule, - StoragePageContent, - StoragePageRef, - StorageTarget -} from '../../../models/storage.ts' +import { + absPathIn, + assetRelPath, + importTree, + moveStored, + pageRelPath, + pruneEmptyDirs, + resolveRoot, + serializePage, + writeFileAtomic +} from '../../../helpers/storageFiles.ts' +import type { ImportSummary } from '../../../helpers/storageFiles.ts' +import type { StorageModule, StorageTarget } from '../../../models/storage.ts' /** Where files go when the target has no path configured, matching the definition's default. */ const DEFAULT_PATH = './data/content' -/** Leading YAML front matter, as `serializePage` writes it. */ -const FRONT_MATTER = /^---\r?\n([\s\S]*?)\r?\n---\r?\n?/ - -/** The extension whose pages are written as one JSON document rather than front matter and a body. */ -const JSON_EXTENSION = 'json' - -/** - * The editor an imported page falls back to. - * - * Only reached for a file the site reserves as a page extension but which no editor writes — `txt` on - * a default site. Markdown renders plain prose as prose, so it is the least surprising answer. - */ -const DEFAULT_PAGE_EDITOR = 'markdown' - -/** - * What the site's root page is filed as. - * - * A page path may be empty, which is the page a site serves at `/`. It still needs a name of its own - * on disk, and `index` is the one every other tool that writes a tree of documents picks. - */ -const ROOT_PAGE_NAME = 'index' - -/** - * Names never picked up by `importAll`. - * - * A half-written file carries the first, and the second is what a Mac leaves in every folder it has - * ever looked at — neither is content somebody meant to put in their wiki. - */ -const IGNORED_FILES = /^\.|\.tmp$/ - -/** - * The root this target writes under, as an absolute path. - * - * A relative setting is resolved from the install directory rather than from the working directory, - * so that `./data/content` means the same folder whichever way the server was started. - */ +/** The root this target writes under, as an absolute path. */ function baseDir(target: StorageTarget): string { - return path.resolve(WIKI.ROOTPATH, target.config.path || DEFAULT_PATH) -} - -/** - * Where an asset belongs under the root, as a slash-separated relative path. - * - * The locale brackets the tree because the tree repeats itself across locales — `guides/logo.png` can - * exist once in each, and all of them would otherwise be the same file. - * - * The site does not, and this is the one thing to know about this layout: a target belongs to exactly - * one site, so the folder an administrator configured IS this site's folder. A level for the site - * inside it would be a folder that never has a sibling, and it would put the tree one step further - * down than the path they typed. - */ -function relPathFor(ref: { locale: string; folderPath: string; fileName: string }): string { - return [ref.locale, ...ref.folderPath.split('/').filter(Boolean), ref.fileName].join('/') -} - -/** - * Where a page's copy belongs, alongside the assets of the same folder. - * - * The extension is the one its editor writes, and is exactly what the wiki reserves against uploads — - * `models/assets.ts` refuses an attachment that would take this name, so the two never meet here. - */ -function pagePathFor(ref: StoragePageRef): string { - const segments = ref.path.split('/').filter(Boolean) - const fileName = segments.pop() ?? ROOT_PAGE_NAME - return [ref.locale, ...segments, `${fileName}.${pageFileExtension(ref.contentType)}`].join('/') + return resolveRoot(target.config.path, DEFAULT_PATH) } -/** - * The absolute path of a stored file, refusing anything that would land outside the root. - * - * Every segment reaching this is either a UUID or a name the tree has already normalized, so this - * catches a stored path that has been tampered with rather than an ordinary mistake — but it is the - * only thing between a `..` in the database and the rest of the file system. - */ -function absPathFor(target: StorageTarget, relPath: string): string { - const base = baseDir(target) - const resolved = path.resolve(base, relPath) - if (resolved !== base && !resolved.startsWith(base + path.sep)) { - throw new Error(`The stored path "${relPath}" resolves outside the storage folder.`) - } - return resolved -} - -/** - * Remove the folders a deleted file leaves behind, stopping at the first one still in use. - * - * Best effort throughout: a folder that turns out not to be empty, or that another request is - * writing into at that moment, is simply left alone. - */ -async function pruneEmptyDirs(target: StorageTarget, fromDir: string): Promise { - const base = baseDir(target) - let dir = fromDir - while (dir !== base && dir.startsWith(base + path.sep)) { - try { - await fs.rmdir(dir) - } catch { - return - } - dir = path.dirname(dir) - } -} - -/** - * Write a file, creating its folder and leaving nothing half-written behind. - * - * Written under a temporary name and renamed, so a reader either finds the previous contents or the - * new ones — never the middle of a write. That matters here more than it does for a cache: this is - * the only copy of an asset once the database target has been purged. - */ -async function writeFileAtomic(filePath: string, data: Buffer | string): Promise { - const tempPath = `${filePath}.${process.pid}.tmp` - await fs.mkdir(path.dirname(filePath), { recursive: true }) - try { - await fs.writeFile(tempPath, data) - await fs.rename(tempPath, filePath) - } catch (err) { - await fs.rm(tempPath, { force: true }).catch(() => {}) - throw err - } -} - -/** - * Move a file, coping with a root that spans devices and with the file not being there. - * - * @returns Whether anything was moved - */ -async function moveFile(from: string, to: string): Promise { - await fs.mkdir(path.dirname(to), { recursive: true }) - try { - await fs.rename(from, to) - return true - } catch (err: any) { - if (err.code === 'ENOENT') { - return false - } - if (err.code !== 'EXDEV') { - throw err - } - // -> `rename` cannot cross a mount point - await fs.copyFile(from, to) - await fs.rm(from, { force: true }) - return true - } -} - -/** The metadata every page file carries, whichever of the two forms it is written in. */ -function pageMeta(page: StoragePageContent): Record { - return { - title: page.title, - description: page.description, - published: page.isPublished, - date: page.updatedAt.toISOString(), - tags: page.tags, - // -> The declaration that makes this a page rather than a file that happens to sit here. Nothing - // is imported as a page without it, so it is the one key that must always be written. - editor: page.editor, - dateCreated: page.createdAt.toISOString() - } -} - -/** - * A page as a file that stands on its own, in one of two forms. - * - * A **text** page is YAML front matter and then the source as the author wrote it — the convention - * every static site generator reads and the one Wiki.js 2.x wrote, so the folder is worth something - * to tools that have never heard of this wiki. - * - * A **JSON** page — a redirection today — is a single JSON document with the same metadata at its top - * level and the source under `content`. Front matter would leave a `.json` file that is not JSON, - * which is worth avoiding for the one editor whose source is already structured. - * - * Either way the metadata is the point: a bare body says nothing about whether it was published or - * what it was called, and none of that is recoverable from the prose. - */ -function serializePage(ref: StoragePageRef, page: StoragePageContent): string { - if (pageFileExtension(ref.contentType) === JSON_EXTENSION) { - let content: any = page.content - try { - content = JSON.parse(page.content) - } catch { - // -> Kept as the string it is. The column is written by the editor and should always parse, - // and a file that says what it holds beats one this refused to write. - } - return `${JSON.stringify({ ...pageMeta(page), content }, null, 2)}\n` - } - return `---\n${dumpYaml(pageMeta(page))}---\n\n${page.content}\n` -} - -/** - * Read a page file back: its declaration, and the source below it. - * - * A page file says it is one by carrying an `editor` — in its front matter, or at the top level of - * the JSON document for the one editor written that way. That declaration is what lets a page and an - * attachment share a folder without this module having to guess which is which from an extension, - * and it is how every file this module writes comes back. - * - * Returning null does not settle it. A file whose extension the site reserves for pages is a page - * whatever it does or does not declare — see `importAll`, which owns that rule and fills the editor - * in from the extension. This only reports whether the file said so itself. - * - * Beyond the one key it is forgiving: a file may have been hand-written or generated by something - * that has never seen this wiki, so a missing title or date is filled in by the caller, and front - * matter that is not YAML is treated as no declaration rather than as a fault. - * - * @returns The declaration and the source, or null for a file that does not declare itself a page - */ -function deserializePage( - raw: string, - ext: string -): { meta: Record; content: string } | null { - if (ext === JSON_EXTENSION) { - let parsed: any - try { - parsed = JSON.parse(raw) - } catch { - return null - } - if (!parsed || typeof parsed !== 'object' || typeof parsed.editor !== 'string') { - return null - } - const { content, ...meta } = parsed - return { - meta, - content: typeof content === 'string' ? content : JSON.stringify(content ?? {}) - } - } - - const match = FRONT_MATTER.exec(raw) - if (!match) { - return null - } - let meta: Record - try { - const parsed = loadYaml(match[1]) - if (!parsed || typeof parsed !== 'object') { - return null - } - meta = parsed as Record - } catch { - return null - } - if (typeof meta.editor !== 'string' || !meta.editor) { - return null - } - return { meta, content: raw.slice(match[0].length).trim() } -} - -/** A front matter date, or undefined for one that is missing or not a date at all. */ -function parseDate(value: unknown): Date | undefined { - if (value instanceof Date) { - return value - } - if (typeof value !== 'string') { - return undefined - } - const date = new Date(value) - return Number.isNaN(date.getTime()) ? undefined : date -} - -/** - * Take everything in a target's folder into the wiki, either filling in what is missing or letting - * the folder win. - * - * The body of both import actions. They differ by one flag and by the verb they report with, because - * the walk, what counts as a page, and what is done with a file that is neither are the same - * question whichever way a collision is settled — see `importAll` for that walk, and the two models' - * `adoptStoredPage` / `adoptStoredFile` for what `overwrite` means once a file has landed on - * something. - */ -async function runImport( - target: StorageTarget, - actorId: string, - { overwrite }: { overwrite: boolean } -): Promise { - const root = baseDir(target) - let entries - try { - entries = await fs.readdir(root, { recursive: true, withFileTypes: true }) - } catch (err: any) { - if (err.code !== 'ENOENT') { - throw err - } +/** What an import run did, in the words the two import actions report it with. */ +function describeImport(summary: ImportSummary | null, overwrite: boolean): string { + if (!summary) { return 'There is nothing in the storage folder for this site yet.' } - - const reserved: string[] = WIKI.sites[target.siteId]?.config?.pageExtensions ?? [] - let pages = 0 - let assets = 0 - let skipped = 0 - let failed = 0 - - for (const entry of entries) { - if (!entry.isFile() || IGNORED_FILES.test(entry.name)) { - continue - } - const filePath = path.join(entry.parentPath, entry.name) - // -> `//`, so a file sitting straight in the root is outside the - // layout and belongs to no locale - const segments = path.relative(root, filePath).split(path.sep) - if (segments.length < 2) { - continue - } - const [locale, ...rest] = segments - const fileName = rest.pop()! - const ext = path.extname(fileName).replace(/^\./, '').toLowerCase() - const folderPath = rest.join('/') - - const raw = await fs.readFile(filePath) - const declared = deserializePage(raw.toString('utf8'), ext) - const isReservedExtension = Boolean(ext) && reserved.includes(ext) - - if (!declared && !isReservedExtension) { - const asset = await WIKI.models.assets.adoptStoredFile({ - siteId: target.siteId, - locale, - folderPath, - fileName, - data: raw, - authorId: actorId, - overwrite - }) - if (asset) { - assets++ - } else { - skipped++ - } - continue - } - - // -> A page is addressed without its extension: `guides/setup.md` is the page at `guides/setup` - const name = fileName.slice(0, fileName.length - (ext ? ext.length + 1 : 0)) - const meta = declared?.meta ?? {} - try { - const imported = await WIKI.models.pages.adoptStoredPage({ - siteId: target.siteId, - locale, - // -> The root page is filed under a name of its own, and takes the empty path back - path: [...rest, ...(name === ROOT_PAGE_NAME ? [] : [name])].join('/'), - // -> The declaration first, then the file itself: a page written by hand carries no title, - // and its name is the next best thing - title: typeof meta.title === 'string' && meta.title ? meta.title : name, - description: typeof meta.description === 'string' ? meta.description : '', - editor: - typeof meta.editor === 'string' && meta.editor - ? meta.editor - : (pageEditorForExtension(ext) ?? DEFAULT_PAGE_EDITOR), - tags: Array.isArray(meta.tags) ? meta.tags.map(String) : [], - isPublished: meta.published !== false, - // -> Undeclared means there was no front matter to strip, so the file is all body - content: declared?.content ?? raw.toString('utf8').trim(), - createdAt: parseDate(meta.dateCreated), - updatedAt: parseDate(meta.date), - authorId: actorId, - overwrite - }) - if (imported) { - pages++ - } else { - skipped++ - } - } catch (err: any) { - // -> One unusable file — an empty body, an editor this wiki does not have, a path it cannot - // address — must not stop the rest of the folder from being imported - failed++ - WIKI.logger.warn(`Could not import the page at ${filePath} [ SKIPPED ]`) - WIKI.logger.warn(err.message) - } - } - - WIKI.logger.info(`Imported ${pages} page(s) and ${assets} asset(s) from ${root} [ OK ]`) + WIKI.logger.info(`Imported ${summary.pages} page(s) and ${summary.assets} asset(s) [ OK ]`) // -> Nothing is reported as merely imported when a run could have replaced something: an // administrator reading "Imported 40 pages" has to be able to tell which of the two they ran const verb = overwrite ? 'Imported or replaced' : 'Imported' const parts = [] - if (pages > 0) { - parts.push(`${verb} ${pages} page(s).`) + if (summary.pages > 0) { + parts.push(`${verb} ${summary.pages} page(s).`) } - if (assets > 0) { - parts.push(`${verb} ${assets} asset(s).`) + if (summary.assets > 0) { + parts.push(`${verb} ${summary.assets} asset(s).`) } if (parts.length < 1) { parts.push(overwrite ? 'There was nothing to import.' : 'There was nothing new to import.') } - if (skipped > 0) { + if (summary.skipped > 0) { // -> With `overwrite` the only thing left to skip is a name a page or a folder owns, which is not // something this action was ever going to take over parts.push( overwrite - ? `${skipped} could not replace what is at their path and were left alone.` - : `${skipped} were already in the wiki and were left alone.` + ? `${summary.skipped} could not replace what is at their path and were left alone.` + : `${summary.skipped} were already in the wiki and were left alone.` ) } - if (failed > 0) { - parts.push(`${failed} could not be imported - see the server log.`) + if (summary.failed > 0) { + parts.push(`${summary.failed} could not be imported - see the server log.`) } return parts.join(' ') } @@ -414,13 +60,16 @@ async function runImport( * Local file system storage module * * Mirrors the wiki's own tree onto disk under the folder the target is configured with, laid out - * `//` — so that what an administrator sees in the file manager is what they - * find in the folder, and so that a wiki's content remains ordinary files: readable, backed up and - * served by whatever else is on the machine. Pages and assets share that tree, a page filed under its - * editor's extension; keeping the two from colliding belongs to the models, not here. + * `//` by default — so that what an administrator sees in the file manager is + * what they find in the folder, and so that a wiki's content remains ordinary files: readable, backed + * up and served by whatever else is on the machine. Pages and assets share that tree, a page filed + * under its editor's extension; keeping the two from colliding belongs to the models, not here. * - * The folder is the site's own, with no level inside it naming the site — see `relPathFor`. Two sites - * therefore must not be pointed at the same path. + * What brackets that tree is the site's to say and not this module's: the site id, the locale, both or + * neither, per `pathPrefixFor`. The folder is normally the site's own, since a target belongs to + * exactly one site — two sites sharing a path is what the site id prefix is for. With the locale + * prefix off the site stores its primary locale and nothing else, so a file in another locale has no + * path here at all, and every operation below has to say what it does about that. * * Nothing records where a file went. Every path is derived from the ref it is given, the same way * every time, which is what lets a copy be read back, moved or deleted with nothing stored about @@ -430,15 +79,36 @@ async function runImport( * else, so writes are atomic and a failure is raised for the caller to fail the upload on. A **page** * is a database row and always will be, so what sits here is a rendering of it written after the * fact, never read back, and allowed to fail. + * + * Everything about the shape of the tree itself — the front matter, what makes a file a page, the + * walk an import does — is in `helpers/storageFiles.ts`, shared with the git target, which keeps the + * same tree inside a repository. */ const diskStorage: StorageModule = { + canStore(target, ref) { + return WIKI.models.storage.pathPrefixFor(target.siteId, ref.locale) !== null + }, + async putAsset(target, ref, data) { - await writeFileAtomic(absPathFor(target, relPathFor(ref)), data) + const relPath = assetRelPath(target, ref) + // -> Guarded rather than skipped, unlike every read and delete below: the model asks `canStore` + // before it dispatches a write, so reaching this means somebody wrote to this target without + // asking, and dropping the only copy of an asset's bytes is not a thing to do quietly + if (!relPath) { + throw new Error( + `${target.title} has no path for ${ref.locale} content, so ${ref.fileName} cannot be stored there.` + ) + } + await writeFileAtomic(absPathIn(baseDir(target), relPath), data) }, async getAsset(target, ref) { + const relPath = assetRelPath(target, ref) + if (!relPath) { + return null + } try { - return await fs.readFile(absPathFor(target, relPathFor(ref))) + return await fs.readFile(absPathIn(baseDir(target), relPath)) } catch (err: any) { if (err.code !== 'ENOENT') { throw err @@ -450,37 +120,55 @@ const diskStorage: StorageModule = { }, async deleteAsset(target, ref) { - const filePath = absPathFor(target, relPathFor(ref)) + const relPath = assetRelPath(target, ref) + if (!relPath) { + return + } + const root = baseDir(target) + const filePath = absPathIn(root, relPath) await fs.rm(filePath, { force: true }) - await pruneEmptyDirs(target, path.dirname(filePath)) + await pruneEmptyDirs(root, path.dirname(filePath)) }, async moveAsset(target, ref, previous) { - const from = absPathFor(target, relPathFor({ ...ref, ...previous })) - if (await moveFile(from, absPathFor(target, relPathFor(ref)))) { - await pruneEmptyDirs(target, path.dirname(from)) - } + await moveStored( + baseDir(target), + assetRelPath(target, { ...ref, ...previous }), + assetRelPath(target, ref) + ) }, async putPage(target, ref, page) { - await writeFileAtomic(absPathFor(target, pagePathFor(ref)), serializePage(ref, page)) + const relPath = pageRelPath(target, ref) + // -> Unlike an asset, a page that has no place here is not a failure worth reporting: it is in + // the database, which is where a page always is, and this copy is the thing the site declined + if (!relPath) { + return + } + await writeFileAtomic(absPathIn(baseDir(target), relPath), serializePage(ref, page)) }, async deletePage(target, ref) { // -> Exactly one name, taken from the page's own content type. Guessing at the others would mean // deleting whatever happens to sit beside it: in this folder `readme.html` is as likely to be // an attachment as it is to be the page `readme`. - const filePath = absPathFor(target, pagePathFor(ref)) + const relPath = pageRelPath(target, ref) + if (!relPath) { + return + } + const root = baseDir(target) + const filePath = absPathIn(root, relPath) await fs.rm(filePath, { force: true }) - await pruneEmptyDirs(target, path.dirname(filePath)) + await pruneEmptyDirs(root, path.dirname(filePath)) }, async movePage(target, ref, previousPath) { // -> Which editor wrote it does not change when a page moves, so both ends share an extension - const from = absPathFor(target, pagePathFor({ ...ref, path: previousPath })) - if (await moveFile(from, absPathFor(target, pagePathFor(ref)))) { - await pruneEmptyDirs(target, path.dirname(from)) - } + await moveStored( + baseDir(target), + pageRelPath(target, { ...ref, path: previousPath }), + pageRelPath(target, ref) + ) }, /** @@ -500,6 +188,7 @@ const diskStorage: StorageModule = { async exportAll(target: StorageTarget): Promise { let assets = 0 let unreadable = 0 + let unstored = 0 for (const asset of await WIKI.models.assets.listStoredAssets(target.siteId)) { // -> Only what the current configuration says belongs here: an administrator who turned this // target on for images alone did not ask for their videos to be written out as well @@ -511,6 +200,12 @@ const diskStorage: StorageModule = { if (!target.contentTypes.activeTypes.includes(contentType)) { continue } + // -> And only what the layout has somewhere to put: a site storing its primary locale alone + // has no path for the rest, and `putAsset` would refuse them one at a time + if (!assetRelPath(target, asset)) { + unstored++ + continue + } const data = await WIKI.models.storage.getAsset(asset) if (!data) { unreadable++ @@ -523,6 +218,10 @@ const diskStorage: StorageModule = { let pages = 0 if (target.contentTypes.activeTypes.includes('pages')) { for (const { ref, content } of await WIKI.models.pages.listForStorage(target.siteId)) { + if (!pageRelPath(target, ref)) { + unstored++ + continue + } await diskStorage.putPage(target, ref, content) pages++ } @@ -540,6 +239,12 @@ const diskStorage: StorageModule = { if (unreadable > 0) { parts.push(`${unreadable} asset(s) could not be read and were skipped.`) } + if (unstored > 0) { + const { primaryLocale } = WIKI.models.storage.pathLayoutFor(target.siteId) + parts.push( + `${unstored} item(s) are not in the ${primaryLocale} locale, which is the only one this site stores.` + ) + } return parts.join(' ') }, @@ -548,30 +253,19 @@ const diskStorage: StorageModule = { * * The direction that makes this folder a store rather than a dumping ground: content arrives here * from outside — restored from a backup, generated by another tool, unpacked from an archive — and - * this is what turns it back into pages and assets. - * - * **Two ways a file is a page**, and everything else is an attachment, read where it lies and - * adopted in place rather than copied: + * this is what turns it back into pages and assets. What counts as a page, and what happens to a + * file that lands on something the wiki already has, are `importTree`'s to say. * - * 1. **Its extension is one the site reserves for pages.** Those extensions address a page by URL - * and cannot be uploaded as attachments, so nothing else can be sitting under one — which makes - * a hand-written `.md` dropped into the folder a page, as whoever dropped it meant. - * 2. **It declares an editor**, in its front matter or at the top level of its JSON. This is what - * every file written here carries, and it is the only way in for an extension the site does not - * reserve — `.adoc` on a default site, where an attachment could just as well be sitting. - * - * Which editor it belongs to comes from that declaration, and only falls back to the extension for - * a reserved one that made none. - * - * A path the wiki already has an entry at is left alone in both directions: nothing on disk is - * overwritten, and nothing in the wiki is. That makes this safe to run repeatedly, and makes it no - * use for picking up a file that changed on both sides — reconciling those two is a merge, and this - * module has no history to do one from. A target that does, git being the obvious one, is where - * that belongs. `importAllOverwrite` is the answer for the case where there is nothing to reconcile - * because the folder is simply right. + * A path the wiki already has an entry at is left alone in both directions, which makes this safe + * to run repeatedly and makes it no use for picking up a file that changed on both sides — that is + * a merge, and this module has no history to do one from. `importAllOverwrite` is the answer for + * the case where there is nothing to reconcile because the folder is simply right. */ async importAll(target: StorageTarget, actorId: string): Promise { - return runImport(target, actorId, { overwrite: false }) + return describeImport( + await importTree({ target, root: baseDir(target), actorId, overwrite: false }), + false + ) }, /** @@ -579,8 +273,7 @@ const diskStorage: StorageModule = { * * For the case `importAll` deliberately refuses: not filling in what the wiki is missing but making * it say what the folder says — a restore onto an instance that already has content, or a tree - * edited outside the wiki that is meant to be taken as the new truth. Nothing else about the import - * changes; only what happens to a file that lands on something. + * edited outside the wiki that is meant to be taken as the new truth. * * The two halves are not equally recoverable, which is the thing to know before running it. A * **page** is replaced by an ordinary save, so its previous version is in its history. An **asset** @@ -588,7 +281,10 @@ const diskStorage: StorageModule = { * are gone. */ async importAllOverwrite(target: StorageTarget, actorId: string): Promise { - return runImport(target, actorId, { overwrite: true }) + return describeImport( + await importTree({ target, root: baseDir(target), actorId, overwrite: true }), + true + ) } } diff --git a/backend/modules/storage/gcs/definition.yml b/backend/modules/storage/gcs/definition.yml new file mode 100644 index 000000000..7a7c6ef92 --- /dev/null +++ b/backend/modules/storage/gcs/definition.yml @@ -0,0 +1,57 @@ +key: gcs +title: Google Cloud Storage +icon: '/_assets/icons/ultraviolet-google.svg' +banner: '/_assets/storage/gcs.jpg' +description: Google Cloud Storage is an online file storage web service for storing and accessing data on Google Cloud Platform infrastructure. +assetDelivery: + isDirectAccessSupported: true +contentTypes: + defaultTypesEnabled: ['images', 'documents', 'others', 'large'] +props: + projectId: + type: String + title: Project ID + default: '' + hint: The project ID from the Google Cloud console, e.g. grape-spaceship-123. Optional when the credentials below name one. + icon: 3d-touch + order: 1 + credentialsJSON: + type: String + title: JSON Credentials + default: '' + hint: Contents of the JSON key file for a service account with Storage Object Admin on the bucket. Leave empty to use Application Default Credentials, which is what a workload identity on GKE or Cloud Run provides. + icon: key + multiline: true + sensitive: true + order: 2 + bucket: + type: String + title: Bucket Name + default: '' + hint: The bucket to store content in. It must already exist - this target will not create it. + icon: open-box + order: 3 + storageClass: + type: String + title: Storage Class + default: STANDARD + hint: What new objects are stored as. The colder classes cost less to keep and more to read, and charge for a minimum storage duration. + icon: scan-stock + order: 4 + enum: + - STANDARD|Standard + - NEARLINE|Nearline + - COLDLINE|Coldline + - ARCHIVE|Archive + apiEndpoint: + type: String + title: API Endpoint + default: '' + hint: Leave empty for Google Cloud Storage itself. Only set this to point at an emulator or a private service endpoint. + icon: api + order: 5 +actions: + exportAll: + label: Export Everything + hint: Write a copy of every page and asset this target is configured to hold to the bucket, overwriting whatever is already there. Nothing in the database is changed and nothing is moved, so this is how content created before the target was enabled gets onto it. + icon: this-way-up diff --git a/backend/modules/storage/gcs/storage.ts b/backend/modules/storage/gcs/storage.ts new file mode 100644 index 000000000..d7192603d --- /dev/null +++ b/backend/modules/storage/gcs/storage.ts @@ -0,0 +1,133 @@ +import { Storage } from '@google-cloud/storage' +import { objectStorageModule, signingBaseUrl } from '../../../helpers/storageObjects.ts' +import type { Bucket } from '@google-cloud/storage' +import type { ObjectStoreClient } from '../../../helpers/storageObjects.ts' +import type { StorageTarget } from '../../../models/storage.ts' + +/** Live buckets, keyed by target. See `bucketFor`. */ +const buckets = new Map() + +/** The settings a client is built from — a change to any of them needs a new one. */ +function configFingerprint(target: StorageTarget): string { + const c = target.config + return JSON.stringify([c.projectId, c.credentialsJSON, c.bucket, c.apiEndpoint]) +} + +/** + * The bucket handle for this target, built once and kept. + * + * **The credentials are optional.** Left empty, the client falls back to Application Default + * Credentials — the workload identity attached to a GKE pod or a Cloud Run service, or the + * `GOOGLE_APPLICATION_CREDENTIALS` file — which is how a deployment on Google's own infrastructure + * avoids putting a service account key in the database at all. + * + * @throws When the pasted credentials are not JSON, which is worth saying plainly: it is a long blob + * somebody pasted into a form, and the client's own error for it is not obviously about that + */ +function bucketFor(target: StorageTarget): Bucket { + const fingerprint = configFingerprint(target) + const cached = buckets.get(target.id) + if (cached && cached.fingerprint === fingerprint) { + return cached.bucket + } + const { projectId, credentialsJSON, bucket, apiEndpoint } = target.config + + let credentials + if (credentialsJSON?.trim()) { + try { + credentials = JSON.parse(credentialsJSON) + } catch { + throw new Error( + 'The JSON credentials for this target are not valid JSON. Paste the whole contents of the service account key file.' + ) + } + } + + const storage = new Storage({ + ...(projectId ? { projectId } : {}), + ...(credentials ? { credentials } : {}), + ...(apiEndpoint ? { apiEndpoint } : {}) + }) + const handle = storage.bucket(bucket) + buckets.set(target.id, { bucket: handle, fingerprint }) + return handle +} + +/** Whether the service is telling us the object simply is not there. */ +function isNotFound(err: any): boolean { + return err?.code === 404 +} + +const gcsClient: ObjectStoreClient = { + async put(target, key, data, contentType) { + await bucketFor(target) + .file(key) + .save(data, { + contentType, + ...(target.config.storageClass && target.config.storageClass !== 'STANDARD' + ? { metadata: { storageClass: target.config.storageClass } } + : {}) + }) + }, + + async get(target, key) { + try { + const [contents] = await bucketFor(target).file(key).download() + return contents + } catch (err: any) { + if (isNotFound(err)) { + // -> This target does not have the file: enabled after the upload, or removed from outside + // the wiki. Not a fault — the caller asks the next target. + return null + } + throw err + } + }, + + async remove(target, key) { + await bucketFor(target).file(key).delete({ ignoreNotFound: true }) + }, + + async copy(target, fromKey, toKey) { + const bucket = bucketFor(target) + try { + await bucket.file(fromKey).copy(bucket.file(toKey)) + return true + } catch (err: any) { + if (isNotFound(err)) { + return false + } + throw err + } + }, + + async presign(target, { key, expiresInSeconds, contentType, downloadAs }) { + const baseUrl = signingBaseUrl(target) + const [url] = await bucketFor(target) + .file(key) + .getSignedUrl({ + version: 'v4', + action: 'read', + expires: Date.now() + expiresInSeconds * 1000, + responseType: contentType, + ...(downloadAs ? { promptSaveAs: downloadAs } : {}), + /* + A V4 signature covers the host, so a URL signed for `storage.googleapis.com` and then moved + onto a custom domain is a signature for the wrong host. `cname` is how the client is told to + sign for that domain in the first place — the same reason the S3 module builds a second + client rather than rewriting its output. + */ + ...(baseUrl ? { cname: baseUrl } : {}) + }) + return url + } +} + +/** + * Google Cloud Storage module + * + * Object names are the same paths the disk target writes, so a bucket and a folder hold the wiki's + * content laid out identically, and the shape of both is the site's `pathPrefixFor` to decide. See + * `helpers/storageObjects.ts` for everything above the four calls below. + */ +export default objectStorageModule(gcsClient) diff --git a/backend/modules/storage/git/definition.yml b/backend/modules/storage/git/definition.yml new file mode 100644 index 000000000..77b0cc429 --- /dev/null +++ b/backend/modules/storage/git/definition.yml @@ -0,0 +1,157 @@ +key: git +title: Git +icon: '/_assets/icons/ultraviolet-git.svg' +banner: '/_assets/storage/git.jpg' +description: Keep this site's content in a Git repository, committed as it changes and synchronized with a remote. Every page and file is an ordinary versioned file, so the wiki's history is also the repository's. +assetDelivery: + isDirectAccessSupported: false +contentTypes: + defaultTypesEnabled: ['pages', 'images', 'documents', 'others'] +props: + authType: + type: String + default: 'ssh' + title: Authentication Type + hint: Use SSH for maximum security. + icon: security-configuration + enum: + - basic|Basic + - ssh|SSH + enumDisplay: buttons + order: 1 + repoUrl: + type: String + title: Repository URI + hint: Git-compliant URI (e.g. git@server.com:org/repo.git for SSH, https://server.com/org/repo.git for basic). Leave empty to keep a purely local repository, committed but never pushed. + icon: dns + order: 2 + branch: + type: String + default: 'main' + title: Branch + hint: The branch to use during pull / push. It must already exist on the remote. + icon: code-fork + order: 3 + syncMode: + type: String + default: 'sync' + title: Sync Direction + hint: Sync pulls and then pushes. Push force-pushes the wiki's commits and never takes anything back, so the wiki always wins. Pull only takes changes in, and is what makes the remote the authority. + icon: synchronize + enum: + - sync|Sync + - push|Push only + - pull|Pull only + enumDisplay: buttons + order: 4 + sshPrivateKeyMode: + type: String + title: SSH Private Key Mode + hint: The mode to use to load the private key. Fill in the corresponding field below. + icon: grand-master-key + order: 11 + default: inline + enum: + - path|File Path + - inline|Inline Contents + enumDisplay: buttons + if: + - { key: 'authType', eq: 'ssh' } + sshPrivateKeyPath: + type: String + title: SSH Private Key Path + hint: Absolute path to the key. The key must NOT be passphrase-protected. + icon: key + order: 12 + if: + - { key: 'authType', eq: 'ssh' } + - { key: 'sshPrivateKeyMode', eq: 'path' } + sshPrivateKeyContent: + type: String + title: SSH Private Key Contents + hint: Paste the contents of the private key. The key must NOT be passphrase-protected. It is written to a file readable only by the wiki's own user. + icon: key + multiline: true + sensitive: true + order: 13 + if: + - { key: 'authType', eq: 'ssh' } + - { key: 'sshPrivateKeyMode', eq: 'inline' } + verifySSL: + type: Boolean + default: true + title: Verify SSL Certificate + hint: Some hosts require SSL certificate checking to be disabled. Leave enabled for proper security. + icon: security-ssl + order: 14 + basicUsername: + type: String + title: Username + hint: Basic Authentication Only + icon: test-account + order: 20 + if: + - { key: 'authType', eq: 'basic' } + basicPassword: + type: String + title: Password / PAT + hint: Basic Authentication Only + icon: password + sensitive: true + order: 21 + if: + - { key: 'authType', eq: 'basic' } + alwaysUseDefaultAuthor: + type: Boolean + default: false + title: Always Commit as the Default Author + hint: Attribute every commit to the default author below instead of to the user who made the change, so that no account name or email address reaches the repository. Turn this on if the remote is somewhere your users' identities should not be published. + icon: data-protection + order: 29 + defaultName: + type: String + title: Default Author Name + default: 'Wiki.js' + hint: The commit author when the change was not made by one person - a scheduled sync, or a folder rename that moved a hundred files - and every commit when the option above is on. + icon: customer + order: 30 + defaultEmail: + type: String + title: Default Author Email + default: 'wiki@example.com' + hint: The commit author email in the same cases as the name above. + icon: email + order: 31 + localRepoPath: + type: String + title: Local Repository Path + default: './data/repo' + hint: Where the working copy is kept. Give each site its own path unless you turn on Add Site ID Prefix under Configuration, since two sites sharing a repository would otherwise write over each other. Relative paths are resolved from the Wiki.js install directory. + icon: symlink-directory + order: 32 + gitBinaryPath: + type: String + title: Git Binary Path + default: '' + hint: Optional - Absolute path to the Git binary, when not available in PATH. Leave empty to use the default PATH location (recommended). + icon: run-command + order: 50 +actions: + sync: + label: Force Sync + hint: Run a sync straight away rather than waiting for the next scheduled one. The Sync Direction above is respected, and a pull applies what it brings in to the wiki. + icon: synchronize + syncUntracked: + label: Add Untracked Changes + hint: Write every page and file this target is configured to hold into the repository and commit whatever is missing. Content created before Git was enabled - or while it was turned off - is untracked until this is run. + icon: database-daily-export + importAll: + label: Import Everything + hint: Take everything currently in the local repository into the wiki, whatever the last commit did. For picking up content from a remote repository that existed before Git was enabled here. + warn: The repository wins every collision. A page it replaces keeps its previous version in its history, but a file has none - the bytes it replaces are gone from every storage target holding them. + icon: database-daily-import + purge: + label: Purge Local Repository + hint: Empty the local working copy and clone it again from the remote. This is the way out of unrelated merge histories or a working copy that git can no longer make sense of. The remote is not touched and nothing is committed. + warn: Any commit that exists only in the local repository and has never been pushed is lost. Run a Force Sync first if you are not sure. + icon: trash diff --git a/backend/modules/storage/git/storage.ts b/backend/modules/storage/git/storage.ts new file mode 100644 index 000000000..b0916e823 --- /dev/null +++ b/backend/modules/storage/git/storage.ts @@ -0,0 +1,934 @@ +import fs from 'node:fs/promises' +import path from 'node:path' +import { CheckRepoActions, simpleGit } from 'simple-git' +import { + absPathIn, + assetRelPath, + importTree, + moveStored, + pageRelPath, + resolveRoot, + serializePage, + walkStored +} from '../../../helpers/storageFiles.ts' +import type { ImportSummary, StoredFile } from '../../../helpers/storageFiles.ts' +import type { SimpleGit } from 'simple-git' +import type { StorageModule, StoragePageRef, StorageTarget } from '../../../models/storage.ts' + +/** Where the working copy goes when the target has no path configured, per the definition default. */ +const DEFAULT_REPO_PATH = './data/repo' + +/** What a change with no one person behind it is committed as, per the definition defaults. */ +const FALLBACK_AUTHOR = { name: 'Wiki.js', email: 'wiki@example.com' } + +/** + * One repository, as this module keeps it between operations. + * + * Cached against the configuration it was set up from, so that changing a setting in the admin area + * takes effect on the next operation rather than at the next restart. + */ +interface Repo { + git: SimpleGit + root: string + /** The configuration this was prepared from — see `configFingerprint`. */ + fingerprint: string + /** Whether the remote has been contacted since this entry was made. See `ensureRemote`. */ + remoteReady: boolean + /** Serializes work on this repository. See `withRepo`. */ + queue: Promise +} + +const repos = new Map() + +/** The working copy for this target, as an absolute path. */ +function repoDir(target: StorageTarget): string { + return resolveRoot(target.config.localRepoPath, DEFAULT_REPO_PATH) +} + +/** + * What the cached repository was prepared from. + * + * Every setting `prepareRepo` writes into the repository or uses to reach the remote. A change to any + * of them has to run the setup again — a new branch, a rotated key, a different URL. The default + * author is in here because it becomes the repository's own `user.name` and `user.email`, i.e. the + * committer of every commit; `alwaysUseDefaultAuthor` is not, because `commitAuthor` reads it per + * commit and there is nothing prepared from it. + */ +function configFingerprint(target: StorageTarget): string { + const c = target.config + return JSON.stringify([ + c.localRepoPath, + c.authType, + c.repoUrl, + c.branch, + c.sshPrivateKeyMode, + c.sshPrivateKeyPath, + c.sshPrivateKeyContent, + c.verifySSL, + c.basicUsername, + c.basicPassword, + c.gitBinaryPath, + c.defaultName, + c.defaultEmail + ]) +} + +/** Where an inline SSH key is written, one file per target so two of them cannot collide. */ +function sshKeyPath(target: StorageTarget): string { + return path.resolve(WIKI.ROOTPATH, WIKI.config.dataPath, 'secure', `git-ssh-${target.id}.pem`) +} + +/** + * The remote URL to talk to, with basic credentials folded in where there are any to fold. + * + * Built here rather than stored, and never logged: the password is a config value an administrator + * can rotate, and a URL with it baked in would otherwise sit in the repository's own `.git/config` + * under a name this module had stopped looking at. + * + * Credentials only ever go into an HTTP URL, which is the only scheme that has anywhere to put them. + * Everything else is passed through untouched — an `ssh://` or `git@host:path` remote authenticates + * with a key, and a bare path or `file://` is a repository on this machine and authenticates with + * nothing at all. A URL with no scheme is the one case worth guessing about: `server.com/org/repo.git` + * is what somebody configuring basic auth types, so it becomes HTTPS. + */ +function remoteUrl(target: StorageTarget): string { + const { authType, repoUrl, basicUsername, basicPassword } = target.config + if (authType !== 'basic') { + return repoUrl + } + // -> A local path, or any scheme that is not HTTP. `git@host:path` counts: the colon is scp syntax + // and there is no scheme at all, so the slash test is what tells it from `host/org/repo.git`. + const isHttp = /^https?:\/\//i.test(repoUrl) + if (!isHttp && (repoUrl.startsWith('/') || /^[a-z][a-z0-9+.-]*:/i.test(repoUrl))) { + return repoUrl + } + // -> Nothing to fold in. `https://:@host` is not the same request as `https://host` and some hosts + // refuse it outright, so an unset username means the URL is left as it is. + if (!basicUsername) { + return isHttp ? repoUrl : `https://${repoUrl}` + } + const credentials = `${encodeURIComponent(basicUsername)}:${encodeURIComponent(basicPassword ?? '')}` + return isHttp + ? repoUrl.replace(/^(https?:\/\/)/i, `$1${credentials}@`) + : `https://${credentials}@${repoUrl}` +} + +/** + * Prepare the local repository, without touching the network. + * + * Everything an ordinary page save needs: a working copy that exists, is a repository, knows who it + * is committing as and has `origin` pointing where the configuration says. Deliberately *not* the + * fetch and the checkout — see `ensureRemote`. A page save must not wait on a remote, and with the + * commits made locally and pushed by the sync there is no reason for it to. + */ +async function prepareRepo(target: StorageTarget): Promise { + const root = repoDir(target) + await fs.mkdir(root, { recursive: true }) + /* + `core.sshCommand` is arbitrary command execution, so simple-git refuses to set it unless the + caller says it means to — a library cannot tell a path an administrator typed from one that + arrived in a query string, and for most of its users the value would be the latter. + + Here it is neither: it comes from this target's own configuration, which is only writable through + `PUT /sites/:siteId/storage` behind `manage:system` — a permission that bypasses every check in + the wiki, so anybody who can set this can already do anything. Granted only for the auth type that + actually needs it, so a target authenticating over HTTPS carries no allowance at all. + */ + const git = simpleGit( + root, + target.config.authType === 'ssh' ? { unsafe: { allowUnsafeSshCommand: true } } : {} + ) + if (target.config.gitBinaryPath) { + git.customBinary(target.config.gitBinaryPath) + } + + /* + `IS_REPO_ROOT`, emphatically not a bare `checkIsRepo()`. That defaults to + `rev-parse --is-inside-work-tree`, which is true for any directory *inside* a repository — and the + default working copy is `/data/repo`, so on any instance whose install directory is itself + a git checkout (every dev install, and any deployment that pulled the source down with git) the + answer is yes and `git init` is skipped. Every command after that then runs against the wiki's own + source repository, which is how this came to fail on a `.gitignore` rule that has nothing to do + with storage. `--git-dir` resolving to `.git` is the question actually being asked here: is this + directory the root of its own repository. + + A repository nested inside another one's working tree is fine, and is what this creates: git uses + the innermost `.git` for commands run here, and reads ignore rules from this root downwards, so + the outer checkout's `.gitignore` stops applying the moment this exists. + */ + if (!(await git.checkIsRepo(CheckRepoActions.IS_REPO_ROOT))) { + WIKI.logger.info(`(STORAGE/GIT) Initializing local repository at ${root}...`) + await git.init(['--initial-branch', target.config.branch || 'main']) + } + + // -> Without this git escapes any non-ASCII path in its own output, and every path this module + // reads back out of a diff would arrive quoted and mangled + await git.addConfig('core.quotepath', 'false') + await git.addConfig('user.name', target.config.defaultName || FALLBACK_AUTHOR.name) + await git.addConfig('user.email', target.config.defaultEmail || FALLBACK_AUTHOR.email) + await git.addConfig('http.sslVerify', target.config.verifySSL === false ? 'false' : 'true') + + if (target.config.authType === 'ssh') { + let keyPath = target.config.sshPrivateKeyPath + if (target.config.sshPrivateKeyMode === 'inline') { + keyPath = sshKeyPath(target) + await fs.mkdir(path.dirname(keyPath), { recursive: true }) + // -> Trailing newline and 0600, both of which ssh insists on: it refuses a key file other + // users can read, and a key without the final newline + await fs.writeFile(keyPath, `${(target.config.sshPrivateKeyContent ?? '').trimEnd()}\n`, { + encoding: 'utf8', + mode: 0o600 + }) + } + if (keyPath) { + await git.addConfig( + 'core.sshCommand', + `ssh -i "${keyPath}" -o StrictHostKeyChecking=no -o IdentitiesOnly=yes` + ) + } + } + + // -> Rewritten rather than added to: the URL carries the credentials, so a remote left over from a + // previous configuration would still be reachable under its old ones + const remotes = await git.getRemotes() + for (const remote of remotes) { + await git.removeRemote(remote.name) + } + if (target.config.repoUrl) { + await git.addRemote('origin', remoteUrl(target)) + } + + return { + git, + root, + fingerprint: configFingerprint(target), + remoteReady: false, + queue: Promise.resolve() + } +} + +/** + * Run something against this target's repository, one operation at a time. + * + * Git takes a lock on the index for the length of a write, so two uploads landing together would have + * one of them fail on `index.lock` rather than wait. Serializing here is what makes concurrent saves + * safe, and the queue is per target because two targets are two working copies. + */ +async function withRepo(target: StorageTarget, run: (repo: Repo) => Promise): Promise { + let repo = repos.get(target.id) + if (!repo || repo.fingerprint !== configFingerprint(target)) { + repo = await prepareRepo(target) + repos.set(target.id, repo) + } + const entry = repo + const result = entry.queue.then( + () => run(entry), + () => run(entry) + ) + // -> The queue holds the settled outcome rather than the result, so one failed operation does not + // reject every operation queued behind it + entry.queue = result.catch(() => {}) + return result +} + +/** + * Bring the working copy onto the configured branch, having contacted the remote. + * + * The part of the 2.x module's `init` that costs a network round trip, split off so that only a sync + * pays for it. A repository with no remote configured is left as the purely local one it is. + * + * @returns Whether the remote already has the branch, which decides whether there is anything to pull + */ +async function ensureRemote(repo: Repo, target: StorageTarget): Promise<{ onRemote: boolean }> { + const branch = target.config.branch || 'main' + if (!target.config.repoUrl) { + return { onRemote: false } + } + if (!repo.remoteReady) { + await repo.git.raw(['remote', 'update', 'origin', '--prune']) + } + + const branches = await repo.git.branch(['-a']) + const onRemote = branches.all.includes(`remotes/origin/${branch}`) + const onLocal = branches.all.includes(branch) + + /* + A remote that does not have the branch yet is the ordinary state of a repository somebody has just + created, and the first push is what creates it — 2.x refused to start at all in that case, which + made an empty remote something an administrator had to go and fix by hand before the wiki would + talk to it. + + What is still worth refusing is a branch that exists nowhere on a remote that has other branches, + because that is a typo rather than a beginning. + */ + if (!onRemote && !onLocal) { + const remoteBranches = branches.all.filter((b) => b.startsWith('remotes/origin/')) + if (remoteBranches.length > 0) { + throw new Error( + `The branch "${branch}" does not exist locally or on the remote, which has ${remoteBranches + .map((b) => b.replace('remotes/origin/', '')) + .join(', ')}. Check the branch name, or create it on the remote first.` + ) + } + } else if (onRemote && branches.current !== branch) { + WIKI.logger.info(`(STORAGE/GIT) Checking out branch ${branch}...`) + await repo.git.checkout(branch) + } + + repo.remoteReady = true + return { onRemote } +} + +/** Whether the repository's own ignore rules exclude this path. */ +async function isIgnored(repo: Repo, relPath: string): Promise { + try { + return (await repo.git.checkIgnore([relPath])).length > 0 + } catch { + // -> `check-ignore` exits non-zero when nothing matches, which simple-git raises + return false + } +} + +/** + * Who to attribute a commit to: whoever made the change, or the target's configured stand-in. + * + * `alwaysUseDefaultAuthor` is the stand-in for everything, for a repository whose history should not + * carry the wiki's accounts — a public mirror, or an instance whose users did not agree to have their + * name and address published with every edit they make. The actor is not looked up at all in that + * case rather than looked up and discarded, so there is nothing to leak by mistake. + * + * The committer is the default author either way: it comes from the repository's own `user.name` and + * `user.email`, which `prepareRepo` sets from these same two settings. This decides the *author*, + * which is the half of a commit that git shows and that would otherwise name the person. + */ +async function commitAuthor(target: StorageTarget, actorId?: string): Promise { + const name = target.config.defaultName || FALLBACK_AUTHOR.name + const email = target.config.defaultEmail || FALLBACK_AUTHOR.email + if (target.config.alwaysUseDefaultAuthor) { + return `${name} <${email}>` + } + const actor = await WIKI.models.storage.actorFor(actorId) + return `${actor?.name || name} <${actor?.email || email}>` +} + +/** + * Commit whatever is staged at these paths, and nothing if that is nothing. + * + * The empty check is not an optimization. Every page save reaches this, and most of them save a page + * whose stored form has not actually changed — a re-publish, a tag reordered into the same order — so + * without it the repository would fill with empty commits that say a file changed when it did not. + */ +async function commitPaths( + repo: Repo, + target: StorageTarget, + paths: string[], + message: string, + actorId?: string +): Promise { + const staged = await repo.git.raw(['diff', '--cached', '--name-only', '--', ...paths]) + if (!staged.trim()) { + return false + } + await repo.git.commit(message, paths, { '--author': await commitAuthor(target, actorId) }) + return true +} + +/** Stage a written file and commit it, unless the repository is told to ignore it. */ +async function stageAndCommit( + repo: Repo, + target: StorageTarget, + relPath: string, + message: string, + actorId?: string +): Promise { + if (await isIgnored(repo, relPath)) { + return + } + await repo.git.add(relPath) + await commitPaths(repo, target, [relPath], message, actorId) +} + +/** + * Stage a deletion and commit it. + * + * `git rm` fails on a path git has never heard of, which is an ordinary situation here — a file + * excluded by `.gitignore`, or one deleted before this target was enabled — so the removal is staged + * from the index instead and a path that was not in it simply leaves nothing to commit. + */ +async function removeAndCommit( + repo: Repo, + target: StorageTarget, + relPath: string, + message: string, + actorId?: string +): Promise { + await fs.rm(absPathIn(repo.root, relPath), { force: true }) + try { + await repo.git.raw(['rm', '--cached', '--ignore-unmatch', '--', relPath]) + } catch (err: any) { + WIKI.logger.warn(`(STORAGE/GIT) Could not unstage ${relPath}: ${err.message}`) + return + } + await commitPaths(repo, target, [relPath], message, actorId) +} + +/** A page's path as a commit message names it: its locale and where it sits. */ +function pageLabel(ref: StoragePageRef): string { + return `[${ref.locale}] ${ref.path || '/'}` +} + +/** An asset's path as a commit message names it. */ +function assetLabel(ref: { locale: string; folderPath: string; fileName: string }): string { + return `[${ref.locale}] ${ref.folderPath ? `${ref.folderPath}/` : ''}${ref.fileName}` +} + +/** One entry of a `--name-status` diff. */ +interface DiffEntry { + status: string + segments: string[] + previousSegments?: string[] +} + +/** + * What changed between two commits, as paths this module can act on. + * + * Read with `--name-status -M` rather than through a diff summary because the three things that + * matter here are exactly what that reports: whether a path arrived, went, or moved. A summary of + * insertions and deletions cannot tell a deleted file from one emptied to nothing, and 2.x guessed at + * that from the line counts. + */ +async function changedPaths(repo: Repo, from: string, to: string): Promise { + const raw = await repo.git.raw(['diff', '--name-status', '-M', '-z', from, to]) + // -> `-z` because a path may contain anything at all, newlines included, and the fields are then + // NUL-separated: `status NUL path` for most, `Rxxx NUL old NUL new` for a rename + const fields = raw.split('\0').filter((f) => f !== '') + const entries: DiffEntry[] = [] + for (let i = 0; i < fields.length;) { + const status = fields[i++] + if (status.startsWith('R') || status.startsWith('C')) { + const previous = fields[i++] + const current = fields[i++] + if (!current) { + break + } + entries.push({ + status: 'R', + segments: current.split('/'), + previousSegments: previous.split('/') + }) + continue + } + const file = fields[i++] + if (!file) { + break + } + entries.push({ status: status[0], segments: file.split('/') }) + } + return entries +} + +/** + * Take a path the repository no longer has out of the wiki. + * + * The half of a pull that `importTree` cannot do, and the one that makes the remote authoritative + * rather than merely a source: a commit somebody pushed that deletes a file deletes the page or the + * asset here too. + * + * Which of the two it was has to be worked out from the tree rather than from the file, since the + * file is exactly what is no longer there. A page is filed under its editor's extension and addressed + * without one, so the stem is looked up first and only counts as the page if the page's own stored + * file name is the one that went — `readme.pdf` disappearing is not the markdown page `readme`. + * + * @returns What was deleted, for the report + */ +async function removeFromWiki( + target: StorageTarget, + segments: string[], + actorId: string +): Promise<'page' | 'asset' | null> { + const stored = WIKI.models.storage.parseStoredPath(target.siteId, segments) + if (!stored) { + return null + } + const rest = [...stored.segments] + const fileName = rest.pop()! + const folderPath = rest.join('/') + const ext = path.extname(fileName).replace(/^\./, '').toLowerCase() + const stem = fileName.slice(0, fileName.length - (ext ? ext.length + 1 : 0)) + + if (stem) { + const asPage = await WIKI.models.tree.getEntryAt({ + siteId: target.siteId, + locale: stored.locale, + parentPath: folderPath || null, + fileName: stem + }) + if ( + asPage?.type === 'page' && + (await WIKI.models.pages.storageFileNameOf(asPage.id)) === fileName + ) { + await WIKI.models.pages.deletePage(target.siteId, asPage.id, { + id: actorId, + permissions: ['manage:system'] + }) + return 'page' + } + } + + const asAsset = await WIKI.models.tree.getEntryAt({ + siteId: target.siteId, + locale: stored.locale, + parentPath: folderPath || null, + fileName + }) + if (asAsset?.type === 'asset') { + await WIKI.models.assets.deleteAsset(target.siteId, asAsset.id, actorId) + return 'asset' + } + return null +} + +/** Every file this target should be holding, written into the working copy. */ +async function writeEverything( + repo: Repo, + target: StorageTarget +): Promise<{ pages: number; assets: number; unstored: number; unreadable: number }> { + const counts = { pages: 0, assets: 0, unstored: 0, unreadable: 0 } + + for (const asset of await WIKI.models.assets.listStoredAssets(target.siteId)) { + const contentType = WIKI.models.storage.contentTypeFor( + target.siteId, + asset.kind, + asset.fileSize + ) + if (!target.contentTypes.activeTypes.includes(contentType)) { + continue + } + const relPath = assetRelPath(target, asset) + if (!relPath) { + counts.unstored++ + continue + } + const data = await WIKI.models.storage.getAsset(asset) + if (!data) { + counts.unreadable++ + continue + } + const filePath = absPathIn(repo.root, relPath) + await fs.mkdir(path.dirname(filePath), { recursive: true }) + await fs.writeFile(filePath, data) + counts.assets++ + } + + if (target.contentTypes.activeTypes.includes('pages')) { + for (const { ref, content } of await WIKI.models.pages.listForStorage(target.siteId)) { + const relPath = pageRelPath(target, ref) + if (!relPath) { + counts.unstored++ + continue + } + const filePath = absPathIn(repo.root, relPath) + await fs.mkdir(path.dirname(filePath), { recursive: true }) + await fs.writeFile(filePath, serializePage(ref, content)) + counts.pages++ + } + } + + return counts +} + +/** What an import run did, in words. */ +function describeImport(summary: ImportSummary | null): string { + if (!summary) { + return 'There is nothing in the local repository yet. Run a Force Sync to fetch from the remote first.' + } + const parts = [] + if (summary.pages > 0 || summary.assets > 0) { + parts.push(`Imported ${summary.pages} page(s) and ${summary.assets} asset(s).`) + } else { + parts.push('There was nothing to import.') + } + if (summary.skipped > 0) { + parts.push(`${summary.skipped} could not replace what is at their path and were left alone.`) + } + if (summary.failed > 0) { + parts.push(`${summary.failed} could not be imported - see the server log.`) + } + return parts.join(' ') +} + +/** + * Git storage module + * + * Keeps the site's content as a git repository: the same tree the local disk target writes, committed + * as it changes and synchronized with a remote. What that buys over the disk target is history — every + * edit is a commit by the person who made it, so the repository is a record of the wiki and not only a + * copy of it — and a second place the content lives that is not this machine. + * + * The layout, the front matter and what makes a file a page are all `helpers/storageFiles.ts`, shared + * with the disk target. This module is what git adds on top: a commit per change, and a sync. + * + * **Local writes, batched network.** A page save commits and returns; nothing waits on a remote. The + * push and the pull happen in `sync`, which the scheduler runs every few minutes and an administrator + * can run on demand. That is why `prepareRepo` and `ensureRemote` are separate: an unreachable remote + * must not be able to make the wiki slow to edit, or fail an upload. + * + * **A pull is authoritative.** What it brings in is applied to the wiki, replacing what is there — and + * a commit that deleted a file deletes the page or the asset here too, which is the whole point of + * pointing a wiki at a repository other people push to. It also means push access to the remote is + * effectively write access to the wiki, which is worth knowing before configuring one. + * + * Everything runs through `withRepo`, one operation at a time per target: git locks its index for the + * length of a write, so two concurrent uploads would otherwise have one of them fail outright. + */ +const gitStorage: StorageModule = { + canStore(target, ref) { + return WIKI.models.storage.pathPrefixFor(target.siteId, ref.locale) !== null + }, + + async putAsset(target, ref, data) { + const relPath = assetRelPath(target, ref) + if (!relPath) { + throw new Error( + `${target.title} has no path for ${ref.locale} content, so ${ref.fileName} cannot be stored there.` + ) + } + await withRepo(target, async (repo) => { + const filePath = absPathIn(repo.root, relPath) + await fs.mkdir(path.dirname(filePath), { recursive: true }) + await fs.writeFile(filePath, data) + await stageAndCommit(repo, target, relPath, `docs: upload ${assetLabel(ref)}`, ref.actorId) + }) + }, + + async getAsset(target, ref) { + const relPath = assetRelPath(target, ref) + if (!relPath) { + return null + } + // -> Read straight off the working copy rather than out of git: what the wiki serves is the + // current state of the branch, which is exactly what is checked out + try { + return await fs.readFile(absPathIn(repoDir(target), relPath)) + } catch (err: any) { + if (err.code !== 'ENOENT') { + throw err + } + return null + } + }, + + async deleteAsset(target, ref) { + const relPath = assetRelPath(target, ref) + if (!relPath) { + return + } + await withRepo(target, (repo) => + removeAndCommit(repo, target, relPath, `docs: delete ${assetLabel(ref)}`, ref.actorId) + ) + }, + + async moveAsset(target, ref, previous) { + const from = assetRelPath(target, { ...ref, ...previous }) + const to = assetRelPath(target, ref) + await withRepo(target, async (repo) => { + const outcome = await moveStored(repo.root, from, to) + if (outcome === 'nothing') { + return + } + const paths = outcome === 'moved' ? [from!, to!] : [from!] + for (const relPath of paths) { + await repo.git.add(['-A', '--', relPath]) + } + await commitPaths( + repo, + target, + paths, + outcome === 'moved' + ? `docs: rename ${assetLabel({ ...ref, ...previous })} to ${assetLabel(ref)}` + : `docs: delete ${assetLabel({ ...ref, ...previous })}`, + ref.actorId + ) + }) + }, + + async putPage(target, ref, page) { + const relPath = pageRelPath(target, ref) + if (!relPath) { + return + } + await withRepo(target, async (repo) => { + const filePath = absPathIn(repo.root, relPath) + // -> Which of the two verbs the commit gets. Read before the write, since afterwards every page + // looks like one that was already there. + const existed = await fs + .access(filePath) + .then(() => true) + .catch(() => false) + await fs.mkdir(path.dirname(filePath), { recursive: true }) + await fs.writeFile(filePath, serializePage(ref, page), 'utf8') + await stageAndCommit( + repo, + target, + relPath, + `docs: ${existed ? 'update' : 'create'} ${pageLabel(ref)}`, + ref.actorId + ) + }) + }, + + async deletePage(target, ref) { + const relPath = pageRelPath(target, ref) + if (!relPath) { + return + } + await withRepo(target, (repo) => + removeAndCommit(repo, target, relPath, `docs: delete ${pageLabel(ref)}`, ref.actorId) + ) + }, + + async movePage(target, ref, previousPath) { + const from = pageRelPath(target, { ...ref, path: previousPath }) + const to = pageRelPath(target, ref) + await withRepo(target, async (repo) => { + const outcome = await moveStored(repo.root, from, to) + if (outcome === 'nothing') { + return + } + const paths = outcome === 'moved' ? [from!, to!] : [from!] + for (const relPath of paths) { + await repo.git.add(['-A', '--', relPath]) + } + await commitPaths( + repo, + target, + paths, + outcome === 'moved' + ? `docs: rename ${pageLabel({ ...ref, path: previousPath })} to ${pageLabel(ref)}` + : `docs: delete ${pageLabel({ ...ref, path: previousPath })}`, + ref.actorId + ) + }) + }, + + /** + * Pull from the remote, push to it, and apply what came back. + * + * The direction is the target's `syncMode`, and it decides which half runs: `push` never takes + * anything in and force-pushes, so the wiki wins; `pull` never sends anything, so the remote does; + * `sync` does both, rebasing the wiki's commits on top of what it pulled. + * + * Whatever a pull brought in is then applied to the wiki — created, replaced, or deleted. That is + * done from a `--name-status` diff between the commit the branch was on before and the one it is on + * now, rather than by walking the tree: a sync runs every few minutes, and reading every file in the + * repository each time to find the two that changed would be absurd. + */ + async sync(target: StorageTarget, actorId: string): Promise { + const mode = target.config.syncMode || 'sync' + const branch = target.config.branch || 'main' + return withRepo(target, async (repo) => { + if (!target.config.repoUrl) { + return 'No repository URI is configured, so there is nothing to sync with. Commits are being made locally.' + } + const { onRemote } = await ensureRemote(repo, target) + + const before = await repo.git.revparse(['HEAD']).catch(() => null) + const parts: string[] = [] + + // -> Nothing to pull from a branch the remote does not have yet; the push below creates it + if (mode !== 'push' && onRemote) { + WIKI.logger.info(`(STORAGE/GIT) Pulling from origin/${branch}...`) + await repo.git.pull('origin', branch, ['--rebase']) + } + if (mode !== 'pull') { + WIKI.logger.info(`(STORAGE/GIT) Pushing to origin/${branch}...`) + // -> `--force` only in push mode, which is the mode that says the wiki is the authority + await repo.git.push( + 'origin', + branch, + mode === 'push' ? ['--signed=if-asked', '--force'] : ['--signed=if-asked'] + ) + } + + if (mode !== 'push' && onRemote) { + const after = await repo.git.revparse(['HEAD']).catch(() => null) + if (!after) { + return 'Synced. The repository has no commits yet.' + } + parts.push(await applyIncoming(repo, target, before, after, actorId)) + } else if (mode === 'pull') { + parts.push( + 'Synced. The remote does not have this branch yet, so there was nothing to pull.' + ) + } else { + parts.push('Pushed to the remote.') + } + return parts.join(' ') + }) + }, + + /** + * Write and commit everything this target should be holding but has never been given. + * + * The way in for content that predates the target: a wiki that ran for a year before git was + * enabled has a repository with nothing in it, and nothing in the ordinary course of things ever + * goes back for those pages. One commit, by the administrator who asked for it, since it is their + * action and not a hundred authors' edits. + */ + async syncUntracked(target: StorageTarget, actorId: string): Promise { + return withRepo(target, async (repo) => { + const counts = await writeEverything(repo, target) + await repo.git.add(['-A', '--', '.']) + const committed = await commitPaths( + repo, + target, + ['.'], + 'docs: add all untracked content', + actorId + ) + WIKI.logger.info( + `(STORAGE/GIT) Wrote ${counts.pages} page(s) and ${counts.assets} asset(s) to ${repo.root} [ OK ]` + ) + const parts = [ + committed + ? `Committed the untracked part of ${counts.pages} page(s) and ${counts.assets} asset(s).` + : `Wrote ${counts.pages} page(s) and ${counts.assets} asset(s); all of it was already tracked.` + ] + if (counts.unreadable > 0) { + parts.push(`${counts.unreadable} asset(s) could not be read and were skipped.`) + } + if (counts.unstored > 0) { + const { primaryLocale } = WIKI.models.storage.pathLayoutFor(target.siteId) + parts.push( + `${counts.unstored} item(s) are not in the ${primaryLocale} locale, which is the only one this site stores.` + ) + } + return parts.join(' ') + }) + }, + + /** + * Take everything in the working copy into the wiki, whatever the last commit did. + * + * For a repository that already had content before this target existed: the sync only ever looks at + * what changed between two commits, so a repository cloned with a thousand files in it has none of + * them in the wiki. The repository wins every collision, consistent with what a pull does — this is + * the same direction, applied to everything at once instead of to a diff. + */ + async importAll(target: StorageTarget, actorId: string): Promise { + return withRepo(target, async (repo) => + describeImport(await importTree({ target, root: repo.root, actorId, overwrite: true })) + ) + }, + + /** + * Throw the working copy away and take it again from the remote. + * + * The answer to a working copy git can no longer make sense of — unrelated histories, a rebase that + * cannot be finished, an index that will not unlock. Nothing about the remote changes and nothing is + * committed, so the cost is any commit that only existed here. + */ + async purge(target: StorageTarget): Promise { + const root = repoDir(target) + return withRepo(target, async () => { + WIKI.logger.info(`(STORAGE/GIT) Purging the local repository at ${root}...`) + await fs.rm(root, { recursive: true, force: true }) + // -> Dropped rather than reused: it holds a SimpleGit bound to a directory that no longer + // exists, and the next operation is what sets the replacement up + repos.delete(target.id) + if (!target.config.repoUrl) { + return 'The local repository has been emptied. It will be initialized again on the next change.' + } + const repo = await prepareRepo(target) + repos.set(target.id, repo) + await ensureRemote(repo, target) + return 'The local repository has been emptied and taken again from the remote. Run Import Everything if the wiki should now say what it holds.' + }) + } +} + +/** + * Apply what a pull brought in to the wiki. + * + * Split out of `sync` because it is the interesting half and reads as its own thing: a diff, and then + * two lists — what to take in and what to take out. A file that arrived or changed is imported with + * the repository winning; one that went is deleted. A rename is both, in that order, which is what + * moves a page rather than losing its history to a delete and a create. + */ +async function applyIncoming( + repo: Repo, + target: StorageTarget, + before: string | null, + after: string, + actorId: string +): Promise { + // -> Nothing came back, which is the ordinary outcome of a sync and worth saying plainly + if (before === after) { + return 'Synced. Nothing had changed on the remote.' + } + + let files: StoredFile[] | null + const removals: string[][] = [] + if (!before) { + // -> Nothing to diff against: the branch had no commits here at all, so everything in it is new + files = await walkStored(repo.root) + } else { + const changes = await changedPaths(repo, before, after) + files = [] + for (const change of changes) { + if (change.status === 'D') { + removals.push(change.segments) + continue + } + if (change.status === 'R' && change.previousSegments) { + removals.push(change.previousSegments) + } + files.push({ + filePath: absPathIn(repo.root, change.segments.join('/')), + segments: change.segments + }) + } + } + + const summary = await importTree({ + target, + root: repo.root, + actorId, + overwrite: true, + files + }) + + let deletedPages = 0 + let deletedAssets = 0 + for (const segments of removals) { + try { + const removed = await removeFromWiki(target, segments, actorId) + if (removed === 'page') { + deletedPages++ + } else if (removed === 'asset') { + deletedAssets++ + } + } catch (err: any) { + // -> One entry the wiki could not let go of must not stop the rest of the commit being applied + WIKI.logger.warn(`(STORAGE/GIT) Could not delete ${segments.join('/')} [ SKIPPED ]`) + WIKI.logger.warn(err.message) + } + } + + const parts = ['Synced.'] + if (summary && (summary.pages > 0 || summary.assets > 0)) { + parts.push(`Took in ${summary.pages} page(s) and ${summary.assets} asset(s).`) + } + if (deletedPages > 0 || deletedAssets > 0) { + parts.push(`Deleted ${deletedPages} page(s) and ${deletedAssets} asset(s) the remote removed.`) + } + if (summary && summary.failed > 0) { + parts.push(`${summary.failed} could not be imported - see the server log.`) + } + if (parts.length === 1) { + parts.push('Nothing the remote changed affected this wiki.') + } + return parts.join(' ') +} + +export default gitStorage diff --git a/backend/modules/storage/s3/definition.yml b/backend/modules/storage/s3/definition.yml new file mode 100644 index 000000000..6167e9a0d --- /dev/null +++ b/backend/modules/storage/s3/definition.yml @@ -0,0 +1,74 @@ +key: s3 +title: S3 Object Storage +icon: '/_assets/icons/ultraviolet-amazon-web-services.svg' +banner: '/_assets/storage/s3.jpg' +description: Amazon S3 and any store that speaks its API - Cloudflare R2, DigitalOcean Spaces, Backblaze B2, Wasabi, MinIO and the rest. Leave the endpoint empty for AWS itself, or point it at whichever service you use. +assetDelivery: + isDirectAccessSupported: true +contentTypes: + defaultTypesEnabled: ['images', 'documents', 'others', 'large'] +props: + endpoint: + type: String + title: Endpoint + default: '' + hint: Leave empty for AWS S3. Otherwise the full URL of the service, e.g. https://.r2.cloudflarestorage.com for Cloudflare R2, https://nyc3.digitaloceanspaces.com for DigitalOcean Spaces. + icon: dns + order: 1 + region: + type: String + title: Region + default: us-east-1 + hint: The region the bucket lives in. Stores that do not have regions usually want "auto" (Cloudflare R2) or accept anything (MinIO). + icon: geography + order: 2 + bucket: + type: String + title: Bucket Name + default: '' + hint: The bucket to store content in. It must already exist - this target will not create it. + icon: open-box + order: 3 + accessKeyId: + type: String + title: Access Key ID + default: '' + hint: Leave both this and the secret empty to use the credentials the machine already has - an IAM role, or the standard AWS environment variables. + icon: 3d-touch + order: 4 + secretAccessKey: + type: String + title: Secret Access Key + default: '' + hint: The secret for the access key above. + icon: key + sensitive: true + order: 5 + storageClass: + type: String + title: Storage Class + default: STANDARD + hint: What new objects are stored as. An AWS concept - most compatible stores ignore it, and leaving it at Standard is always safe. + icon: scan-stock + order: 6 + enum: + - STANDARD|Standard + - STANDARD_IA|Standard Infrequent Access + - INTELLIGENT_TIERING|Intelligent Tiering + - ONEZONE_IA|One Zone Infrequent Access + - REDUCED_REDUNDANCY|Reduced Redundancy + - GLACIER_IR|Glacier Instant Retrieval + - GLACIER|Glacier Flexible Retrieval + - DEEP_ARCHIVE|Glacier Deep Archive + forcePathStyle: + type: Boolean + title: Force Path Style + default: false + hint: Address the bucket as a path (endpoint/bucket/key) rather than as a subdomain. Needed by MinIO and some self-hosted stores; leave off for AWS, R2 and Spaces. + icon: filtration + order: 10 +actions: + exportAll: + label: Export Everything + hint: Write a copy of every page and asset this target is configured to hold to the bucket, overwriting whatever is already there. Nothing in the database is changed and nothing is moved, so this is how content created before the target was enabled gets onto it. + icon: this-way-up diff --git a/backend/modules/storage/s3/storage.ts b/backend/modules/storage/s3/storage.ts new file mode 100644 index 000000000..d7b432caa --- /dev/null +++ b/backend/modules/storage/s3/storage.ts @@ -0,0 +1,208 @@ +import { + CopyObjectCommand, + DeleteObjectCommand, + GetObjectCommand, + PutObjectCommand, + S3Client +} from '@aws-sdk/client-s3' +import { getSignedUrl } from '@aws-sdk/s3-request-presigner' +import { objectStorageModule, signingBaseUrl } from '../../../helpers/storageObjects.ts' +import type { ObjectStoreClient } from '../../../helpers/storageObjects.ts' +import type { StorageTarget } from '../../../models/storage.ts' + +/** Live clients, keyed by target. See `clientFor`. */ +const clients = new Map() + +/** The settings a client is built from — a change to any of them needs a new one. */ +function configFingerprint(target: StorageTarget): string { + const c = target.config + return JSON.stringify([c.endpoint, c.region, c.accessKeyId, c.secretAccessKey, c.forcePathStyle]) +} + +/** + * The S3 client for this target, built once and kept. + * + * Rebuilt when the configuration changes, so a rotated key takes effect on the next operation rather + * than at the next restart. + * + * **Credentials are optional.** Left empty, the SDK falls back to its own chain — an IAM role on the + * instance, the standard `AWS_*` environment variables, a shared credentials file — which is how a + * deployment avoids putting a long-lived secret in the database at all. + */ +function clientFor(target: StorageTarget): S3Client { + const fingerprint = configFingerprint(target) + const cached = clients.get(target.id) + if (cached && cached.fingerprint === fingerprint) { + return cached.client + } + const { endpoint, region, accessKeyId, secretAccessKey, forcePathStyle } = target.config + const client = new S3Client({ + region: region || 'us-east-1', + ...(endpoint ? { endpoint } : {}), + ...(forcePathStyle ? { forcePathStyle: true } : {}), + ...(accessKeyId && secretAccessKey ? { credentials: { accessKeyId, secretAccessKey } } : {}) + }) + clients.set(target.id, { client, fingerprint }) + return client +} + +/** + * The client and bucket a signature should be made against. + * + * SigV4 covers the `Host` header, so a URL signed for the bucket's own address and then rewritten + * onto a CDN domain carries a signature for the wrong host and is rejected. The domain has to be + * signed *for*, which means a client pointed at it rather than the ordinary client with its output + * edited afterwards. + * + * Which of the two forms that takes depends on what sits at the domain, and `forcePathStyle` is the + * target's existing answer to exactly that question for the store itself: + * + * - **off** — the domain *is* the bucket, which is what a Cloudflare R2 custom domain or a Spaces CDN + * endpoint is. The SDK spells this `bucketEndpoint`, and it means the `Bucket` *parameter* carries + * the URL: passing the bucket's name alongside it fails outright. + * - **on** — the bucket is the first path segment, which is what a reverse proxy onto MinIO looks + * like, and the endpoint and the bucket name are then both ordinary. + * + * Not cached: signing is per request, and an administrator changing the base URL must not have to + * wait for anything to expire before seeing it. + */ +function signingTargetFor( + target: StorageTarget, + baseUrl: string | null +): { client: S3Client; bucket: string } { + if (!baseUrl) { + return { client: clientFor(target), bucket: target.config.bucket } + } + const { region, accessKeyId, secretAccessKey, forcePathStyle } = target.config + const credentials = + accessKeyId && secretAccessKey ? { credentials: { accessKeyId, secretAccessKey } } : {} + if (forcePathStyle) { + return { + client: new S3Client({ + region: region || 'us-east-1', + endpoint: baseUrl, + forcePathStyle: true, + ...credentials + }), + bucket: target.config.bucket + } + } + return { + client: new S3Client({ region: region || 'us-east-1', bucketEndpoint: true, ...credentials }), + bucket: baseUrl + } +} + +/** Whether the store is telling us the key simply is not there. */ +function isNotFound(err: any): boolean { + return ( + err?.name === 'NoSuchKey' || err?.name === 'NotFound' || err?.$metadata?.httpStatusCode === 404 + ) +} + +const s3Client: ObjectStoreClient = { + async put(target, key, data, contentType) { + await clientFor(target).send( + new PutObjectCommand({ + Bucket: target.config.bucket, + Key: key, + Body: data, + ContentType: contentType, + // -> Omitted rather than sent as Standard: a compatible store that does not implement + // storage classes will reject the header outright rather than ignore it + ...(target.config.storageClass && target.config.storageClass !== 'STANDARD' + ? { StorageClass: target.config.storageClass } + : {}) + }) + ) + }, + + async get(target, key) { + try { + const resp = await clientFor(target).send( + new GetObjectCommand({ Bucket: target.config.bucket, Key: key }) + ) + const bytes = await resp.Body?.transformToByteArray() + return bytes ? Buffer.from(bytes) : null + } catch (err: any) { + if (isNotFound(err)) { + // -> This target does not have the file: enabled after the upload, or removed from outside + // the wiki. Not a fault — the caller asks the next target. + return null + } + throw err + } + }, + + async remove(target, key) { + try { + await clientFor(target).send( + new DeleteObjectCommand({ Bucket: target.config.bucket, Key: key }) + ) + } catch (err: any) { + // -> S3 itself answers a delete of a missing key with success; not every compatible store does + if (!isNotFound(err)) { + throw err + } + } + }, + + async copy(target, fromKey, toKey) { + try { + await clientFor(target).send( + new CopyObjectCommand({ + Bucket: target.config.bucket, + // -> The source is bucket-qualified and URI-encoded, which is the one part of this API that + // does not take a plain key: a `#` or a `+` in a file name would otherwise be read as + // part of the URL rather than as part of the name + CopySource: encodeURI(`${target.config.bucket}/${fromKey}`), + Key: toKey + }) + ) + return true + } catch (err: any) { + if (isNotFound(err)) { + return false + } + throw err + } + }, + + async presign(target, { key, expiresInSeconds, contentType, downloadAs }) { + const { client, bucket } = signingTargetFor(target, signingBaseUrl(target)) + /* + The response headers travel in the signature rather than being left to the object's own + metadata: the wiki knows what it thinks the file is and whether this request was a download, + and neither of those is necessarily what was stored — an object written by an older instance, + or one uploaded straight into the bucket, carries whatever it carries. + */ + return getSignedUrl( + client, + new GetObjectCommand({ + Bucket: bucket, + Key: key, + ResponseContentType: contentType, + ...(downloadAs + ? { + ResponseContentDisposition: `attachment; filename="${encodeURIComponent(downloadAs)}"` + } + : {}) + }), + { expiresIn: expiresInSeconds } + ) + } +} + +/** + * S3 object storage module + * + * Amazon S3 and everything that speaks its API. One module rather than the three that 2.x shipped — + * an AWS one, a DigitalOcean one and a custom one — because the difference between them is an + * endpoint and a region, and treating that as a preset meant a new module for every service that + * appeared. Empty endpoint is AWS; anything else is whichever store the URL points at. + * + * The keys are the same paths the disk target writes, so a bucket and a folder hold the wiki's content + * laid out identically, and the shape of both is the site's `pathPrefixFor` to decide. See + * `helpers/storageObjects.ts` for everything above the four calls below. + */ +export default objectStorageModule(s3Client) diff --git a/backend/modules/storage/sftp/definition.yml b/backend/modules/storage/sftp/definition.yml new file mode 100644 index 000000000..bfdb33b9b --- /dev/null +++ b/backend/modules/storage/sftp/definition.yml @@ -0,0 +1,100 @@ +key: sftp +title: SFTP +icon: '/_assets/icons/ultraviolet-nas.svg' +banner: '/_assets/storage/ssh.jpg' +description: Store the wiki's content as ordinary files on a remote server over SSH. The same tree the local disk target writes, on a machine that is not this one. Meant as a copy rather than a source, so it cannot be chosen under Content Delivery. +vendor: 'Wiki.js' +website: 'https://js.wiki' +assetDelivery: + isDirectAccessSupported: false + # -> A place to keep a copy of the site's content, not one to serve it from: every image on every + # page would be an SSH round trip. Written to, exported to and imported from as normal; simply + # never offered under Content Delivery. + isDeliverySupported: false +contentTypes: + defaultTypesEnabled: ['pages', 'images', 'documents', 'others', 'large'] +props: + host: + type: String + title: Host + default: '' + hint: Hostname or IP of the remote SSH server. + icon: dns + order: 1 + port: + type: Number + title: Port + default: 22 + hint: SSH port of the remote server. + icon: ethernet-off + order: 2 + authMode: + type: String + title: Authentication Method + default: 'privateKey' + hint: Whether to use Private Key or Password-based authentication. A private key is highly recommended for best security. + icon: grand-master-key + enum: + - privateKey|Private Key + - password|Password + enumDisplay: buttons + order: 3 + username: + type: String + title: Username + default: '' + hint: Username for authentication. + icon: test-account + order: 4 + privateKey: + type: String + title: Private Key Contents + default: '' + hint: Contents of the private key, including the BEGIN and END lines. + icon: key + multiline: true + sensitive: true + order: 5 + if: + - { key: 'authMode', eq: 'privateKey' } + passphrase: + type: String + title: Private Key Passphrase + default: '' + hint: Passphrase if the private key is encrypted, leave empty otherwise. + icon: password + sensitive: true + order: 6 + if: + - { key: 'authMode', eq: 'privateKey' } + password: + type: String + title: Password + default: '' + hint: Password for authentication. + icon: password + sensitive: true + order: 6 + if: + - { key: 'authMode', eq: 'password' } + basePath: + type: String + title: Base Directory Path + default: '/var/wiki' + hint: Where this site's folder tree is written on the remote server. It must already exist and be writable by the user above. Give each site its own path unless you turn on Add Site ID Prefix under Configuration. + icon: symlink-directory + order: 7 +actions: + exportAll: + label: Export Everything + hint: Write a copy of every page and asset this target is configured to hold to the remote server, overwriting whatever is already there. Nothing in the database is changed and nothing is moved, so this is how content created before the target was enabled gets onto it. + icon: this-way-up + importAll: + label: Import Everything + hint: Take every page and asset on the remote server that the wiki does not have yet into the wiki. A file is imported as a page if its extension is one of the site's Page Extensions, or if it declares an editor in its front matter; everything else is imported as an asset. Anything already at the same path is left alone on both sides, so this is safe to run again. + icon: database-daily-import + importAllOverwrite: + label: Import Everything and Overwrite + hint: The same walk, with the remote server winning every collision. For a restore, or a tree edited on the server that is meant to be taken as the new truth. + warn: This replaces what the wiki currently has wherever a remote file lands on it. An overwritten page keeps its previous version in its history, but a file has none - the bytes it replaces are gone from every storage target holding them. + icon: database-restore diff --git a/backend/modules/storage/sftp/storage.ts b/backend/modules/storage/sftp/storage.ts new file mode 100644 index 000000000..8086ea8b2 --- /dev/null +++ b/backend/modules/storage/sftp/storage.ts @@ -0,0 +1,515 @@ +import path from 'node:path' +import SftpClient from 'ssh2-sftp-client' +import { + assetRelPath, + importTree, + pageRelPath, + serializePage +} from '../../../helpers/storageFiles.ts' +import type { ImportSummary, StoredFile } from '../../../helpers/storageFiles.ts' +import type { StorageModule, StorageTarget } from '../../../models/storage.ts' + +/** Where files go when the target has no base path configured, matching the definition default. */ +const DEFAULT_BASE_PATH = '/var/wiki' + +/** Names never walked by an import, as on the local disk. */ +const IGNORED_NAME = /^\./ + +/** One live connection, plus the queue that keeps it to one operation at a time. */ +interface Connection { + client: SftpClient + fingerprint: string + queue: Promise +} + +const connections = new Map() + +/** The remote root this target writes under, without a trailing slash. */ +function baseDir(target: StorageTarget): string { + return (target.config.basePath || DEFAULT_BASE_PATH).replace(/\/+$/, '') || '/' +} + +/** Everything a connection is made from — a change to any of it needs a new one. */ +function configFingerprint(target: StorageTarget): string { + const c = target.config + return JSON.stringify([ + c.host, + c.port, + c.authMode, + c.username, + c.privateKey, + c.passphrase, + c.password + ]) +} + +/** + * The absolute remote path of a stored file, refusing anything that would land outside the root. + * + * `path.posix` throughout rather than `path`, because the shape of the remote file system has nothing + * to do with the shape of this one: a wiki running on Windows still talks to an SSH server in + * slashes, and `path.win32.resolve` would turn every one of these into a backslash path the server + * has never heard of. + */ +function remotePath(target: StorageTarget, relPath: string): string { + const base = baseDir(target) + const resolved = path.posix.resolve(base, relPath) + if (resolved !== base && !resolved.startsWith(base === '/' ? '/' : `${base}/`)) { + throw new Error(`The stored path "${relPath}" resolves outside the base directory.`) + } + return resolved +} + +/** + * Run something against this target's connection, one operation at a time. + * + * A single SFTP client multiplexes badly — `ssh2-sftp-client` is explicit that concurrent operations + * on one instance are not supported — so the queue is what makes two simultaneous uploads safe. Per + * target, because two targets are two servers. + * + * A connection that fails is dropped rather than reused: the failure may be the connection itself, + * and the next operation is what re-establishes it. This is also the reconnect path after the server + * has timed the session out, which for a wiki that uploads a file once a week it always will have. + */ +async function withClient( + target: StorageTarget, + run: (client: SftpClient) => Promise +): Promise { + const fingerprint = configFingerprint(target) + let connection = connections.get(target.id) + if (connection && connection.fingerprint !== fingerprint) { + await connection.client.end().catch(() => {}) + connections.delete(target.id) + connection = undefined + } + if (!connection) { + connection = { client: new SftpClient(), fingerprint, queue: Promise.resolve() } + connections.set(target.id, connection) + connection.queue = connect(target, connection.client).catch((err) => { + connections.delete(target.id) + throw err + }) + } + + const entry = connection + const result = entry.queue.then( + () => run(entry.client), + // -> The previous operation failed; this one still gets its turn, on a connection that may well + // have been replaced underneath it + () => run(entry.client) + ) + entry.queue = result.catch(() => {}) + try { + return await result + } catch (err: any) { + if (isConnectionError(err)) { + await entry.client.end().catch(() => {}) + connections.delete(target.id) + } + throw err + } +} + +/** Open the connection and check the base directory is actually there. */ +async function connect(target: StorageTarget, client: SftpClient): Promise { + const { host, port, authMode, username, privateKey, passphrase, password } = target.config + WIKI.logger.info(`(STORAGE/SFTP) Connecting to ${username}@${host}...`) + await client.connect({ + host, + port: Number(port) || 22, + username, + ...(authMode === 'password' + ? { password } + : { privateKey, ...(passphrase ? { passphrase } : {}) }) + }) + const base = baseDir(target) + if (!(await client.exists(base))) { + // -> Not created: the base path is where somebody has decided this site's content belongs, and + // a typo in it should be a refusal rather than a new directory nobody meant + throw new Error( + `The base directory ${base} does not exist on the remote server, or the user cannot see it.` + ) + } +} + +/** Whether this looks like the session rather than the file being the problem. */ +function isConnectionError(err: any): boolean { + const message = String(err?.message ?? '') + return ( + /connect|closed|ECONNRESET|ETIMEDOUT|EPIPE|not connected|handshake|authentication/i.test( + message + ) && !isMissing(err) + ) +} + +/** Whether the server is telling us the file simply is not there. */ +function isMissing(err: any): boolean { + return err?.code === 2 || /no such file|ENOENT/i.test(String(err?.message ?? '')) +} + +/** Read every file under a remote directory, skipping anything hidden. */ +async function walkRemote(client: SftpClient, root: string, dir: string): Promise { + const found: StoredFile[] = [] + let entries + try { + entries = await client.list(dir) + } catch (err: any) { + if (isMissing(err)) { + return found + } + throw err + } + for (const entry of entries) { + if (IGNORED_NAME.test(entry.name)) { + continue + } + const full = path.posix.join(dir, entry.name) + if (entry.type === 'd') { + found.push(...(await walkRemote(client, root, full))) + } else if (entry.type === '-') { + found.push({ + filePath: full, + segments: path.posix.relative(root, full).split('/') + }) + } + } + return found +} + +/** Write a file, creating the directories above it. */ +async function writeRemote( + client: SftpClient, + target: StorageTarget, + relPath: string, + data: Buffer +): Promise { + const filePath = remotePath(target, relPath) + const dir = path.posix.dirname(filePath) + // -> `true` is recursive, and an existing directory is not an error to this client + await client.mkdir(dir, true).catch(() => {}) + await client.put(data, filePath) +} + +/** Remove a file, and any directories it leaves empty, stopping at the first one still in use. */ +async function removeRemote( + client: SftpClient, + target: StorageTarget, + relPath: string +): Promise { + const base = baseDir(target) + const filePath = remotePath(target, relPath) + try { + await client.delete(filePath) + } catch (err: any) { + if (!isMissing(err)) { + throw err + } + } + let dir = path.posix.dirname(filePath) + while (dir !== base && dir.startsWith(`${base}/`)) { + try { + await client.rmdir(dir) + } catch { + // -> Not empty, or another request is writing into it. Best effort, exactly as on disk. + return + } + dir = path.posix.dirname(dir) + } +} + +/** Follow a rename, where either end may be a locale this site does not store. */ +async function moveRemote( + client: SftpClient, + target: StorageTarget, + fromRel: string | null, + toRel: string | null +): Promise { + if (!fromRel) { + return + } + if (!toRel) { + await removeRemote(client, target, fromRel) + return + } + const from = remotePath(target, fromRel) + const to = remotePath(target, toRel) + await client.mkdir(path.posix.dirname(to), true).catch(() => {}) + try { + await client.rename(from, to) + } catch (err: any) { + // -> Nothing there to move: this target was enabled after the file was uploaded + if (!isMissing(err)) { + throw err + } + return + } + let dir = path.posix.dirname(from) + const base = baseDir(target) + while (dir !== base && dir.startsWith(`${base}/`)) { + try { + await client.rmdir(dir) + } catch { + return + } + dir = path.posix.dirname(dir) + } +} + +/** What an import run did, in the words the two import actions report it with. */ +function describeImport(summary: ImportSummary | null, overwrite: boolean): string { + if (!summary) { + return 'There is nothing in the base directory for this site yet.' + } + const verb = overwrite ? 'Imported or replaced' : 'Imported' + const parts = [] + if (summary.pages > 0) { + parts.push(`${verb} ${summary.pages} page(s).`) + } + if (summary.assets > 0) { + parts.push(`${verb} ${summary.assets} asset(s).`) + } + if (parts.length < 1) { + parts.push(overwrite ? 'There was nothing to import.' : 'There was nothing new to import.') + } + if (summary.skipped > 0) { + parts.push( + overwrite + ? `${summary.skipped} could not replace what is at their path and were left alone.` + : `${summary.skipped} were already in the wiki and were left alone.` + ) + } + if (summary.failed > 0) { + parts.push(`${summary.failed} could not be imported - see the server log.`) + } + return parts.join(' ') +} + +/** + * SFTP storage module + * + * The local disk target, on a machine that is not this one. The same tree, laid out the same way and + * bracketed by whatever the site's `pathPrefixFor` says, written over SSH — so a wiki can keep its + * content on a NAS or a backup host without that machine having to run anything but sshd. + * + * Everything about the shape of the tree is `helpers/storageFiles.ts`, shared with `disk` and `git`. + * What this module adds is the connection: one at a time per target, re-established when it drops, + * and posix paths throughout however this server spells its own. + * + * **It reads as well as writes**, which the 2.x module did not — it declared no streaming support and + * had no way back out. Here `getAsset` is part of the contract, so a site can serve files from the + * remote host, and the two import actions can take a tree on it into the wiki. + */ +const sftpStorage: StorageModule = { + canStore(target, ref) { + return WIKI.models.storage.pathPrefixFor(target.siteId, ref.locale) !== null + }, + + async putAsset(target, ref, data) { + const relPath = assetRelPath(target, ref) + // -> Guarded rather than skipped: the model asks `canStore` before dispatching a write, so + // reaching this means somebody wrote without asking, and an asset may have no other copy + if (!relPath) { + throw new Error( + `${target.title} has no path for ${ref.locale} content, so ${ref.fileName} cannot be stored there.` + ) + } + await withClient(target, (client) => writeRemote(client, target, relPath, data)) + }, + + async getAsset(target, ref) { + const relPath = assetRelPath(target, ref) + if (!relPath) { + return null + } + return withClient(target, async (client) => { + try { + return (await client.get(remotePath(target, relPath))) as Buffer + } catch (err: any) { + if (isMissing(err)) { + // -> This target does not have the file: enabled after the upload, or removed from + // outside the wiki. Not a fault — the caller asks the next target. + return null + } + throw err + } + }) + }, + + async deleteAsset(target, ref) { + const relPath = assetRelPath(target, ref) + if (!relPath) { + return + } + await withClient(target, (client) => removeRemote(client, target, relPath)) + }, + + async moveAsset(target, ref, previous) { + await withClient(target, (client) => + moveRemote( + client, + target, + assetRelPath(target, { ...ref, ...previous }), + assetRelPath(target, ref) + ) + ) + }, + + async putPage(target, ref, page) { + const relPath = pageRelPath(target, ref) + // -> Unlike an asset, a page with no place here is not worth failing over: it is in the + // database, which is where a page always is, and this copy is the thing the site declined + if (!relPath) { + return + } + await withClient(target, (client) => + writeRemote(client, target, relPath, Buffer.from(serializePage(ref, page), 'utf8')) + ) + }, + + async deletePage(target, ref) { + // -> Exactly one name, taken from the page's own content type: in a folder where pages and + // attachments sit together, guessing at the others would delete whatever is beside it + const relPath = pageRelPath(target, ref) + if (!relPath) { + return + } + await withClient(target, (client) => removeRemote(client, target, relPath)) + }, + + async movePage(target, ref, previousPath) { + await withClient(target, (client) => + moveRemote( + client, + target, + pageRelPath(target, { ...ref, path: previousPath }), + pageRelPath(target, ref) + ) + ) + }, + + /** + * Write a copy of everything this target is configured to hold to the remote server. + * + * A plain copy: content is read from wherever it currently lives and written here, overwriting + * whatever is at each path. Nothing in the database is touched, so this is how content that + * predates the target being enabled gets onto it, and running it twice does the same work. + */ + async exportAll(target: StorageTarget): Promise { + let assets = 0 + let unreadable = 0 + let unstored = 0 + let pages = 0 + + await withClient(target, async (client) => { + for (const asset of await WIKI.models.assets.listStoredAssets(target.siteId)) { + const contentType = WIKI.models.storage.contentTypeFor( + target.siteId, + asset.kind, + asset.fileSize + ) + if (!target.contentTypes.activeTypes.includes(contentType)) { + continue + } + const relPath = assetRelPath(target, asset) + if (!relPath) { + unstored++ + continue + } + const data = await WIKI.models.storage.getAsset(asset) + if (!data) { + unreadable++ + continue + } + await writeRemote(client, target, relPath, data) + assets++ + } + + if (target.contentTypes.activeTypes.includes('pages')) { + for (const { ref, content } of await WIKI.models.pages.listForStorage(target.siteId)) { + const relPath = pageRelPath(target, ref) + if (!relPath) { + unstored++ + continue + } + await writeRemote( + client, + target, + relPath, + Buffer.from(serializePage(ref, content), 'utf8') + ) + pages++ + } + } + }) + + WIKI.logger.info( + `(STORAGE/SFTP) Exported ${assets} asset(s) and ${pages} page(s) to ${baseDir(target)} [ OK ]` + ) + const parts = [] + if (assets > 0 || pages > 0) { + parts.push(`Exported ${pages} page(s) and ${assets} asset(s).`) + } else { + parts.push('There was nothing to export.') + } + if (unreadable > 0) { + parts.push(`${unreadable} asset(s) could not be read and were skipped.`) + } + if (unstored > 0) { + const { primaryLocale } = WIKI.models.storage.pathLayoutFor(target.siteId) + parts.push( + `${unstored} item(s) are not in the ${primaryLocale} locale, which is the only one this site stores.` + ) + } + return parts.join(' ') + }, + + /** + * Take everything on the remote server that the wiki does not know about yet into the wiki. + * + * The direction that makes the remote host a store rather than a dumping ground: content arrives + * there from outside — restored from a backup, dropped in over scp — and this is what turns it back + * into pages and assets. What counts as a page is `importTree`'s to say, exactly as it is for the + * local disk; the only difference here is where the bytes are read from. + */ + async importAll(target: StorageTarget, actorId: string): Promise { + return describeImport(await runImport(target, actorId, false), false) + }, + + /** + * The same walk, with the remote server winning every collision. + * + * For a restore, or a tree edited on the server that is meant to be taken as the new truth. A page + * it replaces keeps its previous version in its history; an asset has none. + */ + async importAllOverwrite(target: StorageTarget, actorId: string): Promise { + return describeImport(await runImport(target, actorId, true), true) + } +} + +/** + * Walk the remote tree and hand it to the shared adoption code. + * + * One connection for the whole run — the walk and every read go through the same queued client, + * rather than a fresh operation per file. + */ +async function runImport( + target: StorageTarget, + actorId: string, + overwrite: boolean +): Promise { + const root = baseDir(target) + return withClient(target, async (client) => { + const files = await walkRemote(client, root, root) + return importTree({ + target, + root, + actorId, + overwrite, + files, + // -> `filePath` here is already an absolute remote path, put there by the walk above + readFile: async (filePath) => (await client.get(filePath)) as Buffer + }) + }) +} + +export default sftpStorage diff --git a/backend/package-lock.json b/backend/package-lock.json index 8e69eef94..f3dfb56e2 100644 --- a/backend/package-lock.json +++ b/backend/package-lock.json @@ -9,6 +9,10 @@ "version": "3.0.0", "license": "AGPL-3.0", "dependencies": { + "@aws-sdk/client-s3": "3.1116.0", + "@aws-sdk/s3-request-presigner": "3.1116.0", + "@azure/identity": "4.13.2", + "@azure/storage-blob": "12.33.0", "@fastify/compress": "9.2.0", "@fastify/cookie": "11.1.2", "@fastify/cors": "11.3.0", @@ -20,6 +24,7 @@ "@fastify/swagger": "9.8.1", "@fastify/swagger-ui": "6.1.1", "@fastify/websocket": "11.3.0", + "@google-cloud/storage": "8.0.1", "@gquittet/graceful-server": "6.0.10", "@iconify/utils": "3.1.4", "@simplewebauthn/server": "13.3.2", @@ -47,6 +52,8 @@ "qrcode": "1.5.4", "sanitize-html": "2.17.6", "semver": "7.8.5", + "simple-git": "3.36.0", + "ssh2-sftp-client": "12.1.1", "uuid": "14.0.1", "y-protocols": "1.0.7", "yjs": "13.6.32" @@ -59,6 +66,7 @@ "@types/qrcode": "1.5.6", "@types/sanitize-html": "2.16.1", "@types/semver": "7.8.0", + "@types/ssh2-sftp-client": "9.0.6", "@types/ws": "8.18.1", "drizzle-kit": "1.0.0-rc.4", "nodemon": "3.1.14", @@ -87,6 +95,599 @@ "url": "https://github.com/sponsors/antfu" } }, + "node_modules/@aws-sdk/checksums": { + "version": "3.1000.29", + "resolved": "https://registry.npmjs.org/@aws-sdk/checksums/-/checksums-3.1000.29.tgz", + "integrity": "sha512-Dtu0gr4dnATZAPwEYbpCsG+MpLM7OAliy2gTepEFQwl1vZ6DL3QMH2FveMa3HLvPsOdhJsPRB3KtxVhph9T75A==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/client-s3": { + "version": "3.1116.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/client-s3/-/client-s3-3.1116.0.tgz", + "integrity": "sha512-UKRl9qSVW0rZpvSOauQNpYAy8+ONBAVYnpfKVtCyOF+FZVT1tl6MunYuHvuarCrroD2/YJs+tHTALYNgAlec3Q==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/checksums": "^3.1000.29", + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/credential-provider-node": "^3.972.81", + "@aws-sdk/middleware-sdk-s3": "^3.972.75", + "@aws-sdk/signature-v4-multi-region": "^3.996.46", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/fetch-http-handler": "^5.7.2", + "@smithy/node-http-handler": "^4.11.3", + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/core": { + "version": "3.977.9", + "resolved": "https://registry.npmjs.org/@aws-sdk/core/-/core-3.977.9.tgz", + "integrity": "sha512-reqPFEQrZxDZpeGj4PFMepBeR5LGYHRqq/L0motTzgFkCRBA4rFdaVXDSLYyGHhxVz7sT2PDnPN9CluGSfgyJA==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/types": "^3.974.5", + "@aws-sdk/xml-builder": "^3.972.40", + "@aws/lambda-invoke-store": "^0.3.0", + "@smithy/core": "^3.33.3", + "@smithy/signature-v4": "^5.6.12", + "@smithy/types": "^4.17.2", + "bowser": "^2.11.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-env": { + "version": "3.972.70", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-env/-/credential-provider-env-3.972.70.tgz", + "integrity": "sha512-H404B7dJl2mCrBqahDEYsanB0xhdDp6tXnXcTUnXmmpy2Q3J0Ho0bUajZ2jr/RdwzCyS59Gi8xXIFwPLGBl6Uw==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-http": { + "version": "3.972.72", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-http/-/credential-provider-http-3.972.72.tgz", + "integrity": "sha512-X98zYOrVOeuosCX+6ktf29FC2N2GHPLia7qv6mzPzTc+RPAuHWCDS++Z6JK7eGYqb/v6uaW7bAXaOvDBfol+0w==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/fetch-http-handler": "^5.7.2", + "@smithy/node-http-handler": "^4.11.3", + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-ini": { + "version": "3.973.15", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-ini/-/credential-provider-ini-3.973.15.tgz", + "integrity": "sha512-Rykg6s5ceBuynMOGWgoowO4N+27JfnqXAnVaSunZl0hOO1XodSrxGNz6sCEbnmS0lAfQZDKyb3fbr46gSuv6Sg==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/credential-provider-env": "^3.972.70", + "@aws-sdk/credential-provider-http": "^3.972.72", + "@aws-sdk/credential-provider-login": "^3.972.77", + "@aws-sdk/credential-provider-process": "^3.972.70", + "@aws-sdk/credential-provider-sso": "^3.973.14", + "@aws-sdk/credential-provider-web-identity": "^3.972.76", + "@aws-sdk/nested-clients": "^3.997.44", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/credential-provider-imds": "^4.4.16", + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-login": { + "version": "3.972.77", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-login/-/credential-provider-login-3.972.77.tgz", + "integrity": "sha512-Jb59xfEISoN5mmbnA+HYqdtrSX3CgCtJoof+V5D8/TgUI56W63GEEd5Y58WijU3Ou6+WEgaLD1feVzaRXV5IDQ==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/nested-clients": "^3.997.44", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-node": { + "version": "3.972.81", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-node/-/credential-provider-node-3.972.81.tgz", + "integrity": "sha512-Rml+WitoFvXmv6JZ18U/xGdGDGGvB/mOin0ya0lTnTrdC0Z1lrVxTYh7iNklZBcvcRMrs4DoEf6xy1KWyrLQQw==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/credential-provider-env": "^3.972.70", + "@aws-sdk/credential-provider-http": "^3.972.72", + "@aws-sdk/credential-provider-ini": "^3.973.15", + "@aws-sdk/credential-provider-process": "^3.972.70", + "@aws-sdk/credential-provider-sso": "^3.973.14", + "@aws-sdk/credential-provider-web-identity": "^3.972.76", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/credential-provider-imds": "^4.4.16", + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-process": { + "version": "3.972.70", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-process/-/credential-provider-process-3.972.70.tgz", + "integrity": "sha512-2ry03fGRJr4sV3jI+ocjj5JqALnFD6ymM5KiNCDZMvq8bX2GSbE0vji4aM43TVCl2nXqqLRZaUxdq/KeWRAY4Q==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-sso": { + "version": "3.973.14", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-sso/-/credential-provider-sso-3.973.14.tgz", + "integrity": "sha512-jkhg/8ocAAoc0RFyLMhCw+/zZh7gystQgd4F4hznNa8P4Cc501PQmxd+jGLiMHodPJ+7Zv/3znM62gZojyasmA==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/nested-clients": "^3.997.44", + "@aws-sdk/token-providers": "3.1116.0", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/credential-provider-web-identity": { + "version": "3.972.76", + "resolved": "https://registry.npmjs.org/@aws-sdk/credential-provider-web-identity/-/credential-provider-web-identity-3.972.76.tgz", + "integrity": "sha512-d3AGyVu759PGr35mEB2s22xxlNEA5rpdxtSPJthfPFJvoQ8dt357iVPECqWfUxXp1toJAvKmbtcIYVGigaGsCA==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/nested-clients": "^3.997.44", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/middleware-sdk-s3": { + "version": "3.972.75", + "resolved": "https://registry.npmjs.org/@aws-sdk/middleware-sdk-s3/-/middleware-sdk-s3-3.972.75.tgz", + "integrity": "sha512-wMIsNumRVKaNMKhvU/s9VrdEwE8S6gSzXp4RygFG5BEMnGkkXf8cjh8zf7cKJBpUDpqTWqwbz5isEgp9rH6Lng==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/signature-v4-multi-region": "^3.996.46", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/nested-clients": { + "version": "3.997.44", + "resolved": "https://registry.npmjs.org/@aws-sdk/nested-clients/-/nested-clients-3.997.44.tgz", + "integrity": "sha512-NhEgryjlBF9w38ZXqGymQV28IhkYa1mKhlbYnqIis57AYwWGVYfUPgg/qC2rLRqOUfblxx++irvju10kVTa8Vw==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/signature-v4-multi-region": "^3.996.46", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/fetch-http-handler": "^5.7.2", + "@smithy/node-http-handler": "^4.11.3", + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/s3-request-presigner": { + "version": "3.1116.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/s3-request-presigner/-/s3-request-presigner-3.1116.0.tgz", + "integrity": "sha512-WwaaVpvrZyML5L8SNY7sUGsYlMeCHzQ/8A/ms7dz/7sfYRvPn+qf0OasUWk+zuT8qGwjsYhILD0WVtlqUU08pQ==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/signature-v4-multi-region": "^3.996.46", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/signature-v4-multi-region": { + "version": "3.996.46", + "resolved": "https://registry.npmjs.org/@aws-sdk/signature-v4-multi-region/-/signature-v4-multi-region-3.996.46.tgz", + "integrity": "sha512-L+2xZTye/2T96f3lwCws0Zw6GG2JHZW9e8FpVgGBeeExSKyeoZ6CWRpBml/7DNiK/O26jrgPM9F+Ay8VkgzUWQ==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/types": "^3.974.5", + "@smithy/signature-v4": "^5.6.12", + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/token-providers": { + "version": "3.1116.0", + "resolved": "https://registry.npmjs.org/@aws-sdk/token-providers/-/token-providers-3.1116.0.tgz", + "integrity": "sha512-ygIivKqh8aHzNkucOCXHyIBgBpLPfrSI0mCqXF+vLBsPTUKqj0VSqAY0GFPe7lQl4HntjOcQ+KSyS7oUV2C54Q==", + "license": "Apache-2.0", + "dependencies": { + "@aws-sdk/core": "^3.977.9", + "@aws-sdk/nested-clients": "^3.997.44", + "@aws-sdk/types": "^3.974.5", + "@smithy/core": "^3.33.3", + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/types": { + "version": "3.974.5", + "resolved": "https://registry.npmjs.org/@aws-sdk/types/-/types-3.974.5.tgz", + "integrity": "sha512-LkwLL2BLbC6wNNm4JaH9mbEqBMdOZCct6VAYqhdN4U1xrWM+fUJQEfbHwQgDypapOWTRtlk25akb5afM0P8CIQ==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws-sdk/xml-builder": { + "version": "3.972.40", + "resolved": "https://registry.npmjs.org/@aws-sdk/xml-builder/-/xml-builder-3.972.40.tgz", + "integrity": "sha512-wlFmCIGUlwF4zx/kncw+bmxTQh1HeSJq4mYV/V5cZUSJadDP3kXvGW8Rn21cimj/7y9ju+47oYWXi97vF7czaA==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=20.0.0" + } + }, + "node_modules/@aws/lambda-invoke-store": { + "version": "0.3.0", + "resolved": "https://registry.npmjs.org/@aws/lambda-invoke-store/-/lambda-invoke-store-0.3.0.tgz", + "integrity": "sha512-sl4Bm6yiMNYrZKkqqDFWN0UfnWhlS8ivKxrYl+6t0gCLrqr8y3B2IqZZbFRkfaVVp7C/baApyh71P+LeE1A2sQ==", + "license": "Apache-2.0", + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@azure/abort-controller": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/@azure/abort-controller/-/abort-controller-2.2.0.tgz", + "integrity": "sha512-fNAjWnA/nZ2jz31kxR/AqRaUT8ewHBw/WuBIosK0moMy1C9e5ValbDfFdIxJzVOOYaYkV/b2F1S4H/aHiqfVQg==", + "license": "MIT", + "dependencies": { + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=22.0.0" + } + }, + "node_modules/@azure/core-auth": { + "version": "1.11.0", + "resolved": "https://registry.npmjs.org/@azure/core-auth/-/core-auth-1.11.0.tgz", + "integrity": "sha512-IUZydyTUkDnYdstOW9pFOOUQlBjAepK5teihDE3x6yxsPJs/hsAaaYpeGxdxrgtOiJbBKSjKW7MDk7AEhb4LRg==", + "license": "MIT", + "dependencies": { + "@azure/abort-controller": "^2.1.2", + "@azure/core-util": "^1.13.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=22.0.0" + } + }, + "node_modules/@azure/core-client": { + "version": "1.11.0", + "resolved": "https://registry.npmjs.org/@azure/core-client/-/core-client-1.11.0.tgz", + "integrity": "sha512-JjQWO6akOck45PH/XBrxzsQGAiKrfFl4m5iggJ0ItMIz5omRufOXWpqCPpdjKN3vKDzlSUvFjaMb7Zwf0gvAdA==", + "license": "MIT", + "dependencies": { + "@azure/abort-controller": "^2.1.2", + "@azure/core-auth": "^1.10.0", + "@azure/core-rest-pipeline": "^1.22.0", + "@azure/core-tracing": "^1.3.0", + "@azure/core-util": "^1.13.0", + "@azure/logger": "^1.3.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=22.0.0" + } + }, + "node_modules/@azure/core-http-compat": { + "version": "2.5.0", + "resolved": "https://registry.npmjs.org/@azure/core-http-compat/-/core-http-compat-2.5.0.tgz", + "integrity": "sha512-BoSmXPx2er1Ai+wKlDvj29jIQespCNBwEmKyZVHO2kEFsWbGjAjwMCGzug3DJM5/QYIV3vej0S1zcU5bq9fa8w==", + "license": "MIT", + "dependencies": { + "@azure/abort-controller": "^2.1.2" + }, + "engines": { + "node": ">=22.0.0" + }, + "peerDependencies": { + "@azure/core-client": "^1.10.0", + "@azure/core-rest-pipeline": "^1.22.0" + } + }, + "node_modules/@azure/core-lro": { + "version": "2.7.2", + "resolved": "https://registry.npmjs.org/@azure/core-lro/-/core-lro-2.7.2.tgz", + "integrity": "sha512-0YIpccoX8m/k00O7mDDMdJpbr6mf1yWo2dfmxt5A8XVZVVMz2SSKaEbMCeJRvgQ0IaSlqhjT47p4hVIRRy90xw==", + "license": "MIT", + "dependencies": { + "@azure/abort-controller": "^2.0.0", + "@azure/core-util": "^1.2.0", + "@azure/logger": "^1.0.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@azure/core-paging": { + "version": "1.7.0", + "resolved": "https://registry.npmjs.org/@azure/core-paging/-/core-paging-1.7.0.tgz", + "integrity": "sha512-7GEAoIsaoBr6KELNRb8nypowCqvk8dnCHFCYg4XD4lOQGY2GqjQg5IhkRjyBFRO18CGSMq05PaNqSOE9GQro3g==", + "license": "MIT", + "dependencies": { + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=22.0.0" + } + }, + "node_modules/@azure/core-process": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/@azure/core-process/-/core-process-1.0.0.tgz", + "integrity": "sha512-/shnJ+ooO8WPxDhPEeI/2oRQuubn16gZ6CvlbpWbEswZfzwI9tI/sMAHmF3x1LuQ9yZYXfLW3TjzGMLEC5blKg==", + "license": "MIT", + "engines": { + "node": ">=22.0.0" + } + }, + "node_modules/@azure/core-rest-pipeline": { + "version": "1.25.0", + "resolved": "https://registry.npmjs.org/@azure/core-rest-pipeline/-/core-rest-pipeline-1.25.0.tgz", + "integrity": "sha512-bMs8ekJLjX8wPV+9IPBges1SLPyuDtE9g5gLDWOpxzKcoOFQnpLGkbcT1tdw3FaAmDS1gnPmMmJ6y/T5B96kIA==", + "license": "MIT", + "dependencies": { + "@azure/abort-controller": "^2.1.2", + "@azure/core-auth": "^1.10.0", + "@azure/core-tracing": "^1.3.0", + "@azure/core-util": "^1.13.0", + "@azure/logger": "^1.3.0", + "@typespec/ts-http-runtime": "^0.3.4", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=22.0.0" + } + }, + "node_modules/@azure/core-tracing": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/@azure/core-tracing/-/core-tracing-1.4.0.tgz", + "integrity": "sha512-eGwxD0AtncrxeBM4tG8R55Pc3rdX1hNW2WibJAgYpCVA6E93mvvVH+LcssoVjOBrSKWS55yEIHsk0X8ctHmfOQ==", + "license": "MIT", + "dependencies": { + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=22.0.0" + } + }, + "node_modules/@azure/core-util": { + "version": "1.14.0", + "resolved": "https://registry.npmjs.org/@azure/core-util/-/core-util-1.14.0.tgz", + "integrity": "sha512-9n2pWK61veAuN0V20t9lOuoV4CFMdyAZ1ygZzvBGk/pBBJRib/PjL9PLXa/aI2CcPpyHfqVsxxqLCYl6uZlfDw==", + "license": "MIT", + "dependencies": { + "@azure/abort-controller": "^2.1.2", + "@typespec/ts-http-runtime": "^0.3.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=22.0.0" + } + }, + "node_modules/@azure/core-xml": { + "version": "1.6.0", + "resolved": "https://registry.npmjs.org/@azure/core-xml/-/core-xml-1.6.0.tgz", + "integrity": "sha512-e7lX/dk//F6Qf7BB6PTY4+p2yuOQtyOeHGyapYHNwqSp2OnYpwQt49A/Nin2XmKBQ69pwagR4k/lQBq8lbHQkA==", + "license": "MIT", + "dependencies": { + "fast-xml-parser": "^5.5.9", + "tslib": "^2.8.1" + }, + "engines": { + "node": ">=22.0.0" + } + }, + "node_modules/@azure/identity": { + "version": "4.13.2", + "resolved": "https://registry.npmjs.org/@azure/identity/-/identity-4.13.2.tgz", + "integrity": "sha512-NXL2/pCJctLxgw8bvrwwgge743kEq8LBT+O1pmV0vyUwetzFPH9auP6jhkU/cgZCPPtWoewAe3ncaGCgPo07fA==", + "license": "MIT", + "dependencies": { + "@azure/abort-controller": "^2.0.0", + "@azure/core-auth": "^1.9.0", + "@azure/core-client": "^1.9.2", + "@azure/core-process": "^1.0.0", + "@azure/core-rest-pipeline": "^1.17.0", + "@azure/core-tracing": "^1.0.0", + "@azure/core-util": "^1.11.0", + "@azure/logger": "^1.0.0", + "@azure/msal-browser": "^5.5.0", + "@azure/msal-node": "^5.1.5", + "open": "^10.1.0", + "tslib": "^2.2.0" + }, + "engines": { + "node": ">=22.0.0" + } + }, + "node_modules/@azure/logger": { + "version": "1.4.0", + "resolved": "https://registry.npmjs.org/@azure/logger/-/logger-1.4.0.tgz", + "integrity": "sha512-rbAE25KUfjU/s3XHUdJgceoCP5dEOpMx85J04kF+QMdta73XkuG9JGHHinch+XIoKpBdqljin+KqURpJriSzLA==", + "license": "MIT", + "dependencies": { + "@typespec/ts-http-runtime": "^0.3.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=22.0.0" + } + }, + "node_modules/@azure/msal-browser": { + "version": "5.19.0", + "resolved": "https://registry.npmjs.org/@azure/msal-browser/-/msal-browser-5.19.0.tgz", + "integrity": "sha512-DHe9iRcyByGJuLPkl0K31a1JjOdRY2zX38Q07mQpSbR8zOj1EIgsWfTXhSQVyyijUxlcofVy/br7qWJbrMwVXQ==", + "license": "MIT", + "dependencies": { + "@azure/msal-common": "16.13.0" + }, + "engines": { + "node": ">=0.8.0" + } + }, + "node_modules/@azure/msal-common": { + "version": "16.13.0", + "resolved": "https://registry.npmjs.org/@azure/msal-common/-/msal-common-16.13.0.tgz", + "integrity": "sha512-rOAy0KUcyBbdwVJ+f3uPpthXatFLLZN+/KWAsTLzk1aB23Xl9DRmmXYwSvBFOZyXj4jUQQ5FKxxRkhAFW1fOow==", + "license": "MIT", + "engines": { + "node": ">=0.8.0" + } + }, + "node_modules/@azure/msal-node": { + "version": "5.6.0", + "resolved": "https://registry.npmjs.org/@azure/msal-node/-/msal-node-5.6.0.tgz", + "integrity": "sha512-uFY9NxrWHw8PwZx7gAX6PDn+9vdfS05+levc/kwkx77IkjfaldnQbbcQzzDIZ5Hq5Zdr6/z92oAIoRWKp6MnOA==", + "license": "MIT", + "dependencies": { + "@azure/msal-common": "16.13.0", + "jsonwebtoken": "^9.0.0" + }, + "engines": { + "node": ">=20" + } + }, + "node_modules/@azure/storage-blob": { + "version": "12.33.0", + "resolved": "https://registry.npmjs.org/@azure/storage-blob/-/storage-blob-12.33.0.tgz", + "integrity": "sha512-2SX8oP8PyblUcAFZSg39c8Ls+tFjavM6sBeV+qpw33mRzRhI/5hrFJmJ/x0H9xx5l6ECPvgSP8uPxqTeVbHNIA==", + "license": "MIT", + "dependencies": { + "@azure/abort-controller": "^2.1.2", + "@azure/core-auth": "^1.9.0", + "@azure/core-client": "^1.9.3", + "@azure/core-http-compat": "^2.2.0", + "@azure/core-lro": "^2.2.0", + "@azure/core-paging": "^1.6.2", + "@azure/core-rest-pipeline": "^1.19.1", + "@azure/core-tracing": "^1.2.0", + "@azure/core-util": "^1.11.0", + "@azure/core-xml": "^1.4.5", + "@azure/logger": "^1.1.4", + "@azure/storage-common": "^12.4.1", + "events": "^3.0.0", + "tslib": "^2.8.1" + }, + "engines": { + "node": ">=22.0.0" + } + }, + "node_modules/@azure/storage-common": { + "version": "12.5.0", + "resolved": "https://registry.npmjs.org/@azure/storage-common/-/storage-common-12.5.0.tgz", + "integrity": "sha512-bttzuhQiCIwrkzjPDA+AtAR7dg19L/CC6ztcqJ5LfvWpXuys9mHp0UQ0udYnoUvv9SCT9KTR5kqFvFr0e6k0lQ==", + "license": "MIT", + "dependencies": { + "@azure/abort-controller": "^2.1.2", + "@azure/core-auth": "^1.9.0", + "@azure/core-http-compat": "^2.4.0", + "@azure/core-rest-pipeline": "^1.24.0", + "@azure/core-tracing": "^1.2.0", + "@azure/core-util": "^1.11.0", + "@azure/logger": "^1.1.4", + "events": "^3.3.0", + "tslib": "^2.8.1" + }, + "engines": { + "node": ">=22.0.0" + } + }, "node_modules/@drizzle-team/brocli": { "version": "0.12.0", "resolved": "https://registry.npmjs.org/@drizzle-team/brocli/-/brocli-0.12.0.tgz", @@ -1121,18 +1722,6 @@ "ws": "^8.16.0" } }, - "node_modules/@fastify/websocket/node_modules/duplexify": { - "version": "4.1.3", - "resolved": "https://registry.npmjs.org/duplexify/-/duplexify-4.1.3.tgz", - "integrity": "sha512-M3BmBhwJRZsSx38lZyhE53Csddgzl5R7xGJNk7CVddZD6CcmwMCH8J+7AprIrQKH7TonKxaCjcv27Qmf+sQ+oA==", - "license": "MIT", - "dependencies": { - "end-of-stream": "^1.4.1", - "inherits": "^2.0.3", - "readable-stream": "^3.1.1", - "stream-shift": "^1.0.2" - } - }, "node_modules/@fastify/websocket/node_modules/fastify-plugin": { "version": "6.0.0", "resolved": "https://registry.npmjs.org/fastify-plugin/-/fastify-plugin-6.0.0.tgz", @@ -1149,18 +1738,86 @@ ], "license": "MIT" }, - "node_modules/@fastify/websocket/node_modules/readable-stream": { - "version": "3.6.2", - "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz", - "integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==", + "node_modules/@google-cloud/paginator": { + "version": "7.0.1", + "resolved": "https://registry.npmjs.org/@google-cloud/paginator/-/paginator-7.0.1.tgz", + "integrity": "sha512-k32cWlHAF8yTgg8rciLI8mPMI6UzuJdKp53YRxISRwMFxUl2FYplvs+Mr2UHxKn0W7rXsqZnUZy73AOJFDP8iA==", + "license": "Apache-2.0", + "dependencies": { + "extend": "^3.0.2" + }, + "engines": { + "node": ">=22" + } + }, + "node_modules/@google-cloud/projectify": { + "version": "6.0.1", + "resolved": "https://registry.npmjs.org/@google-cloud/projectify/-/projectify-6.0.1.tgz", + "integrity": "sha512-zA3o4l87UJ56odT/e9PcT4n2bqVnrp6dg9vI75DoGGbl0/YJemX8Fta7454htECt07eALxxc4iaAbrX3T+QulA==", + "license": "Apache-2.0", + "engines": { + "node": ">=22" + } + }, + "node_modules/@google-cloud/promisify": { + "version": "6.0.1", + "resolved": "https://registry.npmjs.org/@google-cloud/promisify/-/promisify-6.0.1.tgz", + "integrity": "sha512-lpN2AtoQ/iimp7jjm5zJFWbpW7OJc5qWmQdt59CI4ENeoIRIx+yf2ShSR2kanQfjFOshA74eSbimXXuRaSCUVg==", + "license": "Apache-2.0", + "engines": { + "node": ">=22" + } + }, + "node_modules/@google-cloud/storage": { + "version": "8.0.1", + "resolved": "https://registry.npmjs.org/@google-cloud/storage/-/storage-8.0.1.tgz", + "integrity": "sha512-qgNayyweyLssLW2NNL4WYUIbwxRCvCVGg52aW8r9Lkunkoj52U6WQa/TSXSsJZ1BrbxQp3+9kwwpRl+scHQ3KQ==", + "license": "Apache-2.0", + "dependencies": { + "@google-cloud/paginator": "^7.0.1", + "@google-cloud/projectify": "^6.0.1", + "@google-cloud/promisify": "^6.0.1", + "abort-controller": "^3.0.0", + "async-retry": "^1.3.3", + "duplexify": "^4.1.3", + "fast-xml-parser": "^5.3.4", + "gaxios": "^6.0.2", + "google-auth-library": "^9.6.3", + "html-entities": "^2.5.2", + "mime": "^3.0.0", + "p-limit": "^3.0.1", + "retry-request": "^9.0.1", + "teeny-request": "^11.0.1" + }, + "engines": { + "node": ">=22" + } + }, + "node_modules/@google-cloud/storage/node_modules/mime": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/mime/-/mime-3.0.0.tgz", + "integrity": "sha512-jSCU7/VB1loIWBZe14aEYHU/+1UMEHoaO7qxCOVJOw9GgH72VAWppxNcjU+x9a2k3GSIBXNKxXQFqRvvZ7vr3A==", + "license": "MIT", + "bin": { + "mime": "cli.js" + }, + "engines": { + "node": ">=10.0.0" + } + }, + "node_modules/@google-cloud/storage/node_modules/p-limit": { + "version": "3.1.0", + "resolved": "https://registry.npmjs.org/p-limit/-/p-limit-3.1.0.tgz", + "integrity": "sha512-TYOanM3wGwNGsZN2cVTYPArw454xnXj5qmWF1bEoAc4+cU/ol7GVh7odevjp1FNHduHc3KZMcFduxU5Xc6uJRQ==", "license": "MIT", "dependencies": { - "inherits": "^2.0.3", - "string_decoder": "^1.1.1", - "util-deprecate": "^1.0.1" + "yocto-queue": "^0.1.0" }, "engines": { - "node": ">= 6" + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" } }, "node_modules/@gquittet/graceful-server": { @@ -1760,6 +2417,21 @@ "node": ">=12" } }, + "node_modules/@kwsites/file-exists": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@kwsites/file-exists/-/file-exists-1.1.1.tgz", + "integrity": "sha512-m9/5YGR18lIwxSFDwfE3oA7bWuq9kdau6ugN4H2rJeyhFQZcG9AgSHkQtSD15a8WvTgfz9aikZMrKPHvbpqFiw==", + "license": "MIT", + "dependencies": { + "debug": "^4.1.1" + } + }, + "node_modules/@kwsites/promise-deferred": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@kwsites/promise-deferred/-/promise-deferred-1.1.1.tgz", + "integrity": "sha512-GaHYm+c0O9MjZRu0ongGBRbinu8gVAMd2UZjji6jVmqKtZluZnptXGWhz1E8j8D2HJ3f/yMxKAUC0b+57wncIw==", + "license": "MIT" + }, "node_modules/@levischuck/tiny-cbor": { "version": "0.2.11", "resolved": "https://registry.npmjs.org/@levischuck/tiny-cbor/-/tiny-cbor-0.2.11.tgz", @@ -1775,6 +2447,18 @@ "node": ">=8" } }, + "node_modules/@nodable/entities": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/@nodable/entities/-/entities-3.0.0.tgz", + "integrity": "sha512-8L9xFeTYKhm49xfIypoe2W5wV1m/3Z58kT+7kR9A8OyFxcPduI4VmxaUMQyKYrRjUoLLSXv6EKKID5Tvj9cUVw==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/nodable" + } + ], + "license": "MIT" + }, "node_modules/@oxfmt/binding-android-arm-eabi": { "version": "0.62.0", "resolved": "https://registry.npmjs.org/@oxfmt/binding-android-arm-eabi/-/binding-android-arm-eabi-0.62.0.tgz", @@ -2643,6 +3327,21 @@ "integrity": "sha512-k2ENnmBugE/rzQfEcdWHcCY+/FM3VLzH9cYEsbdsoqrvzAKRhUZeRNhAZvB8OitQJ1TBed3yqWtdjzS6wJKBwg==", "license": "MIT" }, + "node_modules/@simple-git/args-pathspec": { + "version": "1.0.3", + "resolved": "https://registry.npmjs.org/@simple-git/args-pathspec/-/args-pathspec-1.0.3.tgz", + "integrity": "sha512-ngJMaHlsWDTfjyq9F3VIQ8b7NXbBLq5j9i5bJ6XLYtD6qlDXT7fdKY2KscWWUF8t18xx052Y/PUO1K1TRc9yKA==", + "license": "MIT" + }, + "node_modules/@simple-git/argv-parser": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@simple-git/argv-parser/-/argv-parser-1.1.1.tgz", + "integrity": "sha512-Q9lBcfQ+VQCpQqGJFHe5yooOS5hGdLFFbJ5R+R5aDsnkPCahtn1hSkMcORX65J2Z5lxSkD0lQorMsncuBQxYUw==", + "license": "MIT", + "dependencies": { + "@simple-git/args-pathspec": "^1.0.3" + } + }, "node_modules/@simplewebauthn/server": { "version": "13.3.2", "resolved": "https://registry.npmjs.org/@simplewebauthn/server/-/server-13.3.2.tgz", @@ -2662,6 +3361,87 @@ "node": ">=20.0.0" } }, + "node_modules/@smithy/core": { + "version": "3.33.3", + "resolved": "https://registry.npmjs.org/@smithy/core/-/core-3.33.3.tgz", + "integrity": "sha512-CsOeKq/9kA3y6VJHt+/+VTCtBaxJ4OTFpgrjIUhPpDIKxBci1k2bJaQASF2h/ELWrulGp+t97DZ0mevfAD8idg==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/credential-provider-imds": { + "version": "4.5.2", + "resolved": "https://registry.npmjs.org/@smithy/credential-provider-imds/-/credential-provider-imds-4.5.2.tgz", + "integrity": "sha512-A9uSdn72ozbRUSit0eib0TW7nXuNPlaeM0zcGkJ+nE6tFcSDbnmtwoxbTCFBukVQcszDAyvsd7+rTduPTXpygg==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/core": "^3.33.2", + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/fetch-http-handler": { + "version": "5.7.2", + "resolved": "https://registry.npmjs.org/@smithy/fetch-http-handler/-/fetch-http-handler-5.7.2.tgz", + "integrity": "sha512-nZyWTmSpJEXl6VtWVMBJve/7x12DZu6sIX1z1a+ZMaHlQQRs9Zpu6NbTe/gmxYXVRpkjxyDYpZ5gx2IM6f/Wkw==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/core": "^3.33.2", + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/node-http-handler": { + "version": "4.11.3", + "resolved": "https://registry.npmjs.org/@smithy/node-http-handler/-/node-http-handler-4.11.3.tgz", + "integrity": "sha512-2jY1tSpERfPfWqyBV2pH+iGFaghVsIJszJNsT7hxtQYhVJpWDyc0LqOWI+nXOxOAHaEfZ4PXXtp1wW1TGpHhkA==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/core": "^3.33.3", + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/signature-v4": { + "version": "5.7.3", + "resolved": "https://registry.npmjs.org/@smithy/signature-v4/-/signature-v4-5.7.3.tgz", + "integrity": "sha512-7ImGm+FkHRLcBaRttIAMZ6bzJZWb2cJGoYjq46F2UjycujWzrL9GEN9h4w7eQyXJYnltrUhxbbieBAIRrdqpow==", + "license": "Apache-2.0", + "dependencies": { + "@smithy/core": "^3.33.3", + "@smithy/types": "^4.17.2", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, + "node_modules/@smithy/types": { + "version": "4.17.2", + "resolved": "https://registry.npmjs.org/@smithy/types/-/types-4.17.2.tgz", + "integrity": "sha512-FOKpVZob9MPTn2znRzGrnsMHv7BOsKVw3XiP/cOyYLDVZ9qKp4nifIiSCuUU/fIj5Vu0UOAxCFr+qRAtG0NUkA==", + "license": "Apache-2.0", + "dependencies": { + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=18.0.0" + } + }, "node_modules/@types/fs-extra": { "version": "11.0.4", "resolved": "https://registry.npmjs.org/@types/fs-extra/-/fs-extra-11.0.4.tgz", @@ -2739,6 +3519,43 @@ "dev": true, "license": "MIT" }, + "node_modules/@types/ssh2": { + "version": "1.15.5", + "resolved": "https://registry.npmjs.org/@types/ssh2/-/ssh2-1.15.5.tgz", + "integrity": "sha512-N1ASjp/nXH3ovBHddRJpli4ozpk6UdDYIX4RJWFa9L1YKnzdhTlVmiGHm4DZnj/jLbqZpes4aeR30EFGQtvhQQ==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/node": "^18.11.18" + } + }, + "node_modules/@types/ssh2-sftp-client": { + "version": "9.0.6", + "resolved": "https://registry.npmjs.org/@types/ssh2-sftp-client/-/ssh2-sftp-client-9.0.6.tgz", + "integrity": "sha512-4+KvXO/V77y9VjI2op2T8+RCGI/GXQAwR0q5Qkj/EJ5YSeyKszqZP6F8i3H3txYoBqjc7sgorqyvBP3+w1EHyg==", + "dev": true, + "license": "MIT", + "dependencies": { + "@types/ssh2": "^1.0.0" + } + }, + "node_modules/@types/ssh2/node_modules/@types/node": { + "version": "18.19.130", + "resolved": "https://registry.npmjs.org/@types/node/-/node-18.19.130.tgz", + "integrity": "sha512-GRaXQx6jGfL8sKfaIDD6OupbIHBr9jv7Jnaml9tB7l4v068PAOXqfcujMMo5PhbIs6ggR1XODELqahT2R8v0fg==", + "dev": true, + "license": "MIT", + "dependencies": { + "undici-types": "~5.26.4" + } + }, + "node_modules/@types/ssh2/node_modules/undici-types": { + "version": "5.26.5", + "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-5.26.5.tgz", + "integrity": "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA==", + "dev": true, + "license": "MIT" + }, "node_modules/@types/ws": { "version": "8.18.1", "resolved": "https://registry.npmjs.org/@types/ws/-/ws-8.18.1.tgz", @@ -3086,7 +3903,21 @@ "win32" ], "engines": { - "node": ">=16.20.0" + "node": ">=16.20.0" + } + }, + "node_modules/@typespec/ts-http-runtime": { + "version": "0.3.8", + "resolved": "https://registry.npmjs.org/@typespec/ts-http-runtime/-/ts-http-runtime-0.3.8.tgz", + "integrity": "sha512-bLMpVcWZNzq6lYOybwFwOAR1IXKcHnhUNqYeHjl1bET/qE3jFPFH+p8Wrh3rU4xwdnifPxmKNESBYnvnmc75aA==", + "license": "MIT", + "dependencies": { + "http-proxy-agent": "^7.0.0", + "https-proxy-agent": "^7.0.0", + "tslib": "^2.6.2" + }, + "engines": { + "node": ">=22.0.0" } }, "node_modules/abort-controller": { @@ -3107,6 +3938,15 @@ "integrity": "sha512-2BjRTZxTPvheOvGbBslFSYOUkr+SjPtOnrLP33f+VIWLzezQpZcqVg7ja3L4dBXmzzgwT+a029jRx5PCi3JuiA==", "license": "MIT" }, + "node_modules/agent-base": { + "version": "7.1.4", + "resolved": "https://registry.npmjs.org/agent-base/-/agent-base-7.1.4.tgz", + "integrity": "sha512-MnA+YT8fwfJPgBx3m60MNqakm30XOkyIoH1y6huTQvC0PwZG7ki8NacLBcrPbNoo8vEZy7Jpuk7+jMO+CUovTQ==", + "license": "MIT", + "engines": { + "node": ">= 14" + } + }, "node_modules/ajv": { "version": "8.20.0", "resolved": "https://registry.npmjs.org/ajv/-/ajv-8.20.0.tgz", @@ -3191,12 +4031,33 @@ "url": "https://github.com/sponsors/jonschlinkert" } }, + "node_modules/anynum": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/anynum/-/anynum-1.0.1.tgz", + "integrity": "sha512-N6//FLET/tXYNM/F6ABca1oH6fWB+KlTt909Le28WMDBk8oaT4vY17DCrwg2MvmuqUKt3Ni4N5dGJ/EoBgcO6A==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/NaturalIntelligence" + } + ], + "license": "MIT" + }, "node_modules/argparse": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/argparse/-/argparse-2.0.1.tgz", "integrity": "sha512-8+9WqebbFzpX9OR+Wa6O29asIogeRMzcGtAINdpMHHyAg10f05aSFVBbcEqGf/PXw1EjAZ+q2/bEBg3DvurK3Q==", "license": "Python-2.0" }, + "node_modules/asn1": { + "version": "0.2.6", + "resolved": "https://registry.npmjs.org/asn1/-/asn1-0.2.6.tgz", + "integrity": "sha512-ix/FxPn0MDjeyJ7i/yoHGFt/EX6LyNbxSEhPPXODPL+KB0VPk86UYfL0lMdy+KCnv+fmvIzySwaK5COwqVbWTQ==", + "license": "MIT", + "dependencies": { + "safer-buffer": "~2.1.0" + } + }, "node_modules/asn1js": { "version": "3.0.10", "resolved": "https://registry.npmjs.org/asn1js/-/asn1js-3.0.10.tgz", @@ -3211,6 +4072,15 @@ "node": ">=12.0.0" } }, + "node_modules/async-retry": { + "version": "1.3.3", + "resolved": "https://registry.npmjs.org/async-retry/-/async-retry-1.3.3.tgz", + "integrity": "sha512-wfr/jstw9xNi/0teMHrRW7dsz3Lt5ARhYNZ2ewpadnhaIp5mbALhOAP+EAdsC7t4Z6wqsDVv9+W6gm1Dk9mEyw==", + "license": "MIT", + "dependencies": { + "retry": "0.13.1" + } + }, "node_modules/atomic-sleep": { "version": "1.0.0", "resolved": "https://registry.npmjs.org/atomic-sleep/-/atomic-sleep-1.0.0.tgz", @@ -3259,6 +4129,15 @@ ], "license": "MIT" }, + "node_modules/bcrypt-pbkdf": { + "version": "1.0.2", + "resolved": "https://registry.npmjs.org/bcrypt-pbkdf/-/bcrypt-pbkdf-1.0.2.tgz", + "integrity": "sha512-qeFIXtP4MSoi6NLqO12WfqARWWuCKi2Rn/9hJLEmtB5yTNr9DqFWkJRCf2qShWzPeAMRnOgCrq0sg/KLv5ES9w==", + "license": "BSD-3-Clause", + "dependencies": { + "tweetnacl": "^0.14.3" + } + }, "node_modules/bcryptjs": { "version": "3.0.3", "resolved": "https://registry.npmjs.org/bcryptjs/-/bcryptjs-3.0.3.tgz", @@ -3268,6 +4147,15 @@ "bcrypt": "bin/bcrypt" } }, + "node_modules/bignumber.js": { + "version": "9.3.1", + "resolved": "https://registry.npmjs.org/bignumber.js/-/bignumber.js-9.3.1.tgz", + "integrity": "sha512-Ko0uX15oIUS7wJ3Rb30Fs6SkVbLmPBAKdlm7q9+ak9bbIeFf0MwuBsQV6z7+X768/cHsfg+WlysDWJcmthjsjQ==", + "license": "MIT", + "engines": { + "node": "*" + } + }, "node_modules/binary-extensions": { "version": "2.3.0", "resolved": "https://registry.npmjs.org/binary-extensions/-/binary-extensions-2.3.0.tgz", @@ -3287,6 +4175,12 @@ "integrity": "sha512-JZOSA7Mo9sNGB8+UjSgzdLtokWAky1zbztM3WRLCbZ70/3cTANmQmOdR7y2g+J0e2WXywy1yS468tY+IruqEww==", "license": "ISC" }, + "node_modules/bowser": { + "version": "2.14.1", + "resolved": "https://registry.npmjs.org/bowser/-/bowser-2.14.1.tgz", + "integrity": "sha512-tzPjzCxygAKWFOJP011oxFHs57HzIhOEracIgAePE4pqB3LikALKnSzUyU4MGs9/iCEUuHlAJTjTc5M+u7YEGg==", + "license": "MIT" + }, "node_modules/brace-expansion": { "version": "5.0.6", "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-5.0.6.tgz", @@ -3336,6 +4230,42 @@ "ieee754": "^1.2.1" } }, + "node_modules/buffer-equal-constant-time": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/buffer-equal-constant-time/-/buffer-equal-constant-time-1.0.1.tgz", + "integrity": "sha512-zRpUiDwd/xk6ADqPMATG8vc9VPrkck7T07OIx0gnjmJAnHnTVXNQG3vfvWNuiZIkwu9KrKdA1iJKfsfTVxE6NA==", + "license": "BSD-3-Clause" + }, + "node_modules/buffer-from": { + "version": "1.1.2", + "resolved": "https://registry.npmjs.org/buffer-from/-/buffer-from-1.1.2.tgz", + "integrity": "sha512-E+XQCRwSbaaiChtv6k6Dwgc+bx+Bs6vuKJHHl5kox/BaKbhiXzqQOwK4cO22yElGp2OCmjwVhT3HmxgyPGnJfQ==", + "license": "MIT" + }, + "node_modules/buildcheck": { + "version": "0.0.7", + "resolved": "https://registry.npmjs.org/buildcheck/-/buildcheck-0.0.7.tgz", + "integrity": "sha512-lHblz4ahamxpTmnsk+MNTRWsjYKv965MwOrSJyeD588rR3Jcu7swE+0wN5F+PbL5cjgu/9ObkhfzEPuofEMwLA==", + "optional": true, + "engines": { + "node": ">=10.0.0" + } + }, + "node_modules/bundle-name": { + "version": "4.1.0", + "resolved": "https://registry.npmjs.org/bundle-name/-/bundle-name-4.1.0.tgz", + "integrity": "sha512-tjwM5exMg6BGRI+kNmTntNsvdZS1X8BFYS6tnJ2hdH0kVxM6/eVZ2xy+FqStSWvYmtfFMDLIxurorHwDKfDz5Q==", + "license": "MIT", + "dependencies": { + "run-applescript": "^7.0.0" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, "node_modules/camelcase": { "version": "5.3.1", "resolved": "https://registry.npmjs.org/camelcase/-/camelcase-5.3.1.tgz", @@ -3475,6 +4405,35 @@ "integrity": "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==", "license": "MIT" }, + "node_modules/concat-stream": { + "version": "2.0.0", + "resolved": "https://registry.npmjs.org/concat-stream/-/concat-stream-2.0.0.tgz", + "integrity": "sha512-MWufYdFw53ccGjCA+Ol7XJYpAlW6/prSMzuPOTRnJGcGzuhLn4Scrz7qf6o8bROZ514ltazcIFJZevcfbo0x7A==", + "engines": [ + "node >= 6.0" + ], + "license": "MIT", + "dependencies": { + "buffer-from": "^1.0.0", + "inherits": "^2.0.3", + "readable-stream": "^3.0.2", + "typedarray": "^0.0.6" + } + }, + "node_modules/concat-stream/node_modules/readable-stream": { + "version": "3.6.2", + "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz", + "integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==", + "license": "MIT", + "dependencies": { + "inherits": "^2.0.3", + "string_decoder": "^1.1.1", + "util-deprecate": "^1.0.1" + }, + "engines": { + "node": ">= 6" + } + }, "node_modules/content-disposition": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/content-disposition/-/content-disposition-2.0.1.tgz", @@ -3510,6 +4469,20 @@ "url": "https://opencollective.com/express" } }, + "node_modules/cpu-features": { + "version": "0.0.10", + "resolved": "https://registry.npmjs.org/cpu-features/-/cpu-features-0.0.10.tgz", + "integrity": "sha512-9IkYqtX3YHPCzoVg1Py+o9057a3i0fp7S530UWokCSaFVTc7CwXPRiOjRjBQQ18ZCNafx78YfnG+HALxtVmOGA==", + "hasInstallScript": true, + "optional": true, + "dependencies": { + "buildcheck": "~0.0.6", + "nan": "^2.19.0" + }, + "engines": { + "node": ">=10.0.0" + } + }, "node_modules/cron-parser": { "version": "5.8.1", "resolved": "https://registry.npmjs.org/cron-parser/-/cron-parser-5.8.1.tgz", @@ -3550,6 +4523,15 @@ "url": "https://github.com/sponsors/fb55" } }, + "node_modules/data-uri-to-buffer": { + "version": "4.0.1", + "resolved": "https://registry.npmjs.org/data-uri-to-buffer/-/data-uri-to-buffer-4.0.1.tgz", + "integrity": "sha512-0R9ikRb668HB7QDxT1vkpuUBtqc53YyAwMwGeUFKRojY/NWKvdZ+9UYtRfGmhqNbRkTSVpMbmyhXipFFv2cb/A==", + "license": "MIT", + "engines": { + "node": ">= 12" + } + }, "node_modules/dayjs": { "version": "1.11.21", "resolved": "https://registry.npmjs.org/dayjs/-/dayjs-1.11.21.tgz", @@ -3591,6 +4573,46 @@ "node": ">=0.10.0" } }, + "node_modules/default-browser": { + "version": "5.5.1", + "resolved": "https://registry.npmjs.org/default-browser/-/default-browser-5.5.1.tgz", + "integrity": "sha512-m1pAzaJgZ/gssEqlOhJkPJp8Xly7QyW6xcrkUa2KKcDeDSEMP7X8xipU3snUcfisTQx0w1AGae+9UtJSfVnXGw==", + "license": "MIT", + "dependencies": { + "bundle-name": "^4.1.0", + "default-browser-id": "^5.0.0" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/default-browser-id": { + "version": "5.0.1", + "resolved": "https://registry.npmjs.org/default-browser-id/-/default-browser-id-5.0.1.tgz", + "integrity": "sha512-x1VCxdX4t+8wVfd1so/9w+vQ4vx7lKd2Qp5tDRutErwmR85OgmfX7RlLRMWafRMY7hbEiXIbudNrjOAPa/hL8Q==", + "license": "MIT", + "engines": { + "node": ">=18" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/define-lazy-prop": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/define-lazy-prop/-/define-lazy-prop-3.0.0.tgz", + "integrity": "sha512-N+MeXYoqr3pOgn8xfyRPREN7gHakLYjhsHhWGT3fWAiL4IkAt0iDw14QiiEm2bE30c5XX5q0FtAA3CK5f9/BUg==", + "license": "MIT", + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, "node_modules/depd": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/depd/-/depd-2.0.0.tgz", @@ -3904,6 +4926,41 @@ } } }, + "node_modules/duplexify": { + "version": "4.1.3", + "resolved": "https://registry.npmjs.org/duplexify/-/duplexify-4.1.3.tgz", + "integrity": "sha512-M3BmBhwJRZsSx38lZyhE53Csddgzl5R7xGJNk7CVddZD6CcmwMCH8J+7AprIrQKH7TonKxaCjcv27Qmf+sQ+oA==", + "license": "MIT", + "dependencies": { + "end-of-stream": "^1.4.1", + "inherits": "^2.0.3", + "readable-stream": "^3.1.1", + "stream-shift": "^1.0.2" + } + }, + "node_modules/duplexify/node_modules/readable-stream": { + "version": "3.6.2", + "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz", + "integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==", + "license": "MIT", + "dependencies": { + "inherits": "^2.0.3", + "string_decoder": "^1.1.1", + "util-deprecate": "^1.0.1" + }, + "engines": { + "node": ">= 6" + } + }, + "node_modules/ecdsa-sig-formatter": { + "version": "1.0.11", + "resolved": "https://registry.npmjs.org/ecdsa-sig-formatter/-/ecdsa-sig-formatter-1.0.11.tgz", + "integrity": "sha512-nagl3RYrbNv6kQkeJIpt6NJZy8twLB/2vtz6yN9Z4vRKHN4/QZJIEbqohALSgwKdnksuY3k5Addp5lg8sVoVcQ==", + "license": "Apache-2.0", + "dependencies": { + "safe-buffer": "^5.0.1" + } + }, "node_modules/emittery": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/emittery/-/emittery-2.0.0.tgz", @@ -4057,6 +5114,12 @@ "node": ">=0.8.x" } }, + "node_modules/extend": { + "version": "3.0.2", + "resolved": "https://registry.npmjs.org/extend/-/extend-3.0.2.tgz", + "integrity": "sha512-fjquC59cD7CyW6urNXK0FBufkZcoiGG80wTuPujX590cB5Ttln20E2UB4S/WARVqhXffZl2LNgS+gQdPIIim/g==", + "license": "MIT" + }, "node_modules/fast-decode-uri-component": { "version": "1.0.1", "resolved": "https://registry.npmjs.org/fast-decode-uri-component/-/fast-decode-uri-component-1.0.1.tgz", @@ -4118,6 +5181,45 @@ ], "license": "BSD-3-Clause" }, + "node_modules/fast-xml-builder": { + "version": "1.3.1", + "resolved": "https://registry.npmjs.org/fast-xml-builder/-/fast-xml-builder-1.3.1.tgz", + "integrity": "sha512-pIM/1n3ntFXKYrUZwW7QCK0gAW7XY+wzj1YMIV3tLDvPj/V+zTGJK5e3/4WJfwj0qWw2ElNXiTixda/R+3YSug==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/NaturalIntelligence" + } + ], + "license": "MIT", + "dependencies": { + "path-expression-matcher": "^1.6.2", + "xml-naming": "^0.3.0" + } + }, + "node_modules/fast-xml-parser": { + "version": "5.11.0", + "resolved": "https://registry.npmjs.org/fast-xml-parser/-/fast-xml-parser-5.11.0.tgz", + "integrity": "sha512-9IGxMqvqLOnqP+Egi1nqDHKv5k8aZ7r9n558enxcucmyVGEBNPAU+MOg/8jPIS7rO7sSq4gFm1/nHtiaubMruw==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/NaturalIntelligence" + } + ], + "license": "MIT", + "dependencies": { + "@nodable/entities": "^3.0.0", + "fast-xml-builder": "^1.2.0", + "is-unsafe": "^2.0.0", + "path-expression-matcher": "^1.6.2", + "strnum": "^2.4.2", + "xml-naming": "^0.3.0" + }, + "bin": { + "fxparser": "src/cli/cli.js" + } + }, "node_modules/fastify": { "version": "5.11.3", "resolved": "https://registry.npmjs.org/fastify/-/fastify-5.11.3.tgz", @@ -4228,6 +5330,29 @@ "reusify": "^1.0.4" } }, + "node_modules/fetch-blob": { + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/fetch-blob/-/fetch-blob-3.2.0.tgz", + "integrity": "sha512-7yAQpD2UMJzLi1Dqv7qFYnPbaPx7ZfFK6PiIxQ4PfkGPyNyl2Ugx+a/umUonmKqjhM4DnfbMvdX6otXq83soQQ==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/jimmywarting" + }, + { + "type": "paypal", + "url": "https://paypal.me/jimmywarting" + } + ], + "license": "MIT", + "dependencies": { + "node-domexception": "^1.0.0", + "web-streams-polyfill": "^3.0.3" + }, + "engines": { + "node": "^12.20 || >= 14.13" + } + }, "node_modules/filesize": { "version": "11.0.22", "resolved": "https://registry.npmjs.org/filesize/-/filesize-11.0.22.tgz", @@ -4277,6 +5402,18 @@ "node": ">=8" } }, + "node_modules/formdata-polyfill": { + "version": "4.0.10", + "resolved": "https://registry.npmjs.org/formdata-polyfill/-/formdata-polyfill-4.0.10.tgz", + "integrity": "sha512-buewHzMvYL29jdeQTVILecSaZKnt/RJWjoZCF5OW60Z67/GmSLBkOFM7qh1PI3zFNtJbaZL5eQu1vLfazOwj4g==", + "license": "MIT", + "dependencies": { + "fetch-blob": "^3.1.2" + }, + "engines": { + "node": ">=12.20.0" + } + }, "node_modules/forwarded": { "version": "0.2.0", "resolved": "https://registry.npmjs.org/forwarded/-/forwarded-0.2.0.tgz", @@ -4314,6 +5451,50 @@ "node": "^8.16.0 || ^10.6.0 || >=11.0.0" } }, + "node_modules/gaxios": { + "version": "6.7.1", + "resolved": "https://registry.npmjs.org/gaxios/-/gaxios-6.7.1.tgz", + "integrity": "sha512-LDODD4TMYx7XXdpwxAVRAIAuB0bzv0s+ywFonY46k126qzQHT9ygyoa9tncmOiQmmDrik65UYsEkv3lbfqQ3yQ==", + "license": "Apache-2.0", + "dependencies": { + "extend": "^3.0.2", + "https-proxy-agent": "^7.0.1", + "is-stream": "^2.0.0", + "node-fetch": "^2.6.9", + "uuid": "^9.0.1" + }, + "engines": { + "node": ">=14" + } + }, + "node_modules/gaxios/node_modules/uuid": { + "version": "9.0.1", + "resolved": "https://registry.npmjs.org/uuid/-/uuid-9.0.1.tgz", + "integrity": "sha512-b+1eJOlsR9K8HJpow9Ok3fiWOWSIcIzXodvv0rQjVoOVNpWMpxf1wZNpt4y9h10odCNrqnYp1OBzRktckBe3sA==", + "deprecated": "uuid@10 and below is no longer supported. For ESM codebases, update to uuid@latest. For CommonJS codebases, use uuid@11 (but be aware this version will likely be deprecated in 2028).", + "funding": [ + "https://github.com/sponsors/broofa", + "https://github.com/sponsors/ctavan" + ], + "license": "MIT", + "bin": { + "uuid": "dist/bin/uuid" + } + }, + "node_modules/gcp-metadata": { + "version": "6.1.1", + "resolved": "https://registry.npmjs.org/gcp-metadata/-/gcp-metadata-6.1.1.tgz", + "integrity": "sha512-a4tiq7E0/5fTjxPAaH4jpjkSv/uCaU2p5KC6HVGrvl0cDjA8iBZv4vv1gyzlmK0ZUKqwpOyQMKzZQe3lTit77A==", + "license": "Apache-2.0", + "dependencies": { + "gaxios": "^6.1.1", + "google-logging-utils": "^0.0.2", + "json-bigint": "^1.0.0" + }, + "engines": { + "node": ">=14" + } + }, "node_modules/get-caller-file": { "version": "2.0.5", "resolved": "https://registry.npmjs.org/get-caller-file/-/get-caller-file-2.0.5.tgz", @@ -4353,12 +5534,51 @@ "url": "https://github.com/sponsors/isaacs" } }, + "node_modules/google-auth-library": { + "version": "9.15.1", + "resolved": "https://registry.npmjs.org/google-auth-library/-/google-auth-library-9.15.1.tgz", + "integrity": "sha512-Jb6Z0+nvECVz+2lzSMt9u98UsoakXxA2HGHMCxh+so3n90XgYWkq5dur19JAJV7ONiJY22yBTyJB1TSkvPq9Ng==", + "license": "Apache-2.0", + "dependencies": { + "base64-js": "^1.3.0", + "ecdsa-sig-formatter": "^1.0.11", + "gaxios": "^6.1.1", + "gcp-metadata": "^6.1.0", + "gtoken": "^7.0.0", + "jws": "^4.0.0" + }, + "engines": { + "node": ">=14" + } + }, + "node_modules/google-logging-utils": { + "version": "0.0.2", + "resolved": "https://registry.npmjs.org/google-logging-utils/-/google-logging-utils-0.0.2.tgz", + "integrity": "sha512-NEgUnEcBiP5HrPzufUkBzJOD/Sxsco3rLNo1F1TNf7ieU8ryUzBhqba8r756CjLX7rn3fHl6iLEwPYuqpoKgQQ==", + "license": "Apache-2.0", + "engines": { + "node": ">=14" + } + }, "node_modules/graceful-fs": { "version": "4.2.11", "resolved": "https://registry.npmjs.org/graceful-fs/-/graceful-fs-4.2.11.tgz", "integrity": "sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ==", "license": "ISC" }, + "node_modules/gtoken": { + "version": "7.1.0", + "resolved": "https://registry.npmjs.org/gtoken/-/gtoken-7.1.0.tgz", + "integrity": "sha512-pCcEwRi+TKpMlxAQObHDQ56KawURgyAf6jtIY046fJ5tIv3zDe/LEIubckAO8fj6JnAxLdmWkUfNyulQ2iKdEw==", + "license": "MIT", + "dependencies": { + "gaxios": "^6.0.0", + "jws": "^4.0.0" + }, + "engines": { + "node": ">=14.0.0" + } + }, "node_modules/helmet": { "version": "8.1.0", "resolved": "https://registry.npmjs.org/helmet/-/helmet-8.1.0.tgz", @@ -4368,6 +5588,22 @@ "node": ">=18.0.0" } }, + "node_modules/html-entities": { + "version": "2.6.0", + "resolved": "https://registry.npmjs.org/html-entities/-/html-entities-2.6.0.tgz", + "integrity": "sha512-kig+rMn/QOVRvr7c86gQ8lWXq+Hkv6CbAH1hLu+RG338StTpE8Z0b44SDVaqVu7HGKf27frdmUYEs9hTUX/cLQ==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/mdevils" + }, + { + "type": "patreon", + "url": "https://patreon.com/mdevils" + } + ], + "license": "MIT" + }, "node_modules/htmlparser2": { "version": "10.1.0", "resolved": "https://registry.npmjs.org/htmlparser2/-/htmlparser2-10.1.0.tgz", @@ -4419,6 +5655,32 @@ "url": "https://opencollective.com/express" } }, + "node_modules/http-proxy-agent": { + "version": "7.0.2", + "resolved": "https://registry.npmjs.org/http-proxy-agent/-/http-proxy-agent-7.0.2.tgz", + "integrity": "sha512-T1gkAiYYDWYx3V5Bmyu7HcfcvL7mUrTWiM6yOfa3PIphViJ/gFPbvidQ+veqSOHci/PxBcDabeUNCzpOODJZig==", + "license": "MIT", + "dependencies": { + "agent-base": "^7.1.0", + "debug": "^4.3.4" + }, + "engines": { + "node": ">= 14" + } + }, + "node_modules/https-proxy-agent": { + "version": "7.0.6", + "resolved": "https://registry.npmjs.org/https-proxy-agent/-/https-proxy-agent-7.0.6.tgz", + "integrity": "sha512-vK9P5/iUfdl95AI+JVyUuIcVtd4ofvtrOr3HNtM2yxC9bnMbEdp3x01OhQNnjb8IJYi38VlTE3mBXwcfvywuSw==", + "license": "MIT", + "dependencies": { + "agent-base": "^7.1.2", + "debug": "4" + }, + "engines": { + "node": ">= 14" + } + }, "node_modules/ieee754": { "version": "1.2.1", "resolved": "https://registry.npmjs.org/ieee754/-/ieee754-1.2.1.tgz", @@ -4481,7 +5743,22 @@ "binary-extensions": "^2.0.0" }, "engines": { - "node": ">=8" + "node": ">=8" + } + }, + "node_modules/is-docker": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/is-docker/-/is-docker-3.0.0.tgz", + "integrity": "sha512-eljcgEDlEns/7AXFosB5K/2nCM4P7FQPkGc/DWLy5rmFEWvZayGrik1d9/QIY5nJ4f9YsVvBkA6kJpHn9rISdQ==", + "license": "MIT", + "bin": { + "is-docker": "cli.js" + }, + "engines": { + "node": "^12.20.0 || ^14.13.1 || >=16.0.0" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" } }, "node_modules/is-extglob": { @@ -4516,6 +5793,24 @@ "node": ">=0.10.0" } }, + "node_modules/is-inside-container": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/is-inside-container/-/is-inside-container-1.0.0.tgz", + "integrity": "sha512-KIYLCCJghfHZxqjYBE7rEy0OBuTd5xCHS7tHVgvCLkx7StIoaxwNW3hCALgEUjFfeRk+MG/Qxmp/vtETEF3tRA==", + "license": "MIT", + "dependencies": { + "is-docker": "^3.0.0" + }, + "bin": { + "is-inside-container": "cli.js" + }, + "engines": { + "node": ">=14.16" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, "node_modules/is-number": { "version": "7.0.0", "resolved": "https://registry.npmjs.org/is-number/-/is-number-7.0.0.tgz", @@ -4535,6 +5830,45 @@ "node": ">=0.10.0" } }, + "node_modules/is-stream": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/is-stream/-/is-stream-2.0.1.tgz", + "integrity": "sha512-hFoiJiTl63nn+kstHGBtewWSKnQLpyb155KHheA1l39uvtO9nWIop1p3udqPcUd/xbF1VLMO4n7OI6p7RbngDg==", + "license": "MIT", + "engines": { + "node": ">=8" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/is-unsafe": { + "version": "2.0.2", + "resolved": "https://registry.npmjs.org/is-unsafe/-/is-unsafe-2.0.2.tgz", + "integrity": "sha512-HgbIHPBH0KHHCcjLfGsCvhtPTVxjaAZlXjwdz7/GQC40SjSe4sfQsar8J5VFo8JOSbarkpV0OLG95bbaNd9aAQ==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/NaturalIntelligence" + } + ], + "license": "MIT" + }, + "node_modules/is-wsl": { + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/is-wsl/-/is-wsl-3.1.1.tgz", + "integrity": "sha512-e6rvdUCiQCAuumZslxRJWR/Doq4VpPR82kqclvcS0efgt430SlGIk05vdCN58+VrzgtIcfNODjozVielycD4Sw==", + "license": "MIT", + "dependencies": { + "is-inside-container": "^1.0.0" + }, + "engines": { + "node": ">=16" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, "node_modules/isomorphic.js": { "version": "0.2.5", "resolved": "https://registry.npmjs.org/isomorphic.js/-/isomorphic.js-0.2.5.tgz", @@ -4593,6 +5927,15 @@ "dev": true, "license": "Apache-2.0" }, + "node_modules/json-bigint": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/json-bigint/-/json-bigint-1.0.0.tgz", + "integrity": "sha512-SiPv/8VpZuWbvLSMtTDU8hEfrZWg/mH/nV/b4o0CYbSxu1UIQPLdwKOCIyLQX+VIPO5vrLX3i8qtqFyhdPSUSQ==", + "license": "MIT", + "dependencies": { + "bignumber.js": "^9.0.0" + } + }, "node_modules/json-schema-ref-resolver": { "version": "3.0.0", "resolved": "https://registry.npmjs.org/json-schema-ref-resolver/-/json-schema-ref-resolver-3.0.0.tgz", @@ -4647,6 +5990,49 @@ "graceful-fs": "^4.1.6" } }, + "node_modules/jsonwebtoken": { + "version": "9.0.3", + "resolved": "https://registry.npmjs.org/jsonwebtoken/-/jsonwebtoken-9.0.3.tgz", + "integrity": "sha512-MT/xP0CrubFRNLNKvxJ2BYfy53Zkm++5bX9dtuPbqAeQpTVe0MQTFhao8+Cp//EmJp244xt6Drw/GVEGCUj40g==", + "license": "MIT", + "dependencies": { + "jws": "^4.0.1", + "lodash.includes": "^4.3.0", + "lodash.isboolean": "^3.0.3", + "lodash.isinteger": "^4.0.4", + "lodash.isnumber": "^3.0.3", + "lodash.isplainobject": "^4.0.6", + "lodash.isstring": "^4.0.1", + "lodash.once": "^4.0.0", + "ms": "^2.1.1", + "semver": "^7.5.4" + }, + "engines": { + "node": ">=12", + "npm": ">=6" + } + }, + "node_modules/jwa": { + "version": "2.0.1", + "resolved": "https://registry.npmjs.org/jwa/-/jwa-2.0.1.tgz", + "integrity": "sha512-hRF04fqJIP8Abbkq5NKGN0Bbr3JxlQ+qhZufXVr0DvujKy93ZCbXZMHDL4EOtodSbCWxOqR8MS1tXA5hwqCXDg==", + "license": "MIT", + "dependencies": { + "buffer-equal-constant-time": "^1.0.1", + "ecdsa-sig-formatter": "1.0.11", + "safe-buffer": "^5.0.1" + } + }, + "node_modules/jws": { + "version": "4.0.1", + "resolved": "https://registry.npmjs.org/jws/-/jws-4.0.1.tgz", + "integrity": "sha512-EKI/M/yqPncGUUh44xz0PxSidXFr/+r0pA70+gIYhjv+et7yxM+s29Y+VGDkovRofQem0fs7Uvf4+YmAdyRduA==", + "license": "MIT", + "dependencies": { + "jwa": "^2.0.1", + "safe-buffer": "^5.0.1" + } + }, "node_modules/launder": { "version": "1.7.1", "resolved": "https://registry.npmjs.org/launder/-/launder-1.7.1.tgz", @@ -4726,6 +6112,48 @@ "node": ">=8" } }, + "node_modules/lodash.includes": { + "version": "4.3.0", + "resolved": "https://registry.npmjs.org/lodash.includes/-/lodash.includes-4.3.0.tgz", + "integrity": "sha512-W3Bx6mdkRTGtlJISOvVD/lbqjTlPPUDTMnlXZFnVwi9NKJ6tiAk6LVdlhZMm17VZisqhKcgzpO5Wz91PCt5b0w==", + "license": "MIT" + }, + "node_modules/lodash.isboolean": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/lodash.isboolean/-/lodash.isboolean-3.0.3.tgz", + "integrity": "sha512-Bz5mupy2SVbPHURB98VAcw+aHh4vRV5IPNhILUCsOzRmsTmSQ17jIuqopAentWoehktxGd9e/hbIXq980/1QJg==", + "license": "MIT" + }, + "node_modules/lodash.isinteger": { + "version": "4.0.4", + "resolved": "https://registry.npmjs.org/lodash.isinteger/-/lodash.isinteger-4.0.4.tgz", + "integrity": "sha512-DBwtEWN2caHQ9/imiNeEA5ys1JoRtRfY3d7V9wkqtbycnAmTvRRmbHKDV4a0EYc678/dia0jrte4tjYwVBaZUA==", + "license": "MIT" + }, + "node_modules/lodash.isnumber": { + "version": "3.0.3", + "resolved": "https://registry.npmjs.org/lodash.isnumber/-/lodash.isnumber-3.0.3.tgz", + "integrity": "sha512-QYqzpfwO3/CWf3XP+Z+tkQsfaLL/EnUlXWVkIk5FUPc4sBdTehEqZONuyRt2P67PXAk+NXmTBcc97zw9t1FQrw==", + "license": "MIT" + }, + "node_modules/lodash.isplainobject": { + "version": "4.0.6", + "resolved": "https://registry.npmjs.org/lodash.isplainobject/-/lodash.isplainobject-4.0.6.tgz", + "integrity": "sha512-oSXzaWypCMHkPC3NvBEaPHf0KsA5mvPrOPgQWDsbg8n7orZ290M0BmC/jgRZ4vcJ6DTAhjrsSYgdsW/F+MFOBA==", + "license": "MIT" + }, + "node_modules/lodash.isstring": { + "version": "4.0.1", + "resolved": "https://registry.npmjs.org/lodash.isstring/-/lodash.isstring-4.0.1.tgz", + "integrity": "sha512-0wJxfxH1wgO3GrbuP+dTTk7op+6L41QCXbGINEmD+ny/G/eCqGzxyCsh7159S+mgDDcoarnBw6PC1PS5+wUGgw==", + "license": "MIT" + }, + "node_modules/lodash.once": { + "version": "4.1.1", + "resolved": "https://registry.npmjs.org/lodash.once/-/lodash.once-4.1.1.tgz", + "integrity": "sha512-Sb487aTOCr9drQVL8pIxOzVhafOjZN9UU54hiN8PU3uAiSV7lx1yYNpbNmex2PK6dSJoNTSJUUswT651yww3Mg==", + "license": "MIT" + }, "node_modules/lru-cache": { "version": "11.2.4", "resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-11.2.4.tgz", @@ -4823,6 +6251,13 @@ "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", "license": "MIT" }, + "node_modules/nan": { + "version": "2.28.0", + "resolved": "https://registry.npmjs.org/nan/-/nan-2.28.0.tgz", + "integrity": "sha512-fTsDz99OTq2sVePhGdp4qQhggZFtKr64ZNVyVajRKtMOkJxYekplBh577PiJB12v/D3s2E5cGtOI45LWp6rnLQ==", + "license": "MIT", + "optional": true + }, "node_modules/nanoid": { "version": "6.0.1", "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-6.0.1.tgz", @@ -4853,6 +6288,46 @@ "node": ">= 8.0.0" } }, + "node_modules/node-domexception": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/node-domexception/-/node-domexception-1.0.0.tgz", + "integrity": "sha512-/jKZoMpw0F8GRwl4/eLROPA3cfcXtLApP0QzLmUT/HuPCZWyB7IY9ZrMeKw2O/nFIqPQB3PVM9aYm0F312AXDQ==", + "deprecated": "Use your platform's native DOMException instead", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/jimmywarting" + }, + { + "type": "github", + "url": "https://paypal.me/jimmywarting" + } + ], + "license": "MIT", + "engines": { + "node": ">=10.5.0" + } + }, + "node_modules/node-fetch": { + "version": "2.7.0", + "resolved": "https://registry.npmjs.org/node-fetch/-/node-fetch-2.7.0.tgz", + "integrity": "sha512-c4FRfUm/dbcWZ7U+1Wq0AwCyFL+3nt2bEw05wfxSz+DWpWsitgmSgYmy2dQdWyKC1694ELPqMs/YzUSNozLt8A==", + "license": "MIT", + "dependencies": { + "whatwg-url": "^5.0.0" + }, + "engines": { + "node": "4.x || >=6.0.0" + }, + "peerDependencies": { + "encoding": "^0.1.0" + }, + "peerDependenciesMeta": { + "encoding": { + "optional": true + } + } + }, "node_modules/nodemon": { "version": "3.1.14", "resolved": "https://registry.npmjs.org/nodemon/-/nodemon-3.1.14.tgz", @@ -4969,6 +6444,24 @@ "wrappy": "1" } }, + "node_modules/open": { + "version": "10.2.0", + "resolved": "https://registry.npmjs.org/open/-/open-10.2.0.tgz", + "integrity": "sha512-YgBpdJHPyQ2UE5x+hlSXcnejzAvD0b22U2OuAP+8OnlJT+PjWPxtgmGqKKc+RgTM63U9gN0YzrYc71R2WT/hTA==", + "license": "MIT", + "dependencies": { + "default-browser": "^5.2.1", + "define-lazy-prop": "^3.0.0", + "is-inside-container": "^1.0.0", + "wsl-utils": "^0.1.0" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, "node_modules/openapi-types": { "version": "12.1.3", "resolved": "https://registry.npmjs.org/openapi-types/-/openapi-types-12.1.3.tgz", @@ -5195,6 +6688,21 @@ "node": ">=8" } }, + "node_modules/path-expression-matcher": { + "version": "1.6.2", + "resolved": "https://registry.npmjs.org/path-expression-matcher/-/path-expression-matcher-1.6.2.tgz", + "integrity": "sha512-enSlaiat05iasnzmgNxRj8reFdj3puY2QpNgP1aPIaVfT6nn9ICuPoFlKHk8EN22HcwewshO+mN2DGbkCEOtqQ==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/NaturalIntelligence" + } + ], + "license": "MIT", + "engines": { + "node": ">=14.0.0" + } + }, "node_modules/path-scurry": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/path-scurry/-/path-scurry-2.0.1.tgz", @@ -5630,6 +7138,28 @@ "node": ">=10" } }, + "node_modules/retry": { + "version": "0.13.1", + "resolved": "https://registry.npmjs.org/retry/-/retry-0.13.1.tgz", + "integrity": "sha512-XQBQ3I8W1Cge0Seh+6gjj03LbmRFWuoszgK9ooCpwYIrhhoO80pfq4cUkU5DkknwfOfFteRwlZ56PYOGYyFWdg==", + "license": "MIT", + "engines": { + "node": ">= 4" + } + }, + "node_modules/retry-request": { + "version": "9.0.1", + "resolved": "https://registry.npmjs.org/retry-request/-/retry-request-9.0.1.tgz", + "integrity": "sha512-4kGHEBl0ME6gR+8QB1sPMyg58jUxLUcCZNeqLDrfqzk0eWQN8XJFmeu7fmZCjlZCRW0S2u64Ik1HkO/0sWgCYQ==", + "license": "MIT", + "dependencies": { + "extend": "^3.0.2", + "teeny-request": "^11.0.0" + }, + "engines": { + "node": ">=22" + } + }, "node_modules/reusify": { "version": "1.1.0", "resolved": "https://registry.npmjs.org/reusify/-/reusify-1.1.0.tgz", @@ -5646,6 +7176,18 @@ "integrity": "sha512-q1b3N5QkRUWUl7iyylaaj3kOpIT0N2i9MqIEQXP73GVsN9cw3fdx8X63cEmWhJGi2PPCF23Ijp7ktmd39rawIA==", "license": "MIT" }, + "node_modules/run-applescript": { + "version": "7.1.0", + "resolved": "https://registry.npmjs.org/run-applescript/-/run-applescript-7.1.0.tgz", + "integrity": "sha512-DPe5pVFaAsinSaV6QjQ6gdiedWDcRCbUuiQfQa2wmWV7+xC9bGulGI8+TdRmoFkAPaBXk8CrAbnlY2ISniJ47Q==", + "license": "MIT", + "engines": { + "node": ">=18" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, "node_modules/safe-buffer": { "version": "5.2.1", "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz", @@ -5918,6 +7460,23 @@ } } }, + "node_modules/simple-git": { + "version": "3.36.0", + "resolved": "https://registry.npmjs.org/simple-git/-/simple-git-3.36.0.tgz", + "integrity": "sha512-cGQjLjK8bxJw4QuYT7gxHw3/IouVESbhahSsHrX97MzCL1gu2u7oy38W6L2ZIGECEfIBG4BabsWDPjBxJENv9Q==", + "license": "MIT", + "dependencies": { + "@kwsites/file-exists": "^1.1.1", + "@kwsites/promise-deferred": "^1.1.1", + "@simple-git/args-pathspec": "^1.0.3", + "@simple-git/argv-parser": "^1.1.0", + "debug": "^4.4.0" + }, + "funding": { + "type": "github", + "url": "https://github.com/steveukx/git-js?sponsor=1" + } + }, "node_modules/simple-update-notifier": { "version": "2.0.0", "resolved": "https://registry.npmjs.org/simple-update-notifier/-/simple-update-notifier-2.0.0.tgz", @@ -5958,6 +7517,40 @@ "node": ">= 10.x" } }, + "node_modules/ssh2": { + "version": "1.17.0", + "resolved": "https://registry.npmjs.org/ssh2/-/ssh2-1.17.0.tgz", + "integrity": "sha512-wPldCk3asibAjQ/kziWQQt1Wh3PgDFpC0XpwclzKcdT1vql6KeYxf5LIt4nlFkUeR8WuphYMKqUA56X4rjbfgQ==", + "hasInstallScript": true, + "dependencies": { + "asn1": "^0.2.6", + "bcrypt-pbkdf": "^1.0.2" + }, + "engines": { + "node": ">=10.16.0" + }, + "optionalDependencies": { + "cpu-features": "~0.0.10", + "nan": "^2.23.0" + } + }, + "node_modules/ssh2-sftp-client": { + "version": "12.1.1", + "resolved": "https://registry.npmjs.org/ssh2-sftp-client/-/ssh2-sftp-client-12.1.1.tgz", + "integrity": "sha512-wYVDgwkpcKG2iPGQQ+QR33xkWqLFIaVrYvA+uON4pmxTPaPuB81f1aooUEPN75e/9DCK6rrKYXb6zR6zP3+EtA==", + "license": "Apache-2.0", + "dependencies": { + "concat-stream": "^2.0.0", + "ssh2": "^1.16.0" + }, + "engines": { + "node": ">=18.20.4" + }, + "funding": { + "type": "individual", + "url": "https://square.link/u/4g7sPflL" + } + }, "node_modules/statuses": { "version": "2.0.2", "resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.2.tgz", @@ -5967,6 +7560,15 @@ "node": ">= 0.8" } }, + "node_modules/stream-events": { + "version": "1.0.5", + "resolved": "https://registry.npmjs.org/stream-events/-/stream-events-1.0.5.tgz", + "integrity": "sha512-E1GUzBSgvct8Jsb3v2X15pjzN1tYebtbLaMg+eBOUOAxgbLoSbT2NS91ckc5lJD1KfLjId+jXJRgo0qnV5Nerg==", + "license": "MIT", + "dependencies": { + "stubs": "^3.0.0" + } + }, "node_modules/stream-shift": { "version": "1.0.3", "resolved": "https://registry.npmjs.org/stream-shift/-/stream-shift-1.0.3.tgz", @@ -6008,6 +7610,60 @@ "node": ">=8" } }, + "node_modules/strnum": { + "version": "2.4.2", + "resolved": "https://registry.npmjs.org/strnum/-/strnum-2.4.2.tgz", + "integrity": "sha512-rDG3Ah4TV0k1hWvLSzkZtMmLN9+eS+h3knq4MP6A42Y3Yh5qGNnOUs1jJkoSr8FG5dsL28c7KgkIBzSEykqtuw==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/NaturalIntelligence" + } + ], + "license": "MIT", + "dependencies": { + "anynum": "^1.0.1" + } + }, + "node_modules/stubs": { + "version": "3.0.0", + "resolved": "https://registry.npmjs.org/stubs/-/stubs-3.0.0.tgz", + "integrity": "sha512-PdHt7hHUJKxvTCgbKX9C1V/ftOcjJQgz8BZwNfV5c4B6dcGqlpelTbJ999jBGZ2jYiPAwcX5dP6oBwVlBlUbxw==", + "license": "MIT" + }, + "node_modules/teeny-request": { + "version": "11.0.1", + "resolved": "https://registry.npmjs.org/teeny-request/-/teeny-request-11.0.1.tgz", + "integrity": "sha512-bNr5j2YjSdajgCVsp+8JVjRf1uHIbpNISLeKp/7V1NnVMX3Gh6H/kECNGlm2Q3I68ACODQ0iv6aZ3Rvm8WgLGA==", + "license": "Apache-2.0", + "dependencies": { + "http-proxy-agent": "^7.0.0", + "https-proxy-agent": "^7.0.1", + "node-fetch": "^3.3.2", + "stream-events": "^1.0.5" + }, + "engines": { + "node": ">=22" + } + }, + "node_modules/teeny-request/node_modules/node-fetch": { + "version": "3.3.2", + "resolved": "https://registry.npmjs.org/node-fetch/-/node-fetch-3.3.2.tgz", + "integrity": "sha512-dRB78srN/l6gqWulah9SrxeYnxeddIG30+GOqK/9OlLVyLg3HPnr6SqOWTWOXKRwC2eGYCkZ59NNuSgvSrpgOA==", + "license": "MIT", + "dependencies": { + "data-uri-to-buffer": "^4.0.0", + "fetch-blob": "^3.1.4", + "formdata-polyfill": "^4.0.10" + }, + "engines": { + "node": "^12.20.0 || ^14.13.1 || >=16.0.0" + }, + "funding": { + "type": "opencollective", + "url": "https://opencollective.com/node-fetch" + } + }, "node_modules/thread-stream": { "version": "4.0.0", "resolved": "https://registry.npmjs.org/thread-stream/-/thread-stream-4.0.0.tgz", @@ -6080,6 +7736,12 @@ "nodetouch": "bin/nodetouch.js" } }, + "node_modules/tr46": { + "version": "0.0.3", + "resolved": "https://registry.npmjs.org/tr46/-/tr46-0.0.3.tgz", + "integrity": "sha512-N3WMsuqV66lT30CrXNbEjx4GEwlow3v6rr4mCcv6prnfwhS01rkgyFdjPNBYd9br7LpXV1+Emh01fHnq2Gdgrw==", + "license": "MIT" + }, "node_modules/tslib": { "version": "2.8.1", "resolved": "https://registry.npmjs.org/tslib/-/tslib-2.8.1.tgz", @@ -6104,6 +7766,12 @@ "integrity": "sha512-Xni35NKzjgMrwevysHTCArtLDpPvye8zV/0E4EyYn43P7/7qvQwPh9BGkHewbMulVntbigmcT7rdX3BNo9wRJg==", "license": "0BSD" }, + "node_modules/tweetnacl": { + "version": "0.14.5", + "resolved": "https://registry.npmjs.org/tweetnacl/-/tweetnacl-0.14.5.tgz", + "integrity": "sha512-KXXFFdAbFXY4geFIwoyNK+f5Z1b7swfXABfL7HXCmoIWMKU3dmS26672A4EeQtDzLKy7SXmfBu51JolvEKwtGA==", + "license": "Unlicense" + }, "node_modules/type-is": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/type-is/-/type-is-2.0.1.tgz", @@ -6118,6 +7786,12 @@ "node": ">= 0.6" } }, + "node_modules/typedarray": { + "version": "0.0.6", + "resolved": "https://registry.npmjs.org/typedarray/-/typedarray-0.0.6.tgz", + "integrity": "sha512-/aCDEGatGvZ2BIk+HmLf4ifCJFwvKFNb9/JeZPMulfgFracn9QFcAf5GO8B/mweUjSoblS5In0cWhqpfs/5PQA==", + "license": "MIT" + }, "node_modules/typescript": { "version": "7.0.2", "resolved": "https://registry.npmjs.org/typescript/-/typescript-7.0.2.tgz", @@ -6213,6 +7887,21 @@ "node": ">= 0.8" } }, + "node_modules/web-streams-polyfill": { + "version": "3.3.3", + "resolved": "https://registry.npmjs.org/web-streams-polyfill/-/web-streams-polyfill-3.3.3.tgz", + "integrity": "sha512-d2JWLCivmZYTSIoge9MsgFCZrt571BikcWGYkjC1khllbTeDlGqZ2D8vD8E/lJa8WGWbb7Plm8/XJYV7IJHZZw==", + "license": "MIT", + "engines": { + "node": ">= 8" + } + }, + "node_modules/webidl-conversions": { + "version": "3.0.1", + "resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-3.0.1.tgz", + "integrity": "sha512-2JAn3z8AR6rjK8Sm8orRC0h/bcl/DqL7tRPdGZ4I1CjdF+EaMLmYxBHyXuKL849eucPFhvBoxMsflfOb8kxaeQ==", + "license": "BSD-2-Clause" + }, "node_modules/whatwg-encoding": { "version": "3.1.1", "resolved": "https://registry.npmjs.org/whatwg-encoding/-/whatwg-encoding-3.1.1.tgz", @@ -6247,6 +7936,16 @@ "node": ">=18" } }, + "node_modules/whatwg-url": { + "version": "5.0.0", + "resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-5.0.0.tgz", + "integrity": "sha512-saE57nupxk6v3HY35+jzBwYa0rKSy0XR8JSxZPwgLr7ys0IBzhGviA1/TUGJLmSVqs8pb9AnvICXEuOHLprYTw==", + "license": "MIT", + "dependencies": { + "tr46": "~0.0.3", + "webidl-conversions": "^3.0.0" + } + }, "node_modules/which-module": { "version": "2.0.1", "resolved": "https://registry.npmjs.org/which-module/-/which-module-2.0.1.tgz", @@ -6294,6 +7993,36 @@ } } }, + "node_modules/wsl-utils": { + "version": "0.1.0", + "resolved": "https://registry.npmjs.org/wsl-utils/-/wsl-utils-0.1.0.tgz", + "integrity": "sha512-h3Fbisa2nKGPxCpm89Hk33lBLsnaGBvctQopaBSOW/uIs6FTe1ATyAnKFJrzVs9vpGdsTe73WF3V4lIsk4Gacw==", + "license": "MIT", + "dependencies": { + "is-wsl": "^3.1.0" + }, + "engines": { + "node": ">=18" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } + }, + "node_modules/xml-naming": { + "version": "0.3.0", + "resolved": "https://registry.npmjs.org/xml-naming/-/xml-naming-0.3.0.tgz", + "integrity": "sha512-ghig2TBE/H11aOVgmahA3MhimvkBr6JIYknH/Dhdk10nXwdbIqBJsbfMxpvFPG8bAw77gN29aQWvKpmVoPlvPQ==", + "funding": [ + { + "type": "github", + "url": "https://github.com/sponsors/NaturalIntelligence" + } + ], + "license": "MIT", + "engines": { + "node": ">=16.0.0" + } + }, "node_modules/xtend": { "version": "4.0.2", "resolved": "https://registry.npmjs.org/xtend/-/xtend-4.0.2.tgz", @@ -6395,6 +8124,18 @@ "type": "GitHub Sponsors ❤", "url": "https://github.com/sponsors/dmonad" } + }, + "node_modules/yocto-queue": { + "version": "0.1.0", + "resolved": "https://registry.npmjs.org/yocto-queue/-/yocto-queue-0.1.0.tgz", + "integrity": "sha512-rVksvsnNCdJ/ohGc6xgPwyN8eheCxsiLM8mxuE/t/mOVqJewPuO1miLpTHQiRgTKCLexL4MeAFVagts7HmNZ2Q==", + "license": "MIT", + "engines": { + "node": ">=10" + }, + "funding": { + "url": "https://github.com/sponsors/sindresorhus" + } } } } diff --git a/backend/package.json b/backend/package.json index fd3b621fe..58007409d 100644 --- a/backend/package.json +++ b/backend/package.json @@ -35,6 +35,10 @@ "db-up": "drizzle-kit up --dialect=postgresql --out=./db/migrations" }, "dependencies": { + "@aws-sdk/client-s3": "3.1116.0", + "@aws-sdk/s3-request-presigner": "3.1116.0", + "@azure/identity": "4.13.2", + "@azure/storage-blob": "12.33.0", "@fastify/compress": "9.2.0", "@fastify/cookie": "11.1.2", "@fastify/cors": "11.3.0", @@ -46,6 +50,7 @@ "@fastify/swagger": "9.8.1", "@fastify/swagger-ui": "6.1.1", "@fastify/websocket": "11.3.0", + "@google-cloud/storage": "8.0.1", "@gquittet/graceful-server": "6.0.10", "@iconify/utils": "3.1.4", "@simplewebauthn/server": "13.3.2", @@ -73,6 +78,8 @@ "qrcode": "1.5.4", "sanitize-html": "2.17.6", "semver": "7.8.5", + "simple-git": "3.36.0", + "ssh2-sftp-client": "12.1.1", "uuid": "14.0.1", "y-protocols": "1.0.7", "yjs": "13.6.32" @@ -88,6 +95,7 @@ "@types/qrcode": "1.5.6", "@types/sanitize-html": "2.16.1", "@types/semver": "7.8.0", + "@types/ssh2-sftp-client": "9.0.6", "@types/ws": "8.18.1", "drizzle-kit": "1.0.0-rc.4", "nodemon": "3.1.14", diff --git a/backend/tasks/simple/sync-storage-targets.ts b/backend/tasks/simple/sync-storage-targets.ts new file mode 100644 index 000000000..404e56a48 --- /dev/null +++ b/backend/tasks/simple/sync-storage-targets.ts @@ -0,0 +1,69 @@ +/** + * Sync every storage target that has a remote to keep in step with, and whose site is due. + * + * The git target's schedule, and what makes it a synchronized store rather than a local repository + * that happens to get committed to. A change is committed the moment it is made — that happens on the + * request, not here — and this is what pushes those commits and brings back what other people pushed. + * + * **This runs every minute; the site's `syncInterval` decides what actually happens.** The interval is + * a per-site setting, so one schedule cannot express it — the tick is therefore as fine as the + * shortest interval anybody can ask for, and each site is skipped on the ticks that are not its own. + * + * Due-ness is read off the clock rather than off a record of when each target last synced. A site with + * a five-minute interval syncs on every fifth minute of the epoch, which needs nothing stored, means + * two instances agree without coordinating, survives a restart, and cannot drift. What it gives up is + * catching up on a missed tick: an instance that was down at the moment simply waits for the next one, + * which for something that runs all day is the right trade. + * + * Every target is attempted whatever the ones before it did: a site whose credentials have expired + * must not stop the rest of them syncing. `executeAction` records the outcome on the target itself, + * so a failure shows up on its Status card in the admin area rather than only in this log. + * + * **One instance at a time.** The scheduler hands a job to a single instance, which is the one whose + * working copy is synced. Every instance in a high-availability set keeps its own, so they each fall + * in step at their own turn rather than fighting over one repository. + */ +export async function task(): Promise { + const syncable = await WIKI.models.storage.syncableTargets() + if (syncable.length < 1) { + return + } + + const minute = Math.floor(Temporal.Now.instant().epochMilliseconds / 60_000) + const targets = syncable.filter((target) => { + const interval = WIKI.models.storage.syncIntervalFor(target.siteId) + // -> An interval nothing can be made of is a site that is never synced on a schedule, rather than + // one synced every minute. `validateSiteConfig` refuses to store such a value in the first + // place, so this is the belt to that braces. + return interval > 0 && minute % interval === 0 + }) + if (targets.length < 1) { + return + } + + // -> A pull creates content, and content records who authored it. There is nobody behind a + // scheduled run, so it is recorded against the wiki's own administrator. + const actorId = await WIKI.models.users.getSystemActorId() + if (!actorId) { + WIKI.logger.warn( + 'Syncing storage targets: no active administrator to attribute incoming content to [ SKIPPED ]' + ) + return + } + + WIKI.logger.info(`Syncing ${targets.length} storage target(s)...`) + let failed = 0 + for (const target of targets) { + try { + const message = await WIKI.models.storage.executeAction(target, 'sync', actorId) + WIKI.logger.info(`Synced ${target.title} for site ${target.siteId}: ${message ?? 'done'}`) + } catch (err: any) { + failed++ + WIKI.logger.warn(`Could not sync ${target.title} for site ${target.siteId} [ FAILED ]`) + WIKI.logger.warn(err.message) + } + } + WIKI.logger.info( + `Syncing storage targets: ${targets.length - failed} of ${targets.length} succeeded [ COMPLETED ]` + ) +} diff --git a/frontend/src/layouts/AdminLayout.vue b/frontend/src/layouts/AdminLayout.vue index 75d69b721..5fe4d5a14 100644 --- a/frontend/src/layouts/AdminLayout.vue +++ b/frontend/src/layouts/AdminLayout.vue @@ -216,8 +216,12 @@ {{ t('admin.storage.title') }} - - + + 0 ? 'count-badge count-badge--filled' : 'count-badge' } +/** + * Whether every enabled storage target of the current site is behaving. + * + * Read off the store rather than fetched here, so that the storage page can put the light right the + * moment it learns something without the sidebar having to ask again. + */ +const storageHealthy = computed(() => adminStore.storageHealth.status === 'healthy') + // WATCHERS watch( @@ -640,6 +652,10 @@ watch( if (newValue && route.params.siteid !== newValue) { router.push({ params: { siteid: newValue } }) } + // -> Storage is configured per site, so the light belongs to whichever one is selected + if (newValue && userStore.can('manage:sites')) { + adminStore.fetchStorageStatus(newValue) + } } ) @@ -659,6 +675,10 @@ onMounted(async () => { }) } adminStore.fetchInfo() + // -> Only for a role that can see the Storage item at all; anyone else would be asking for a 403 + if (adminStore.currentSiteId && userStore.can('manage:sites')) { + adminStore.fetchStorageStatus(adminStore.currentSiteId) + } }) diff --git a/frontend/src/pages/AdminStorage.vue b/frontend/src/pages/AdminStorage.vue index 3cfc368e0..dbae6d3b0 100644 --- a/frontend/src/pages/AdminStorage.vue +++ b/frontend/src/pages/AdminStorage.vue @@ -69,9 +69,7 @@ }} - + @@ -140,10 +138,13 @@ - + + - {{ t('admin.storage.config') }} + {{ t('admin.storage.targetConfig') }} + + + + + + {{ t('admin.storage.deliveryConfig') }} + + + + + + {{ t(`admin.storage.deliveryModeStreaming`) }} + {{ + t(`admin.storage.deliveryModeStreamingHint`) + }} + + + + + + + + + + {{ t(`admin.storage.deliveryModeDirect`) }} + {{ + t(`admin.storage.deliveryModeDirectHint`) + }} + + + + + + + + @@ -297,7 +383,25 @@ heading spends on its margin. --> - {{ t('admin.storage.status') }} + + {{ t('admin.storage.status') }} + + + - + {{ currentState.label }} - + states: "Error" on its own only sends an administrator to the server log. + + The captions carry their own top margins rather than the section carrying a + gap: a gap belongs to `WItemSection`, which every item in the admin area uses + for the tight label-and-hint pairing that wants no space at all. Only the two + unhealthy states set either of these, and they set both, so a one-line card + never ends up with a margin hanging off it. + + Uneven on purpose. The wider gap under the status separates the heading from + the detail, and the narrow one keeps the message and the moment it happened + reading as the one thing they are. --> + {{ currentState.message }} - + {{ relativeDate(currentState.since) }} @@ -416,6 +532,69 @@ :aria-label="t(`admin.storage.largeThreshold`)" /> + + + + + {{ t(`admin.storage.syncInterval`) }} + {{ t(`admin.storage.syncIntervalHint`) }} + + + + + + + + + + {{ t(`admin.storage.sitePrefix`) }} + {{ t(`admin.storage.sitePrefixHint`) }} + + + + + + + + + + {{ t(`admin.storage.localePrefix`) }} + {{ t(`admin.storage.localePrefixHint`) }} + + + + + + + + + + {{ t(`admin.storage.directAccessFallback`) }} + {{ t(`admin.storage.directAccessFallbackHint`) }} + + + + + + + {{ + t('admin.storage.pathLayoutHint') + }} + @@ -470,12 +649,17 @@ const state = reactive({ displayMode: 'targets', runningAction: false, runningActionHandler: '', + refreshingState: false, selectedTarget: '', desiredTarget: '', target: null, targets: [], - /** Site-wide, hence not on a target: see the Configuration tab. */ - largeThreshold: '' + /** Site-wide, hence not on a target: the three of them are the Configuration tab. */ + largeThreshold: '', + syncInterval: '', + directAccessFallback: 'stream', + sitePrefix: false, + localePrefix: true }) // CONSTANTS @@ -552,6 +736,7 @@ const currentState = computed(() => { label: t('admin.storage.stateError'), text: 'text-negative', dot: 'bg-negative', + flash: true, message: health.message, since: health.updatedAt } @@ -561,6 +746,7 @@ const currentState = computed(() => { label: t('admin.storage.stateWarning'), text: 'text-warning', dot: 'bg-warning', + flash: true, message: health.message, since: health.updatedAt } @@ -585,6 +771,16 @@ const actionsNotice = computed(() => { return savedEnabled.value ? null : t('admin.storage.actionsInactiveWarn') }) +/** + * What a site can do when a target set to hand out direct links cannot sign one. + * + * Computed rather than a constant so the labels follow the interface language. + */ +const directAccessFallbackOptions = computed(() => [ + { value: 'stream', label: t('admin.storage.directAccessFallbackStream') }, + { value: 'error', label: t('admin.storage.directAccessFallbackError') } +]) + /** The database target, which pages are always read from. */ const dbTarget = computed(() => state.targets.find((tgt) => tgt.module === 'db') ?? null) const dbTargetId = computed(() => dbTarget.value?.id ?? null) @@ -652,7 +848,15 @@ watch( */ function sourceOptions(type) { return state.targets - .filter((tgt) => tgt.isEnabled && tgt.contentTypes.activeTypes.includes(type)) + .filter( + (tgt) => + tgt.isEnabled && + tgt.contentTypes.activeTypes.includes(type) && + // -> A target may hold a content type without being somewhere to read it back from. SFTP is + // the one: every image on every page would be an SSH round trip, so it is a copy of the + // site's content rather than a source for it, and the server refuses the nomination too. + tgt.assetDelivery.isDeliverySupported !== false + ) .map((tgt) => ({ label: tgt.title, value: tgt.id })) } @@ -674,6 +878,7 @@ function sourceFor(type) { (tgt) => tgt.isEnabled && tgt.contentTypes.activeTypes.includes(type) && + tgt.assetDelivery.isDeliverySupported !== false && (tgt.assetDelivery.servedTypes ?? []).includes(type) ) if (nominated) { @@ -750,11 +955,16 @@ async function load() { try { const resp = await API_CLIENT.get(`sites/${adminStore.currentSiteId}/storage`).json() state.largeThreshold = resp?.largeThreshold ?? '' + state.syncInterval = resp?.syncInterval ?? '' + state.directAccessFallback = resp?.directAccessFallback ?? 'stream' + state.sitePrefix = resp?.sitePrefix ?? false + state.localePrefix = resp?.localePrefix ?? true state.targets = (resp?.targets ?? []).map((tgt) => ({ ...tgt, config: buildConfigEditor(tgt.props, tgt.config), saved: savedSnapshot(tgt) })) + adminStore.applyStorageTargets(state.targets) } catch (err) { notify({ type: 'negative', @@ -767,6 +977,42 @@ async function load() { state.loading-- } +/** + * Read every target's health back from the server, and nothing else. + * + * Deliberately not `load()`. What the Status card shows is the one part of this page the server + * writes on its own — `recordState`, as an upload is refused or a sync finishes — so that is the only + * part worth asking about again. Reloading the whole form would also throw away whatever content + * types or configuration the administrator has changed and not yet saved, which is a steep price for + * looking at a status line. + * + * `state.target` is a member of `state.targets` rather than a copy of one, so patching the array is + * what puts the new status in the card. + */ +async function refreshState() { + if (state.refreshingState) { + return + } + state.refreshingState = true + try { + const resp = await API_CLIENT.get(`sites/${adminStore.currentSiteId}/storage`).json() + for (const fresh of resp?.targets ?? []) { + const tgt = state.targets.find((item) => item.id === fresh.id) + if (tgt) { + tgt.state = fresh.state + } + } + adminStore.applyStorageTargets(state.targets) + } catch (err) { + notify({ + type: 'negative', + message: t('admin.storage.loadFailed'), + caption: apiErrorMessage(err) + }) + } + state.refreshingState = false +} + function configIfCheck(ifs) { if (!ifs || ifs.length < 1) { return true @@ -793,9 +1039,9 @@ function payloadFor(tgt) { activeTypes: tgt.contentTypes.activeTypes }, assetDelivery: { - // -> `streaming` and `directAccess` are deliberately not sent: nothing in this page edits them - // any more, and the server keeps whatever it has for a field a patch leaves out - // + mode: tgt.assetDelivery.mode, + baseUrl: tgt.assetDelivery.baseUrl ?? '', + linkExpiration: tgt.assetDelivery.linkExpiration ?? '', // -> Kept in step with the content types on the way out: a target that stopped storing a kind // cannot go on being the source for it, and the server refuses the pair outright servedTypes: (tgt.assetDelivery.servedTypes ?? []).filter((type) => @@ -822,6 +1068,10 @@ async function save() { const resp = await API_CLIENT.put(`sites/${adminStore.currentSiteId}/storage`, { json: { largeThreshold: state.largeThreshold, + syncInterval: state.syncInterval, + directAccessFallback: state.directAccessFallback, + sitePrefix: state.sitePrefix, + localePrefix: state.localePrefix, targets: state.targets.map(payloadFor) } }).json() @@ -888,6 +1138,28 @@ async function setEnabled(isEnabled) { } } +/** + * The light beside a target in the list. + * + * Three states, and the first two are about configuration rather than health: a target that is off + * is dark and still, and an enabled one pulses to say it is in use. The third is the reason this is a + * function and not a ternary — an enabled target that last failed at something turns amber, so the + * list says which target to go and look at without every row having to be opened. + * + * Amber for `error` as well as `warning`, matching the sidebar: what failed was something the wiki + * tried to do, and the wiki is still serving. A dark red light here means "switched off", which is a + * different thing entirely and already has this colour. + */ +function targetLight(target) { + if (!target.isEnabled) { + return { color: 'negative', pulse: false } + } + if (['warning', 'error'].includes(target.state?.status)) { + return { color: 'warning', pulse: true } + } + return { color: 'positive', pulse: true } +} + function getTargetSubtitle(target) { if (!target.isEnabled) { return t('admin.storage.inactiveTarget') @@ -943,6 +1215,9 @@ async function executeAction(act) { } state.runningAction = false state.runningActionHandler = '' + // -> An action is the heaviest thing a target is ever asked to do and the likeliest to change how + // it is behaving, either way round: this is where a failure appears, and where one clears + await refreshState() } // -> An action that declares a warning destroys something, so it is never run on a single click @@ -980,4 +1255,37 @@ onMounted(() => { .admin-storage-logo { border-radius: 5px; } + +/* + The dot fades away and back rather than growing a halo. Nothing about its size or position changes, + so it does not nudge the label beside it, and fading reads as a signal on a 10px dot in a way a 3px + glow could not -- the halo was competing with the coloured fill it sat around. + + Never quite to nothing: a dot that disappears reads as one that has gone out, and half of the time + the card would be showing a status with no colour against it. +*/ +.status-dot--alert { + animation: status-dot-alert 1.5s ease-in-out infinite; +} + +@keyframes status-dot-alert { + 0%, + 100% { + opacity: 1; + } + 50% { + opacity: 0.2; + } +} + +/* + A flashing dot is the one thing on this page that moves on its own, so it is also the one thing + that has to stop when the reader has asked for less of that. Nothing is lost by holding still: the + colour and the word beside it say the same thing. +*/ +@media (prefers-reduced-motion: reduce) { + .status-dot--alert { + animation: none; + } +} diff --git a/frontend/src/stores/admin.js b/frontend/src/stores/admin.js index 46b760f12..77616b641 100644 --- a/frontend/src/stores/admin.js +++ b/frontend/src/stores/admin.js @@ -23,6 +23,14 @@ export const useAdminStore = defineStore('admin', { isMetricsEnabled: false, isSchedulerHealthy: false }, + /** + * How the current site's storage is behaving, for the status light on the sidebar's Storage item. + * + * `degraded` names the targets that are not healthy, so the light can say more than that + * something is wrong somewhere. Kept here rather than on the storage page because the sidebar + * outlives it: the point of the light is to be visible from everywhere else in the admin area. + */ + storageHealth: { status: 'healthy', degraded: [] }, overlay: null, overlayOpts: {}, sites: [], @@ -69,6 +77,35 @@ export const useAdminStore = defineStore('admin', { this.info.isMailConfigured = resp?.isMailConfigured ?? false this.info.isSchedulerHealthy = resp?.isSchedulerHealthy ?? false }, + /** + * Work out the site's storage health from a list of targets. + * + * The one place the rule lives, called both by `fetchStorageStatus` and by the storage page, + * which has the same target list in front of it already and no reason to ask again for what it + * just loaded. Both pass the same shape — `isEnabled` and `state.status` — which is why the + * status endpoint answers with those fields rather than with a verdict of its own. + */ + applyStorageTargets(targets) { + const degraded = (targets ?? []) + .filter((tgt) => tgt.isEnabled && ['warning', 'error'].includes(tgt.state?.status)) + .map((tgt) => ({ id: tgt.id, title: tgt.title, status: tgt.state.status })) + this.storageHealth = { + // -> Worst wins: one target that refused an upload is not softened by four that are fine + status: degraded.some((tgt) => tgt.status === 'error') + ? 'error' + : degraded.length > 0 + ? 'warning' + : 'healthy', + degraded + } + }, + async fetchStorageStatus(siteId) { + if (!siteId) { + return + } + const resp = await API_CLIENT.get(`sites/${siteId}/storage/status`).json() + this.applyStorageTargets(resp?.targets ?? []) + }, async fetchSites() { this.sites = (await API_CLIENT.get('sites').json()) ?? [] if (!this.currentSiteId) {