diff --git a/docs/compatibility.md b/docs/compatibility.md index 98efca4..1b545e1 100644 --- a/docs/compatibility.md +++ b/docs/compatibility.md @@ -72,7 +72,7 @@ The plugin lives off data it does not own: the durable session log (event shapes ## Low-level parity -The plugin's low-level logic mirrors the newest supported dsh implementation: token estimation re-fits `dsh-token-meter`'s `estimateSystemMessage` (`estimateSystemTokens` / `estimateSystemContent`), and the TPS chip's first-token rule matches `dsh-llm`'s `assistantStreamFirstTokenTime`. The Context card's occupancy percentage is the composer ring's own derivation — `@deepseek-ai/dsh-client-ui-conversation`'s `contextOccupancy` (`Math.min(100, Math.round(used / window * 100))`, `used` = `contextPressure.projectedTokens ?? pressureTokens`) — read off the same token-meter projection and printed on a 0.1 grid, so the card and the ring can never disagree on the value and only the printed precision differs (the ring reads an integer, the card one decimal). The kernel's helper is internal to its browser bundle with no package export, so the plugin mirrors the formula (`occupancyPercent` in `src/client/headline.ts`) instead of importing it; where the meter's projection key is absent altogether (an older line), the card falls back to its own derived anchor. The Current Context bar's two auto-compaction reserve masks mirror `dsh-compaction-basic`'s own pressure trigger the same way (`src/client/reserve.ts`): `thresholdTokens = floor(min(contextWindow × thresholdRatio, contextWindow − reservedOutput − headroomTokens))`, with the session's durable window (`request/context`'s `contextWindow`) and routed output cap (`request/header`'s materialized `config.maxTokens` — the envelope value the backend's own `reservedCompletionTokens` reads) as the inputs, and that package's two shipped defaults (`thresholdRatio` 0.8, `headroomTokens` 65,536) as the policy: the web composition mounts the backend inside each agent preset's isolated realm, so no root-level plugin can read a deployment's configured policy, and only those defaults are mirrored — and because the backend REFUSES (throws `TargetPressureConfigError`, disabling pressure compaction) rather than clamps when `contextWindow − reservedOutput − headroomTokens` is non-positive, the bar reserves nothing and paints no masks for such a window instead of implying a trigger at zero. The timing card's wait/generation boundary deliberately diverges from that rule: it opens at the first OBSERVABLE instant — the first token, or an earlier `block-start` marker the token packing could not stamp (a redacted reasoning block leaves no chunk behind) — so the marker-tiled decode buckets and the wait/generation split read one window and the legend's shares never double-count it. When a supported line moves these implementations, the plugin's copies are updated to the newest one — the fold's statistics are differentially verified against the harness's own projection values on real logs (see "What each check means"). +The plugin's low-level logic mirrors the newest supported dsh implementation: token estimation re-fits `dsh-token-meter`'s `estimateSystemMessage` (`estimateSystemTokens` / `estimateSystemContent`) and its per-block text arm (`estimateTextTokens`, shared with the fold's message price), a file attachment's price re-fits `dsh-llm`'s `fileHandleText` as that request projection prices it (see the attachment-card rule), and the TPS chip's first-token rule matches `dsh-llm`'s `assistantStreamFirstTokenTime`. The Context card's occupancy percentage is the composer ring's own derivation — `@deepseek-ai/dsh-client-ui-conversation`'s `contextOccupancy` (`Math.min(100, Math.round(used / window * 100))`, `used` = `contextPressure.projectedTokens ?? pressureTokens`) — read off the same token-meter projection and printed on a 0.1 grid, so the card and the ring can never disagree on the value and only the printed precision differs (the ring reads an integer, the card one decimal). The kernel's helper is internal to its browser bundle with no package export, so the plugin mirrors the formula (`occupancyPercent` in `src/client/headline.ts`) instead of importing it; where the meter's projection key is absent altogether (an older line), the card falls back to its own derived anchor. The Current Context bar's two auto-compaction reserve masks mirror `dsh-compaction-basic`'s own pressure trigger the same way (`src/client/reserve.ts`): `thresholdTokens = floor(min(contextWindow × thresholdRatio, contextWindow − reservedOutput − headroomTokens))`, with the session's durable window (`request/context`'s `contextWindow`) and routed output cap (`request/header`'s materialized `config.maxTokens` — the envelope value the backend's own `reservedCompletionTokens` reads) as the inputs, and that package's two shipped defaults (`thresholdRatio` 0.8, `headroomTokens` 65,536) as the policy: the web composition mounts the backend inside each agent preset's isolated realm, so no root-level plugin can read a deployment's configured policy, and only those defaults are mirrored — and because the backend REFUSES (throws `TargetPressureConfigError`, disabling pressure compaction) rather than clamps when `contextWindow − reservedOutput − headroomTokens` is non-positive, the bar reserves nothing and paints no masks for such a window instead of implying a trigger at zero. The timing card's wait/generation boundary deliberately diverges from that rule: it opens at the first OBSERVABLE instant — the first token, or an earlier `block-start` marker the token packing could not stamp (a redacted reasoning block leaves no chunk behind) — so the marker-tiled decode buckets and the wait/generation split read one window and the legend's shares never double-count it. When a supported line moves these implementations, the plugin's copies are updated to the newest one — the fold's statistics are differentially verified against the harness's own projection values on real logs (see "What each check means"). ## Web client seams @@ -92,7 +92,7 @@ The guide capsule carries `order`, `title`, `description`, and `icon` (the `guid The File Activity card's file-name affordance is an optional-column story: the plugin opens the file's preview tab through `ctx.sidebarRight.openResource` (the built-in Files sidebar's own path), building the `dsh-resource://file/…` address with the harness's browser-safe `fileAddressFor` (inlined by the client bundle, exactly as the Sidebar's own file types inline it). Where the preview face is absent (or refuses the address), the name keeps its original system-open behavior; the `sidebar.nav` probe pins the navigation face's spelling. -The Context browser's attachment cards are another optional-column story. A message's `image` and `file` blocks render as one ordered card list, and a click hands the attachment to the harness's preview column through the same `previewAddressOf` → `openResourceVia` path the File Activity card rides. The path itself comes from the plugin's own `/api/dsh-context/attachment` fetch route (same authenticated `/api` fence, Connection exact-route registry, and cold-read gate as the detail and fleet routes): the route answers only an id the SESSION'S OWN log cites (a payload walk bounded in depth and total over hostile input, so an id no logged content block carries — the harness's own gateway is stricter still, reading declared fields per event type — resolves to nothing) and answers the store's named alias. A verbatim file already has one (`attachments.fileHostPath` → `/attachments/v1/files///`); a normalized image does not — its object is digest-named and suffixless, while the preview column dispatches its viewer by suffix — so the route publishes (or reuses) a named hard link under `/cache/dsh-context/previews///` and answers that. The link shares the immutable, digest-verified object bytes; the cache is disposable, and a deployment whose attachment store is not host-file-backed, or whose session log cannot be read, answers a typed `null` and the card reports "preview unavailable" instead. Image thumbnails keep riding the harness's own session-authorized `uiConversation.imageUrl` loader; the file-type glyph resolves defensively off the primitives module (`FileTypeGlyph`), and every card degrades to its metadata when a face is absent. +The Context browser's attachment cards are another optional-column story. A message's `image` and `file` blocks render as one ordered card list, and a click hands the attachment to the harness's preview column through the same `previewAddressOf` → `openResourceVia` path the File Activity card rides. The path itself comes from the plugin's own `/api/dsh-context/attachment` fetch route (same authenticated `/api` fence, Connection exact-route registry, and cold-read gate as the detail and fleet routes): the route answers only an id the SESSION'S OWN log cites (a payload walk bounded in depth and total over hostile input, so an id no logged content block carries — the harness's own gateway is stricter still, reading declared fields per event type — resolves to nothing) and answers the store's named alias. A verbatim file already has one (`attachments.fileHostPath` → `/attachments/v1/files///`); a normalized image does not — its object is digest-named and suffixless, while the preview column dispatches its viewer by suffix — so the route publishes (or reuses) a named hard link under `/cache/dsh-context/previews///` and answers that. The link shares the immutable, digest-verified object bytes; the cache is disposable, and a deployment whose attachment store is not host-file-backed, or whose session log cannot be read, answers a typed `null` and the card reports "preview unavailable" instead. Image thumbnails keep riding the harness's own session-authorized `uiConversation.imageUrl` loader; the file-type glyph resolves defensively off the primitives module (`FileTypeGlyph`), and every card degrades to its metadata when a face is absent. A card's token row is the REQUEST's price, never the attachment's size: an image row is DeepSeek's own vision calculator on the sent dimensions, and a file row is the handle the model actually receives. dsh projects every `file` block to `fileHandleText` at request assembly (`dsh-llm/content.ts` — no provider takes a file block natively), and the token meter prices that text in place of the block's structural JSON price (`dsh-token-meter/route-pricing.ts`), so a file costs a fixed handle whatever its byte size. The plugin mirrors that sentence verbatim and prices it with the meter's own text arm (`estimateTextTokens`), from the three facts a log already carries: name, byte size, and the digest's first eight characters. The handle's stored-copy path is execution-world state no log carries (`llm.fileRequestText` resolves it live through the attachment store and the fs mapping), so the card prices the pathless arm — the one the harness emits when the environment cannot read the copy — the floor the path-bearing handle only grows from, and the row therefore reads as a lower bound (`≥`) rather than an `≈`. A reference the log proves no name or no byte size for prices nothing and shows no row, exactly like its size row. The fold keeps the meter's FIXED structural price for that same block, which is what `estimateMessage` charges without a route projection; the card answers the request's question and the fold the surface's, so the two figures differ by design, as they do in the harness. The session jump — a Context Insights session card or an Agent network node bringing its session forward — rides the harness's own selection verb (`openSessionVia` in `src/client/services.ts`): `uiWorkspace.openSession`, the verb the built-in sidebar's row click uses on every supported line. The face is re-proved per call and a deployment serving none (or a hostile face) swallows silently — the verb's own navigation returns the center column to the conversation (`selectPanel(null)`), so a dead jump simply stays on the page. Issue #90 was exactly the retired-spelling trap: the plugin once called the sessions service's own `open(id)`, which the 0.1.6 selection refactor removed from the service (the click then did nothing on `0.1.7-rc.2`), so the jump follows the view owner and the per-baseline `sessionNav` probe pins the verb's signature and the retired spelling's absence. diff --git a/src/client/components/attachments.tsx b/src/client/components/attachments.tsx index 1ca4345..0358bff 100644 --- a/src/client/components/attachments.tsx +++ b/src/client/components/attachments.tsx @@ -10,6 +10,7 @@ import { useCallback, useEffect, useMemo, useRef, useState, type ReactElement } from 'react' import { attachmentExtension } from '../../shared/attachment' import type { AttachmentRef } from '../../shared/attachment' +import { estimateFileTokens } from '../../shared/fileTokens' import { estimateImageTokens } from '../../shared/imageTokens' import { fmtBytes } from '../format' import type { AttachmentOpener, ImageLoader, ImageRefLike } from '../services' @@ -45,6 +46,13 @@ function nameOf(ref: AttachmentRef, t: ViewKit['t']): string { return ref.name ?? t(ref.kind === 'image' ? 'attach.image' : 'attach.file') } +/** One labelled figure of a card. */ +interface Fact { + label: string + value: string + tip?: string +} + /** The type line of an attachment (the harness file card's own rule): the upper-cased extension, * else the attachment's kind. */ function typeOf(ref: AttachmentRef, t: ViewKit['t']): string { @@ -81,15 +89,22 @@ function useImageSrc(attachment: AttachmentRef, load: ImageLoader | undefined): return state } -/** An attachment's facts as label/value rows: an image's dimensions and token estimate, a file's - * size — the same figures the chat and the harness's own file card show. */ -function factsOf(ref: AttachmentRef, t: ViewKit['t'], fmt: ViewKit['fmt']): Array<{ label: string; value: string; tip?: string }> { +/** An attachment's facts as label/value rows: an image's dimensions and its vision price, a file's + * size and the floor its handle costs — the same figures the chat and the harness's own file card show. */ +function factsOf(ref: AttachmentRef, t: ViewKit['t'], fmt: ViewKit['fmt']): Fact[] { const dimsOf = (w: number | undefined, h: number | undefined): string | null => w !== undefined && h !== undefined ? `${w}×${h}` : null if (ref.kind !== 'image') { - return ref.bytes === undefined ? [] : [{ label: t('attach.size'), value: fmtBytes(ref.bytes) }] + const rows: Fact[] = [] + if (ref.bytes !== undefined) rows.push({ label: t('attach.size'), value: fmtBytes(ref.bytes) }) + // A lower bound: the handle also names the stored copy's read path, which no log carries. + const tokens = estimateFileTokens(ref) + if (tokens !== null) { + rows.push({ label: t('attach.token'), value: `≥${fmt(tokens)}`, tip: t('attach.fileTokensTip') }) + } + return rows } - const rows: Array<{ label: string; value: string; tip?: string }> = [] + const rows: Fact[] = [] // No byte size is stored for the raw raster — dims only. const rawDims = ref.originalDimensions !== undefined ? dimsOf(ref.originalDimensions.width, ref.originalDimensions.height) @@ -105,7 +120,9 @@ function factsOf(ref: AttachmentRef, t: ViewKit['t'], fmt: ViewKit['fmt']): Arra const tokens = ref.width !== undefined && ref.height !== undefined ? estimateImageTokens(ref.width, ref.height) : null - if (tokens !== null) rows.push({ label: t('attach.token'), value: `≈${fmt(tokens)}`, tip: t('attach.tokensTip') }) + if (tokens !== null) { + rows.push({ label: t('attach.token'), value: `≈${fmt(tokens)}`, tip: t('attach.imageTokensTip') }) + } return rows } diff --git a/src/client/i18nEn.ts b/src/client/i18nEn.ts index 6452cbf..3945402 100644 --- a/src/client/i18nEn.ts +++ b/src/client/i18nEn.ts @@ -447,7 +447,8 @@ export const DICT_EN = { 'attach.raw': 'Raw', 'attach.sent': 'Sent', 'attach.token': 'Token', - 'attach.tokensTip': 'Estimated token cost via DeepSeek\'s official image-size conversion (384-token cap per image)', + 'attach.imageTokensTip': 'Estimated token cost via DeepSeek\'s official image-size conversion (384-token cap per image)', + 'attach.fileTokensTip': 'A lower bound: the model receives a handle naming this file, its size, its digest, and where the stored copy can be read — never its contents — so the cost barely moves with the file\'s size', 'ov.title': 'Context Insights', 'ov.unavailable': 'The session list is unavailable', 'ov.range.label': 'Range', diff --git a/src/client/i18nZh.ts b/src/client/i18nZh.ts index dbfebfb..0190312 100644 --- a/src/client/i18nZh.ts +++ b/src/client/i18nZh.ts @@ -446,7 +446,8 @@ export const DICT_ZH = { 'attach.raw': '原图', 'attach.sent': '发送', 'attach.token': 'Token', - 'attach.tokensTip': '按 DeepSeek 官方图片尺寸换算(单图上限 384 tokens)估算的 token 消耗', + 'attach.imageTokensTip': '按 DeepSeek 官方图片尺寸换算(单图上限 384 tokens)估算的 token 消耗', + 'attach.fileTokensTip': '这是一个下限:模型收到的是一个句柄(文件名、大小、摘要前缀,以及副本的读取路径),不会收到文件内容,所以开销几乎不随文件大小变化', 'ov.title': '上下文洞察', 'ov.unavailable': '会话列表暂不可用', 'ov.range.label': '统计范围', diff --git a/src/host/pricing.ts b/src/host/pricing.ts index 6f763cd..9ebf7d6 100644 --- a/src/host/pricing.ts +++ b/src/host/pricing.ts @@ -6,6 +6,7 @@ * (shared/imageTokens.ts), falling back to the meter's JSON price. */ +import { estimateTextTokens } from '../shared/estimate' import { estimateImageTokens } from '../shared/imageTokens' const CHARS_PER_TOKEN = 4 @@ -42,7 +43,7 @@ function estimateBlocks(blocks: unknown): number { switch (block.type) { case 'text': case 'reasoning': - tokens += Math.ceil((block.text || '').length / CHARS_PER_TOKEN) + BLOCK_OVERHEAD + tokens += estimateTextTokens(block.text || '') break case 'tool-call': tokens += Math.ceil((block.name || '').length / CHARS_PER_TOKEN) diff --git a/src/shared/estimate.ts b/src/shared/estimate.ts index 4c11ad8..6a10c22 100644 --- a/src/shared/estimate.ts +++ b/src/shared/estimate.ts @@ -1,14 +1,21 @@ /** Token heuristics shared by the host fold and the client boundary, mirroring the harness - * token-meter's fixed density (4 chars ≈ 1 token, +4 role framing) on both sides. */ + * token-meter's fixed density (4 chars ≈ 1 token, +4 per content block, +4 role framing) on both + * sides. */ const CHARS_PER_TOKEN = 4 const ROLE_OVERHEAD = 4 +const BLOCK_OVERHEAD = 4 export function estimateSystemTokens(text: unknown): number { if (typeof text !== 'string' || text.length === 0) return 0 return Math.ceil(text.length / CHARS_PER_TOKEN) + ROLE_OVERHEAD } +/** Price one text block exactly like the harness meter's text arm. */ +export function estimateTextTokens(text: string): number { + return Math.ceil(text.length / CHARS_PER_TOKEN) + BLOCK_OVERHEAD +} + /** Price a `system/message` payload exactly like the harness's `estimateSystemMessage`: text * density over every block plus role framing, no per-block overhead. */ export function estimateSystemContent(blocks: unknown): number { diff --git a/src/shared/fileTokens.ts b/src/shared/fileTokens.ts new file mode 100644 index 0000000..a629a7e --- /dev/null +++ b/src/shared/fileTokens.ts @@ -0,0 +1,27 @@ +/** + * Per-file token estimate for a durable file attachment — a file's BYTES never reach a model. + * + * dsh projects every `file` block to `dsh-llm`'s `fileHandleText` at request assembly and the token + * meter prices that text in place of the block's structural JSON price, so a file costs its handle + * whatever its size. The handle names the file, its byte size, and its digest prefix — the facts a + * log carries. It also names the stored copy's read path, which is execution-world state no log + * holds (`llm.fileRequestText` resolves it through the live attachment store and fs mapping), so + * this prices the pathless arm the harness emits when the copy is unreadable: the FLOOR the + * path-bearing handle grows from, which is why the card shows it as a lower bound. + */ + +import { estimateTextTokens } from './estimate' + +const DIGEST_PREFIX_LENGTH = 'sha256:'.length +const DIGEST_SHOWN = 8 + +/** One file's handle price, or null when the log proves neither the name nor the byte size the + * handle's identity is built from. */ +export function estimateFileTokens(ref: { attachmentId: string; name?: string; bytes?: number }): number | null { + if (ref.name === undefined || ref.bytes === undefined) return null + const digest = ref.attachmentId.slice(DIGEST_PREFIX_LENGTH, DIGEST_PREFIX_LENGTH + DIGEST_SHOWN) + const identity = `File ${JSON.stringify(ref.name)} (${ref.bytes} bytes, sha256:${digest})` + const handle = `[${identity} was uploaded, but the current execution environment cannot access a` + + ' readable path. Report that limitation if its contents are needed; do not claim to have read it.]' + return estimateTextTokens(handle) +} diff --git a/tests/client/components/attachments.spec.ts b/tests/client/components/attachments.spec.ts index 6d9570a..ff9de26 100644 --- a/tests/client/components/attachments.spec.ts +++ b/tests/client/components/attachments.spec.ts @@ -56,9 +56,21 @@ describe('the attachment block', () => { assert.ok(text(imageCard).includes('3462×1211'), 'raw dimensions') assert.ok(text(imageCard).includes('1568×1350 · 146.2 kB'), 'sent dimensions and size') assert.ok(text(imageCard).includes('Token') && text(imageCard).includes('≈325')) - // A file card keeps only its size: the extension badge already names the type. + // A file card keeps its size and the floor its handle costs: the extension badge already names the type. assert.ok(!text(cards[0]).includes('Type')) assert.ok(text(cards[0]).includes('Size') && text(cards[0]).includes('2.6 kB')) + assert.ok(text(cards[0]).includes('Token') && text(cards[0]).includes('≥58'), 'the handle floor') + assert.ok( + text(cards[1]).includes('≥56') && text(cards[2]).includes('≥56'), + 'a 778 kB docx costs no more than a 47 kB markdown: only the handle travels', + ) + // The two token rows read differently on purpose: a vision price, and a lower bound. + const fileTip = queryAll(cards[0], '.lc-att-row').pop()?.getAttribute('title') ?? '' + assert.ok(fileTip.includes('lower bound') && fileTip.includes('never its contents'), fileTip) + assert.ok( + (queryAll(cards[3], '.lc-att-row').pop()?.getAttribute('title') ?? '').includes('image-size conversion'), + 'the image row keeps the vision calculator as its explanation', + ) assert.ok(text(cards[3]).includes('Type'), 'an image has no badge, so its type keeps the row') // Every card invites the same one click. assert.ok(queryAll(m.container, '.lc-att-hint').every(hint => text(hint).includes('Click to preview'))) @@ -78,12 +90,14 @@ describe('the attachment block', () => { await bare.unmount() }) - test('an unnamed file names its kind on the badge, and shows no size row', async () => { + test('an unnamed file names its kind on the badge, and shows no size or token row', async () => { const m = await mountSection({ attachments: [{ kind: 'file', attachmentId: 'sha256:u' }] }) const card = query(m.container, '.lc-att-item') assert.ok(text(query(card, '.lc-att-name')).includes('File')) assert.equal(text(query(card, '.lc-att-ext')), 'File', 'the kind stands in for a missing extension') assert.ok(!text(card).includes('Size')) + // The handle names the file and its size, so an unproven reference prices nothing. + assert.ok(!text(card).includes('Token')) await m.unmount() }) diff --git a/tests/shared/fileTokens.spec.ts b/tests/shared/fileTokens.spec.ts new file mode 100644 index 0000000..df5f3f5 --- /dev/null +++ b/tests/shared/fileTokens.spec.ts @@ -0,0 +1,47 @@ +// The per-file attachment price (src/shared/fileTokens.ts): what a durable file costs in a request, +// which is the harness's model-facing handle and never the file's bytes. + +import assert from 'node:assert/strict' +import { describe, test } from 'vitest' +import { estimateFileTokens } from '../../src/shared/fileTokens' + +const DIGEST = 'sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef' + +function file(over: { name?: string; bytes?: number; attachmentId?: string } = {}) { + return { attachmentId: DIGEST, name: 'notes.md', bytes: 120, ...over } +} + +describe('estimateFileTokens', () => { + test('the price is the harness handle priced as one text block', () => { + // The handle the harness emits when the environment cannot read the stored copy, verbatim. + const handle = '[File "notes.md" (120 bytes, sha256:01234567) was uploaded, but the current execution' + + ' environment cannot access a readable path. Report that limitation if its contents are needed;' + + ' do not claim to have read it.]' + assert.equal(estimateFileTokens(file()), Math.ceil(handle.length / 4) + 4) + assert.equal(estimateFileTokens(file()), 57) + }) + + test('a 778 kB file costs one token more than a 120 B one: the bytes are never sent', () => { + assert.equal(estimateFileTokens(file({ bytes: 778_285 })), 58) + }) + + test('only the digest prefix reaches the handle', () => { + assert.equal(estimateFileTokens(file({ attachmentId: `sha256:01234567${'0'.repeat(56)}` })), 57) + assert.equal(estimateFileTokens(file({ attachmentId: `sha256:${'f'.repeat(64)}` })), 57) + }) + + test('the price is a floor: the arm that names the stored copy costs more', () => { + // The other harness arm appends the copy's read path, which only the live environment resolves. + const pathArm = ' verbatim read-only copy saved at "/home/u/.dsh/attachments/v1/files/ab/0123456789abcdef' + + '/notes.md". Read that path with your file tools when its contents are needed; copy it to a' + + ' writable location before modifying it. When delegating file work, include this saved path in the' + + ' delegation prompt; only subagents sharing this execution environment can read it.' + assert.ok(Math.ceil((400 + pathArm.length) / 4) + 4 > (estimateFileTokens(file()) as number)) + }) + + test('a name or a byte size the log does not prove prices nothing', () => { + assert.equal(estimateFileTokens(file({ name: undefined })), null) + assert.equal(estimateFileTokens(file({ bytes: undefined })), null) + assert.equal(estimateFileTokens(file({ name: undefined, bytes: undefined })), null) + }) +}) \ No newline at end of file