redact: a value masked once is masked again wherever a request has it, whatever words are around it

The masking rules find a value by what is around it: DB_PASSWORD= before
a password, a bound an email may not touch (so an email that ends a
sentence isn't one). The vendor writes about a placeholder in words of
its own ("your password is {{SECRET_…}}", "I will write to {{EMAIL_…}}."),
the agent reads the value there (Restore), and on the next turn that text
is in the history, where no rule matches it, so it went out as written.

Now every value a placeholder was made for (redact's in-memory values,
bounded by maxValues, never written anywhere) is also matched as an exact
string in each string Mask and MaskJSON look at, and goes as the same
placeholder it had (same kind, same keyed hash), so history and vendor
caches stay stable:

- only from 8 characters (the assignment rule's minimum): the 3-character
  passwords the user:password@ rule takes, and the user's own words
  (which Options.Words matches everywhere already), are not;
- personal values only while personal masking is on, secrets only while
  secret masking is on;
- not as a piece of a longer value: a phone or card number with a digit
  beside it, an email inside a longer address (a full stop after it may
  end the sentence), a secret with a letter or digit against it;
- where a rule finds the same span, the rule's kind still wins, so
  nothing masked before changes placeholder;
- in what a vendor signed (7137feb4), the values masked elsewhere in the
  request go back as their placeholders as before, and so do known
  secrets, for a turn whose other copies were compacted away; known
  personal values don't, since a number the vendor wrote of its own
  (13800138000) may be one masked in another conversation, and masking
  it would break the signature.

A prefix index and a 4-byte bitset keep the lookup cheap: a ~1 MB body
masked in ~79 ms with 0, 1k or 200k known values alike (rules dominate).
Every route goes through redact.MaskJSON / Mask (chat, Responses,
Anthropic, Gemini, count_tokens, embeddings/rerank, the Codex backend,
image and video prompts), so each gets it; Scrub, the archive's, gets the
known secrets too.

Tests: TestEchoedValueMaskedNextTurn (gateway: two chat turns where the
fake vendor echoes the placeholders in other words; the second request
must not hold the password or the email, and keeps the first turn's
placeholders; then the echoed text alone through /v1/responses,
/v1/messages, count_tokens, Gemini generateContent, embeddings and the
Codex backend) fails on origin/main with the second turn sending
"…password is Mv4kestrelPass28, and I will write to
tomas.ng@quietfield.dev.". TestKnownValueMaskedInOtherWords and
TestKnownInSigned fail there too; TestShortValueNotMaskedEverywhere and
TestKnownReindexed pin the bounds and the eviction rebuild.

Run: gofmt -l (nothing); go vet -tags nogui ./... on darwin,
GOOS=linux and GOOS=windows; go build -tags nogui for darwin, linux,
windows; go test -tags nogui ./... under a temp HOME with pinned caches
(all ok); internal/redact -count=3 and -race. Not tried against a real
vendor.

docs/subsystems/gateway-routing.md

@@ -30,7 +30,7 @@ served by a plugin are in [Provider and plugin ownership](provider-plugins.md).
 ## Runtime path
 
 1. `handle(proto)` reads the body and the model (`requestBody`, `requestModel`), then calls `serveAgent`. Every route that reads an agent's body (each API, count_tokens, Gemini's, embeddings, images, videos, System One, the Codex backend path) reads it through `readBoundedRequestBody` in [`request_bounds.go`](../../internal/gateway/request_bounds.go). It decodes a `Content-Encoding` of `gzip` or `zstd` (`decodedBody`; Codex sends zstd to a custom provider's `/v1/responses` too, #1223) and drops the header, so the provider is sent plain JSON. The size limit holds for the decoded bytes (413), and any other encoding is refused 415. `keyLimited` reads a limited key's compressed body the same way, so its reservation is the decoded size. `serveAgent` runs the [middleware](gateway-middleware.md) chain, then `serve`.
-2. `serve` redacts secrets in the request (`redacted`; `redact.MaskJSON` leaves ids alone, and what the vendor sealed with a signature, a thinking block, Gemini's thoughts or Responses' encrypted reasoning, goes back as the vendor wrote it: only the values masked elsewhere in the request are put back as their placeholders), resolves the model to a provider or a group, and refuses what can't be served. A request whose provider and fallbacks, or whose group's models, all skip redaction (`unredactedRoute`, `provider.SkipsRedaction`: set *Send requests unmasked*, `Unredacted`, with every address on this machine or the local network) goes to them as the agent wrote it; the log and recent calls keep the masked body. count_tokens and embeddings follow the same rule.
+2. `serve` redacts secrets in the request (`redacted`; `redact.MaskJSON` leaves ids alone, and what the vendor sealed with a signature, a thinking block, Gemini's thoughts or Responses' encrypted reasoning, goes back as the vendor wrote it: only the values masked elsewhere in the request, and secrets masked before, are put back as their placeholders; a value masked once, at least 8 characters long, is masked again wherever a request has it, as an exact match whatever words are around it, so what the vendor wrote about a placeholder doesn't go back with the value in it, `redact/known.go`), resolves the model to a provider or a group, and refuses what can't be served. A request whose provider and fallbacks, or whose group's models, all skip redaction (`unredactedRoute`, `provider.SkipsRedaction`: set *Send requests unmasked*, `Unredacted`, with every address on this machine or the local network) goes to them as the agent wrote it; the log and recent calls keep the masked body. count_tokens and embeddings follow the same rule.
    - `DecidesModel` gives a 400.
    - A gateway key held to some models (`keyHolds`, #882) gets a 403 for any other model.
    - Sealed tasks (#619) are checked here. A Codex subagent's task its lead's `spawn_agent` sealed goes only to a ChatGPT account, or back to the Responses provider that answered the lead (`leadProvider`, a relay of the ChatGPT backend such as Sub2API, #1109), relayed as it came even when web search is offered. A lead's turn carrying its subagent's sealed reply (`send_message`) goes back to the Responses provider that answered the lead's own conversation (`leadProvider` of the request's `conversationID`, #1237); otherwise it is refused before anyone is asked (`sealedReader`, `sealedTaskError`). Who answered is first what affinity remembers; when affinity has let it go (a day since, a restart past the 512 conversations `affinity.json` keeps, or the lead's last reply broken off, `unanswered`), the usage log's answers to the lead's thread and the request's own conversation in the last 30 days count too (`loggedAnswers`, #1367), and `leadFirst` reads the lead's key or account from them. A refused or failed call and a title answer nothing, and Codex's own sign-in is told of as the ChatGPT backend. The refusal names who sealed the task when it is known, and says how to go on with it.

internal/gateway/redact_known_test.go

@@ -0,0 +1,150 @@
+package gateway
+
+import (
+	"encoding/json"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"regexp"
+	"strings"
+	"sync"
+	"testing"
+
+	"github.com/yetone/magpie/internal/provider"
+	"github.com/yetone/magpie/internal/settings"
+)
+
+// The vendor answers about a placeholder in words of its own, the agent
+// reads the value there, and the next turn's history has it in words no
+// masking rule knows: it goes as the same placeholder again, on every
+// route a request can take.
+func TestEchoedValueMaskedNextTurn(t *testing.T) {
+	t.Setenv("XDG_CONFIG_HOME", t.TempDir())
+	t.Setenv("XDG_CACHE_HOME", t.TempDir())
+	const secret, email = "Mv4kestrelPass28", "tomas.ng@quietfield.dev"
+	placeholder := regexp.MustCompile(`\{\{[A-Z_]+_[a-z2-7]{8}\}\}`)
+	var mu sync.Mutex
+	var sent []string
+	up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		b, _ := io.ReadAll(r.Body)
+		mu.Lock()
+		sent = append(sent, string(b))
+		mu.Unlock()
+		if !strings.HasSuffix(r.URL.Path, "/chat/completions") {
+			w.Header().Set("Content-Type", "application/json")
+			w.WriteHeader(400)
+			io.WriteString(w, `{"error":{"message":"not here","type":"invalid_request_error"}}`)
+			return
+		}
+		// what it was given, in its own words: no DB_PASSWORD= before the
+		// one, a full stop after the other
+		var secretP, emailP string
+		for _, p := range placeholder.FindAllString(string(b), -1) {
+			switch {
+			case strings.HasPrefix(p, "{{SECRET_"):
+				secretP = p
+			case strings.HasPrefix(p, "{{EMAIL_"):
+				emailP = p
+			}
+		}
+		reply := "Noted: your database password is " + secretP + ", and I will write to " + emailP + "."
+		w.Header().Set("Content-Type", "application/json")
+		json.NewEncoder(w).Encode(map[string]any{
+			"id": "c1", "object": "chat.completion", "model": "m1",
+			"choices": []any{map[string]any{"index": 0, "message": map[string]any{"role": "assistant", "content": reply}, "finish_reason": "stop"}},
+			"usage":   map[string]any{"prompt_tokens": 3, "completion_tokens": 1},
+		})
+	}))
+	defer up.Close()
+	if err := provider.Save(provider.Provider{ID: "fake", Name: "Fake", Key: "k", Models: []string{"m1"}, Chat: up.URL + "/v1", Anthropic: up.URL}); err != nil {
+		t.Fatal(err)
+	}
+	if err := settings.Save(settings.Settings{Redact: true, RedactPersonal: true}); err != nil {
+		t.Fatal(err)
+	}
+	leaked := func(b string) bool { return strings.Contains(b, secret) || strings.Contains(b, email) }
+
+	user := `{"role":"user","content":"DB_PASSWORD=` + secret + ` and I am ` + email + `"}`
+	code, body := post(t, "/v1/chat/completions", `{"model":"fake/m1","messages":[`+user+`]}`)
+	if code != 200 || len(sent) != 1 || leaked(sent[0]) {
+		t.Fatalf("first turn: %d %s\nsent: %v", code, body, sent)
+	}
+	var first struct {
+		Choices []struct {
+			Message struct {
+				Content string `json:"content"`
+			} `json:"message"`
+		} `json:"choices"`
+	}
+	json.Unmarshal([]byte(body), &first)
+	if len(first.Choices) == 0 || !strings.Contains(first.Choices[0].Message.Content, secret) || !strings.Contains(first.Choices[0].Message.Content, email) {
+		t.Fatalf("the agent should read its values back: %s", body)
+	}
+	echo := first.Choices[0].Message.Content
+	asst, _ := json.Marshal(map[string]any{"role": "assistant", "content": echo})
+	code, body = post(t, "/v1/chat/completions", `{"model":"fake/m1","messages":[`+user+`,`+string(asst)+`,{"role":"user","content":"go on"}]}`)
+	if code != 200 || len(sent) != 2 {
+		t.Fatalf("second turn: %d %s", code, body)
+	}
+	if leaked(sent[1]) {
+		t.Fatalf("the second turn sent a value to the vendor: %s", sent[1])
+	}
+	// the same placeholders as the first turn, so what the vendor cached
+	// of it holds
+	for _, p := range placeholder.FindAllString(sent[0], -1) {
+		if strings.Count(sent[1], p) != 2 {
+			t.Fatalf("%s not in the second turn twice: %s", p, sent[1])
+		}
+	}
+
+	// the reply alone, on every other route a request can take
+	e, _ := json.Marshal(echo)
+	routes := []struct{ name, path, body string }{
+		{"responses", "/v1/responses", `{"model":"fake/m1","input":` + string(e) + `}`},
+		{"anthropic", "/v1/messages", `{"model":"fake/m1","max_tokens":10,"messages":[{"role":"assistant","content":` + string(e) + `},{"role":"user","content":"go on"}]}`},
+		{"count_tokens", "/v1/messages/count_tokens", `{"model":"fake/m1","messages":[{"role":"user","content":` + string(e) + `}]}`},
+		{"gemini", "/v1beta/models/fake/m1:generateContent", `{"contents":[{"role":"model","parts":[{"text":` + string(e) + `}]},{"role":"user","parts":[{"text":"go on"}]}]}`},
+		{"embeddings", "/v1/embeddings", `{"model":"fake/m1","input":` + string(e) + `}`},
+	}
+	for _, r := range routes {
+		mu.Lock()
+		before := len(sent)
+		mu.Unlock()
+		post(t, r.path, r.body)
+		mu.Lock()
+		got := append([]string(nil), sent[before:]...)
+		mu.Unlock()
+		if len(got) == 0 {
+			t.Errorf("%s: nothing reached the vendor", r.name)
+		}
+		for _, b := range got {
+			if leaked(b) {
+				t.Errorf("%s: a value went to the vendor: %s", r.name, b)
+			}
+		}
+	}
+
+	// and Codex's own backend
+	var codex []string
+	chatgpt(t, func(w http.ResponseWriter, r *http.Request) {
+		b, _ := io.ReadAll(r.Body)
+		codex = append(codex, string(b))
+		w.Header()["Content-Type"] = nil
+		io.WriteString(w, sse(
+			`data: {"type":"response.created","response":{"id":"r1"}}`,
+			`data: {"type":"response.completed","response":{"id":"r1","usage":{"input_tokens":9,"output_tokens":2}}}`))
+	})
+	t.Setenv("XDG_CONFIG_HOME", t.TempDir())
+	if err := settings.Save(settings.Settings{Redact: true, RedactPersonal: true}); err != nil {
+		t.Fatal(err)
+	}
+	codexPost(t, `{"model":"gpt-5.5","stream":true,"input":[{"type":"message","role":"assistant","content":[{"type":"output_text","text":`+string(e)+`}]}]}`)
+	if len(codex) == 0 {
+		t.Fatal("codex: nothing reached the backend")
+	}
+	for _, b := range codex {
+		if leaked(b) {
+			t.Errorf("codex: a value went to the backend: %s", b)
+		}
+	}
+}

internal/redact/json.go

@@ -284,8 +284,9 @@ func jsonEscape(s string) string {
 // thoughts, Responses' encrypted reasoning) is checked by the vendor
 // against the text it wrote, so it goes back as written: the values
 // masked elsewhere in the request become their placeholders again, which
-// is what the vendor wrote where the agent reads the value (Restore), and
-// nothing else in it is touched. Masked by the rules instead, a value the
+// is what the vendor wrote where the agent reads the value (Restore), as
+// do the secrets masked before (knownSpans), and nothing else in it is
+// touched. Masked by the rules instead, a value the
 // vendor wrote as a placeholder in other words around it ("the password
 // is {{SECRET_…}}") went back as the value itself, and what the vendor
 // wrote of its own that looks like an email or a phone number went as a
@@ -312,18 +313,31 @@ func MaskJSON(body []byte, o Options) ([]byte, int) {
 		t, n := Mask(string(body), o)
 		return []byte(t), n
 	}
-	if len(signed) == 0 || len(seen) == 0 {
+	if len(signed) == 0 {
 		return out, total
 	}
-	remask := replacerOf(seen)
+	var remask *strings.Replacer
+	if len(seen) > 0 {
+		remask = replacerOf(seen)
+	}
 	out, _ = walk(out, func(path, key, s string) string {
 		if keepIn(path, key, s, args) || !under(path, signed) {
 			return s
 		}
-		t := remask.Replace(s)
-		if t != s {
-			total++
+		t := s
+		if remask != nil {
+			if t = remask.Replace(s); t != s {
+				total++
+			}
 		}
+		// and a secret masked before, though nothing else in this request
+		// has it now (the turn that brought it was compacted away): the
+		// vendor wrote its placeholder there too. Not personal data: a
+		// number or an address the vendor wrote of its own can be one
+		// masked in another conversation (13800138000 is in many), and
+		// masked, its text would no longer match its signature
+		t, n := apply(t, knownSpans(t, Options{Secrets: o.Secrets}), placeholder)
+		total += n
 		return t
 	})
 	return out, total

internal/redact/known.go

@@ -0,0 +1,146 @@
+package redact
+
+import "strings"
+
+// A value masked once is masked again wherever it is in a request, as it
+// is, whatever is around it. The rules find a value by what is around it
+// (DB_PASSWORD=, a bound an email may not touch), and the vendor answers
+// about a placeholder in words of its own: "the password is {{SECRET_…}}",
+// "write to {{EMAIL_…}}.". The agent reads the value there (Restore), and
+// on the next turn that text comes back in the history, where no rule
+// matches it. Matched as known, it goes as the same placeholder again.
+//
+// What is known is what values holds (in memory only, bounded by
+// maxValues): the values of the placeholders made since magpie started,
+// for every request alike, since a placeholder is the same for the same
+// value whichever conversation it is in.
+
+// minKnown is the shortest value matched as known. Shorter ones (a
+// password of three letters in a URL, which the rule takes from its
+// user:password@) would turn up all over ordinary text; 8 is what the rule
+// for a password in an assignment asks of its value.
+const minKnown = 8
+
+type knownValue struct{ v, p, kind string }
+
+var (
+	// known are the values matched as known, by their first minKnown
+	// bytes; guarded by mu, as values is
+	known = map[string][]knownValue{}
+	// knownBits has a bit for the first four bytes of each known value, so
+	// most places in a text are passed over without a map lookup
+	knownBits [1 << 14]uint64
+)
+
+func gram(s string, i int) uint32 {
+	g := uint32(s[i]) | uint32(s[i+1])<<8 | uint32(s[i+2])<<16 | uint32(s[i+3])<<24
+	return (g * 2654435761) >> 12 // 20 bits
+}
+
+// addKnown adds the value of placeholder p; called with mu held. The
+// user's own words are left to Options.Words, which match them anywhere
+// already, and are only as long as two letters.
+func addKnown(p, v string) {
+	kind := kindOf(p)
+	if kind == "TERM" || len(v) < minKnown {
+		return
+	}
+	head := v[:minKnown]
+	for i, k := range known[head] {
+		if k.v == v {
+			// one value under two kinds: the same one wins whichever came
+			// first, so it is the same after reindexKnown
+			if p < k.p {
+				known[head][i] = knownValue{v, p, kind}
+			}
+			return
+		}
+	}
+	known[head] = append(known[head], knownValue{v, p, kind})
+	g := gram(v, 0)
+	knownBits[g/64] |= 1 << (g % 64)
+}
+
+// reindexKnown makes known again from values, after some were let go;
+// called with mu held.
+func reindexKnown() {
+	known = map[string][]knownValue{}
+	knownBits = [1 << 14]uint64{}
+	for p, v := range values {
+		addKnown(p, v)
+	}
+}
+
+// kindOf is the kind in a placeholder: SECRET of {{SECRET_abcdefgh}}.
+func kindOf(p string) string {
+	if i := strings.LastIndexByte(p, '_'); i > 2 {
+		return p[2:i]
+	}
+	return ""
+}
+
+var personalKinds = map[string]bool{"EMAIL": true, "ID_CARD": true, "PHONE": true, "BANK_CARD": true}
+
+// knownSpans are where s has a known value o masks: a personal one when o
+// masks personal data, any other when it masks secrets. Where two start
+// at the same place, the longer.
+func knownSpans(s string, o Options) []span {
+	if !o.Secrets && !o.Personal || len(s) < minKnown {
+		return nil
+	}
+	mu.RLock()
+	defer mu.RUnlock()
+	if len(known) == 0 {
+		return nil
+	}
+	var out []span
+	for i := 0; i+minKnown <= len(s); i++ {
+		if g := gram(s, i); knownBits[g/64]&(1<<(g%64)) == 0 {
+			continue
+		}
+		var best *knownValue
+		for j, k := range known[s[i:i+minKnown]] {
+			if personalKinds[k.kind] && !o.Personal || !personalKinds[k.kind] && !o.Secrets {
+				continue
+			}
+			if best != nil && len(k.v) <= len(best.v) || !strings.HasPrefix(s[i:], k.v) || !standsAlone(k.kind, s, i, i+len(k.v)) {
+				continue
+			}
+			best = &known[s[i:i+minKnown]][j]
+		}
+		if best != nil {
+			out = append(out, span{i, i + len(best.v), best.kind})
+			i += len(best.v) - 1
+		}
+	}
+	return out
+}
+
+// standsAlone says s[a:b] is the known value itself, not a piece of a
+// longer one: a number with no digit beside it, an email that is not part
+// of a longer address (though a full stop may end the sentence after it),
+// a secret with no letter or digit against it (twelve hex digits a rule of
+// the user's took are also in the middle of a hash).
+func standsAlone(kind, s string, a, b int) bool {
+	before, after := byte(0), byte(0)
+	if a > 0 {
+		before = s[a-1]
+	}
+	if b < len(s) {
+		after = s[b]
+	}
+	is := func(c byte, set string) bool { return c != 0 && strings.IndexByte(set, c) >= 0 }
+	switch kind {
+	case "PHONE", "BANK_CARD":
+		return !is(before, digits) && !is(after, digits)
+	case "ID_CARD":
+		return !is(before, alnum) && !is(after, alnum)
+	case "EMAIL":
+		if is(before, alnum+"._%+-") || is(after, alnum+"_%+") {
+			return false
+		}
+		// a.b@c.com.cn is another address; a.b@c.com. ends a sentence
+		return !is(after, ".-") || b+1 >= len(s) || !is(s[b+1], alnum)
+	}
+	return !(is(before, alnum) && is(s[a], alnum)) && !(is(after, alnum) && is(s[b-1], alnum))
+}

internal/redact/known_test.go

@@ -0,0 +1,103 @@
+package redact
+
+import (
+	"strings"
+	"testing"
+)
+
+// A value masked once goes as its placeholder wherever it is next, in
+// whatever words are around it: what the vendor wrote about it, as the
+// agent sends it back.
+func TestKnownValueMaskedInOtherWords(t *testing.T) {
+	o := Options{Secrets: true, Personal: true}
+	const secret, email, phone = "Zq8hunterPass42", "rina.k@acme-labs.io", "13912345670"
+	ps, n := Mask("DB_PASSWORD="+secret, o)
+	pe, _ := Mask("I am "+email, o)
+	pp, _ := Mask("call "+phone, o)
+	if n != 1 || strings.Contains(ps+pe+pp, secret) || strings.Contains(pe, email) || strings.Contains(pp, phone) {
+		t.Fatalf("not masked to begin with: %q %q %q", ps, pe, pp)
+	}
+	ps, pe, pp = strings.TrimPrefix(ps, "DB_PASSWORD="), strings.TrimPrefix(pe, "I am "), strings.TrimPrefix(pp, "call ")
+
+	// the vendor's reply as the agent has it back: no rule matches these
+	reply := "Your password is " + secret + ", and I will write to " + email + ". Or call " + phone + "!"
+	out, n := Mask(reply, o)
+	if strings.Contains(out, secret) || strings.Contains(out, email) || strings.Contains(out, phone) {
+		t.Fatalf("a known value went out: %q", out)
+	}
+	if want := "Your password is " + ps + ", and I will write to " + pe + ". Or call " + pp + "!"; out != want || n != 3 {
+		t.Fatalf("%d masked: %q, want the same placeholders as before: %q", n, out, want)
+	}
+	if back := Restore(out, false); back != reply {
+		t.Fatalf("round trip: %q", back)
+	}
+
+	// in a request body, as the next turn's history has it
+	body := `{"messages":[{"role":"assistant","content":"` + reply + `"},{"role":"user","content":"thanks"}]}`
+	if got, n := MaskJSON([]byte(body), o); strings.Contains(string(got), secret) || strings.Contains(string(got), email) || strings.Contains(string(got), phone) || n != 3 {
+		t.Fatalf("%d masked: %s", n, got)
+	}
+
+	// only what the settings mask: personal data off, secrets off
+	if out, _ := Mask(reply, Options{Secrets: true}); !strings.Contains(out, email) || !strings.Contains(out, phone) || strings.Contains(out, secret) {
+		t.Fatalf("personal data off: %q", out)
+	}
+	if out, _ := Mask(reply, Options{Personal: true}); !strings.Contains(out, secret) || strings.Contains(out, email) {
+		t.Fatalf("secrets off: %q", out)
+	}
+
+	// a piece of a longer number or address is another one
+	for _, s := range []string{"order 2" + phone, "order " + phone + "9", "mail " + email + ".cn now", "mail x" + email} {
+		if out, _ := Mask(s, Options{Personal: true}); strings.Contains(out, pp) || strings.Contains(out, pe) {
+			t.Errorf("%q: masked as the known value: %q", s, out)
+		}
+	}
+	// a secret in other punctuation is the secret still; inside a longer
+	// word it is another word
+	if out, _ := Mask("`"+secret+"`/"+secret+"_v2", Options{Secrets: true}); strings.Contains(out, secret) {
+		t.Errorf("secret left: %q", out)
+	}
+	if out, _ := Mask("pass"+secret+"x", Options{Secrets: true}); out != "pass"+secret+"x" {
+		t.Errorf("a longer word masked: %q", out)
+	}
+}
+
+// A value too short to be told from ordinary text is masked where a rule
+// finds it, and nowhere else.
+func TestShortValueNotMaskedEverywhere(t *testing.T) {
+	o := Options{Secrets: true}
+	if out, n := Mask("postgres://app:ab12cd@db/app", o); n != 1 || strings.Contains(out, "ab12cd") {
+		t.Fatalf("the rule should mask it: %q", out)
+	}
+	if out, n := Mask("hex ab12cd0 and ab12cd", o); n != 0 || out != "hex ab12cd0 and ab12cd" {
+		t.Fatalf("a short value masked everywhere: %q", out)
+	}
+}
+
+// What a vendor sealed has a secret masked before as its placeholder, the
+// way the vendor wrote it, though nothing else in the request has it; a
+// number or address of its own it keeps, though one masked before.
+func TestKnownInSigned(t *testing.T) {
+	o := Options{Secrets: true, Personal: true}
+	const secret, phone = "Wv7otterPass93", "13612345674"
+	Mask("DB_PASSWORD="+secret, o)
+	Mask("call "+phone, o)
+	body := `{"messages":[{"role":"assistant","content":[{"type":"thinking","thinking":"the password is ` + secret + `; order ` + phone + `","signature":"Eqk1"}]}]}`
+	out, _ := MaskJSON([]byte(body), o)
+	if strings.Contains(string(out), secret) || !strings.Contains(string(out), "order "+phone) {
+		t.Fatalf("%s", out)
+	}
+}
+
+// After values are let go, the ones still held are matched as before.
+func TestKnownReindexed(t *testing.T) {
+	o := Options{Secrets: true}
+	const secret = "Kp3heronPass71"
+	p, _ := Mask("API_TOKEN="+secret, o)
+	mu.Lock()
+	reindexKnown()
+	mu.Unlock()
+	if out, _ := Mask("it is "+secret, o); out != "it is "+strings.TrimPrefix(p, "API_TOKEN=") {
+		t.Fatalf("%q", out)
+	}
+}

internal/redact/redact.go

@@ -11,7 +11,9 @@
 // matches the text it was made for — and one placeholder is never two
 // values. The values are held in memory only, by placeholder, as requests
 // bring them; a restart loses nothing, since the next request brings them
-// again.
+// again. A value once masked is masked wherever a request has it, whatever
+// words are around it (known.go): the vendor writes about a placeholder in
+// words no rule knows, and the agent sends that text back with the value.
 package redact
 
 import (
@@ -263,8 +265,10 @@ func placeholder(kind, v string) string {
 					break
 				}
 			}
+			reindexKnown()
 		}
 		values[p] = v
+		addKnown(p, v)
 	}
 	mu.Unlock()
 	return p
@@ -329,6 +333,15 @@ func mask(s string, o Options, put func(kind, v string) string) (string, int) {
 			found = append(found, span{a, b, r.kind})
 		}
 	}
+	// after the rules', so where a rule finds the same value it keeps the
+	// kind it has always had
+	found = append(found, knownSpans(s, o)...)
+	return apply(s, found, put)
+}
+
+// apply swaps each of found in s for what put makes of its value, the
+// first and longest where they overlap, nothing inside a placeholder.
+func apply(s string, found []span, put func(kind, v string) string) (string, int) {
 	if len(found) == 0 {
 		return s, 0
 	}