redact: what a vendor wrote and checks goes back as it wrote it — signed thinking and tool call ids (Jeremy.Zhou on Discord: OpenCode Zen answered 400 "invalid …" until 脱敏密钥 and 脱敏个人信息 were off)

Masking ran its rules over every string of the request except a fixed list
of keys, including two kinds of text the vendor itself wrote and checks
when it comes back:

- A thinking block (Anthropic's signature; also Gemini's thoughtSignature
  and Responses' encrypted reasoning). The vendor thinks about the
  placeholders it was given, the agent gets the values back (Restore), and
  on the next turn the rules masked that text again. Rules depend on
  what is around a value: "the password is {{SECRET_…}}" came back as
  "the password is hunter2abc1", which no rule matches without its
  DB_PASSWORD=, an email ending a sentence fails the email rule's bound,
  and a number the vendor wrote of its own ("13800138000") turned into a
  {{PHONE_…}}. The text no longer matched its signature, and Anthropic
  answers 400 "messages.1.content.0: Invalid `signature` in `thinking`
  block" — and the value itself went to the vendor in the clear.
  Now an object that carries a non-empty signature, thoughtSignature,
  thought_signature or encrypted_content is masked only by putting back
  the placeholders of the values masked elsewhere in the same request
  (longest first): what the vendor wrote, its own words untouched.
- A chat tool result's tool_call_id. "id", "tool_use_id" and "call_id"
  were kept, "tool_call_id" wasn't. GLM's ids are call_ and 19 digits,
  and ~5% of them pass Luhn as a bank card number, so the tool message's
  id became call_-{{BANK_CARD_…}} while the assistant's tool_calls id
  stayed, and the upstream refused the orphaned tool result ("messages
  with role 'tool' must be a response to a preceeding message with
  'tool_calls'"). Now every key named id, …_id, …Id (and their plurals)
  is left alone.

Account masking (账号打码) is display only and doesn't touch requests.

Tests: TestSignedThinkingGoesBackAsSigned (a fake Anthropic upstream that
signs its thinking with an HMAC and checks it, two turns through the
gateway) and TestToolCallIDsGoBackAsGiven (a fake chat upstream that
refuses a tool result with no matching call) fail on origin/main with
exactly those 400s; TestMaskJSONSignedGoesBackAsWritten (Anthropic,
Gemini thought and functionCall, Responses reasoning shapes, plus an
unsigned thinking block still masked) and TestMaskJSONKeepsIDs fail there
too. All pass with the change.

Run: gofmt -l (nothing), go vet -tags nogui ./... on darwin, GOOS=linux
and GOOS=windows, go build -tags nogui (darwin, linux, windows),
go test -tags nogui ./... under a temp HOME: all ok. Not tried against the
real OpenCode Zen, and the reporter's exact error text and model weren't
seen; both upstream checks are modeled on Anthropic's and OpenAI's
documented errors.

docs/subsystems/gateway-routing.md

@@ -30,7 +30,7 @@ served by a plugin are in [Provider and plugin ownership](provider-plugins.md).
 ## Runtime path
 
 1. `handle(proto)` reads the body and the model (`requestBody`, `requestModel`), then calls `serveAgent`. Every route that reads an agent's body (each API, count_tokens, Gemini's, embeddings, images, videos, System One, the Codex backend path) reads it through `readBoundedRequestBody` in [`request_bounds.go`](../../internal/gateway/request_bounds.go). It decodes a `Content-Encoding` of `gzip` or `zstd` (`decodedBody`; Codex sends zstd to a custom provider's `/v1/responses` too, #1223) and drops the header, so the provider is sent plain JSON. The size limit holds for the decoded bytes (413), and any other encoding is refused 415. `keyLimited` reads a limited key's compressed body the same way, so its reservation is the decoded size. `serveAgent` runs the [middleware](gateway-middleware.md) chain, then `serve`.
-2. `serve` redacts secrets in the request (`redacted`), resolves the model to a provider or a group, and refuses what can't be served. A request whose provider and fallbacks, or whose group's models, all skip redaction (`unredactedRoute`, `provider.SkipsRedaction`: set *Send requests unmasked*, `Unredacted`, with every address on this machine or the local network) goes to them as the agent wrote it; the log and recent calls keep the masked body. count_tokens and embeddings follow the same rule.
+2. `serve` redacts secrets in the request (`redacted`; `redact.MaskJSON` leaves ids alone, and what the vendor sealed with a signature, a thinking block, Gemini's thoughts or Responses' encrypted reasoning, goes back as the vendor wrote it: only the values masked elsewhere in the request are put back as their placeholders), resolves the model to a provider or a group, and refuses what can't be served. A request whose provider and fallbacks, or whose group's models, all skip redaction (`unredactedRoute`, `provider.SkipsRedaction`: set *Send requests unmasked*, `Unredacted`, with every address on this machine or the local network) goes to them as the agent wrote it; the log and recent calls keep the masked body. count_tokens and embeddings follow the same rule.
    - `DecidesModel` gives a 400.
    - A gateway key held to some models (`keyHolds`, #882) gets a 403 for any other model.
    - Sealed tasks (#619) are checked here. A Codex subagent's task its lead's `spawn_agent` sealed goes only to a ChatGPT account, or back to the Responses provider that answered the lead (`leadProvider`, a relay of the ChatGPT backend such as Sub2API, #1109), relayed as it came even when web search is offered. A lead's turn carrying its subagent's sealed reply (`send_message`) goes back to the Responses provider that answered the lead's own conversation (`leadProvider` of the request's `conversationID`, #1237); otherwise it is refused before anyone is asked (`sealedReader`, `sealedTaskError`). Who answered is first what affinity remembers; when affinity has let it go (a day since, a restart past the 512 conversations `affinity.json` keeps, or the lead's last reply broken off, `unanswered`), the usage log's answers to the lead's thread and the request's own conversation in the last 30 days count too (`loggedAnswers`, #1367), and `leadFirst` reads the lead's key or account from them. A refused or failed call and a title answer nothing, and Codex's own sign-in is told of as the ChatGPT backend. The refusal names who sealed the task when it is known, and says how to go on with it.

internal/gateway/redact_roundtrip_test.go

@@ -0,0 +1,166 @@
+package gateway
+
+import (
+	"crypto/hmac"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"regexp"
+	"strings"
+	"testing"
+
+	"github.com/yetone/magpie/internal/provider"
+	"github.com/yetone/magpie/internal/settings"
+)
+
+// What a vendor wrote itself and checks when it comes back goes back as the
+// vendor wrote it, with masking on (Jeremy.Zhou on Discord: OpenCode Zen
+// answered 400 until 脱敏密钥 and 脱敏个人信息 were turned off).
+
+// sign is the fake vendor's thinking signature: it covers the text exactly,
+// as Anthropic's does.
+func sign(text string) string {
+	m := hmac.New(sha256.New, []byte("vendor"))
+	m.Write([]byte(text))
+	return hex.EncodeToString(m.Sum(nil))
+}
+
+// A thinking block that names a placeholder comes back to the agent with
+// the value in it; sent again, it goes as the vendor signed it, placeholder
+// and all, so its signature still holds and the value doesn't leak.
+func TestSignedThinkingGoesBackAsSigned(t *testing.T) {
+	t.Setenv("XDG_CONFIG_HOME", t.TempDir())
+	t.Setenv("XDG_CACHE_HOME", t.TempDir())
+	const email = "jeremy.zhou@gmail.com"
+	const secret = "hunter2abc1"
+	placeholder := regexp.MustCompile(`\{\{[A-Z_]+_[a-z2-7]{8}\}\}`)
+	var sent []string
+	up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		b, _ := io.ReadAll(r.Body)
+		sent = append(sent, string(b))
+		var req struct {
+			Messages []struct {
+				Role    string          `json:"role"`
+				Content json.RawMessage `json:"content"`
+			} `json:"messages"`
+		}
+		if err := json.Unmarshal(b, &req); err != nil {
+			w.WriteHeader(400)
+			io.WriteString(w, `{"type":"error","error":{"type":"invalid_request_error","message":"invalid JSON"}}`)
+			return
+		}
+		for i, m := range req.Messages {
+			var blocks []map[string]any
+			if json.Unmarshal(m.Content, &blocks) != nil {
+				continue
+			}
+			for j, b := range blocks {
+				if b["type"] != "thinking" {
+					continue
+				}
+				text, _ := b["thinking"].(string)
+				if sig, _ := b["signature"].(string); sig != sign(text) {
+					w.WriteHeader(400)
+					io.WriteString(w, `{"type":"error","error":{"type":"invalid_request_error","message":"messages.`+itoa(i)+`.content.`+itoa(j)+`: Invalid `+"`signature`"+` in `+"`thinking`"+` block"}}`)
+					return
+				}
+			}
+		}
+		// the vendor thinks about what it was given, placeholders and all,
+		// in its own words
+		ps := placeholder.FindAllString(string(b), -1)
+		// and of its own: a number that reads as a phone number
+		thinking := "Order 13800138000: the password is " + strings.Join(ps, ", for the user ") + "."
+		w.Header().Set("Content-Type", "application/json")
+		json.NewEncoder(w).Encode(map[string]any{
+			"id": "msg_1", "type": "message", "role": "assistant", "model": "claude-sonnet-4-5",
+			"content":     []any{map[string]any{"type": "thinking", "thinking": thinking, "signature": sign(thinking)}, map[string]any{"type": "text", "text": "ok"}},
+			"stop_reason": "end_turn", "usage": map[string]any{"input_tokens": 10, "output_tokens": 5},
+		})
+	}))
+	defer up.Close()
+	if err := provider.Save(provider.Provider{ID: "zen", Name: "Zen", Key: "k", Models: []string{"claude-sonnet-4-5"}, Anthropic: up.URL}); err != nil {
+		t.Fatal(err)
+	}
+	if err := settings.Save(settings.Settings{Redact: true, RedactPersonal: true}); err != nil {
+		t.Fatal(err)
+	}
+	user := `{"role":"user","content":"DB_PASSWORD=` + secret + ` and I am ` + email + `"}`
+	code, body := post(t, "/v1/messages", `{"model":"zen/claude-sonnet-4-5","max_tokens":100,"thinking":{"type":"enabled","budget_tokens":64},"messages":[`+user+`]}`)
+	if code != 200 {
+		t.Fatalf("first turn: %d %s", code, body)
+	}
+	var first struct {
+		Content []map[string]any `json:"content"`
+	}
+	json.Unmarshal([]byte(body), &first)
+	if len(first.Content) == 0 || first.Content[0]["type"] != "thinking" {
+		t.Fatalf("no thinking: %s", body)
+	}
+	if th := first.Content[0]["thinking"].(string); !strings.Contains(th, email) || !strings.Contains(th, secret) {
+		t.Fatalf("the agent should read its values back in the thinking: %q", th)
+	}
+	asst, _ := json.Marshal(map[string]any{"role": "assistant", "content": first.Content})
+	code, body = post(t, "/v1/messages", `{"model":"zen/claude-sonnet-4-5","max_tokens":100,"thinking":{"type":"enabled","budget_tokens":64},"messages":[`+user+`,`+string(asst)+`,{"role":"user","content":"go on"}]}`)
+	if code != 200 {
+		t.Fatalf("second turn: %d %s\nsent: %s", code, body, sent[len(sent)-1])
+	}
+	if last := sent[len(sent)-1]; strings.Contains(last, email) || strings.Contains(last, secret) {
+		t.Fatalf("a value went to the vendor: %s", last)
+	}
+}
+
+// A tool result goes back under the id the vendor gave the call, whatever
+// digits the id has: GLM's are call_ and 19 of them, which can read as a
+// bank card number.
+func TestToolCallIDsGoBackAsGiven(t *testing.T) {
+	t.Setenv("XDG_CONFIG_HOME", t.TempDir())
+	t.Setenv("XDG_CACHE_HOME", t.TempDir())
+	const id = "call_-6090648166611204696" // its 19 digits pass Luhn
+	var sent string
+	up := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		b, _ := io.ReadAll(r.Body)
+		sent = string(b)
+		var req struct {
+			Messages []struct {
+				Role       string `json:"role"`
+				ToolCallID string `json:"tool_call_id"`
+				ToolCalls  []struct {
+					ID string `json:"id"`
+				} `json:"tool_calls"`
+			} `json:"messages"`
+		}
+		json.Unmarshal(b, &req)
+		calls := map[string]bool{}
+		for _, m := range req.Messages {
+			for _, c := range m.ToolCalls {
+				calls[c.ID] = true
+			}
+			if m.Role == "tool" && !calls[m.ToolCallID] {
+				w.WriteHeader(400)
+				io.WriteString(w, `{"error":{"message":"Invalid parameter: messages with role 'tool' must be a response to a preceeding message with 'tool_calls'.","type":"invalid_request_error"}}`)
+				return
+			}
+		}
+		w.Header().Set("Content-Type", "application/json")
+		io.WriteString(w, `{"id":"c1","object":"chat.completion","model":"glm-4.6","choices":[{"index":0,"message":{"role":"assistant","content":"done"},"finish_reason":"stop"}],"usage":{"prompt_tokens":3,"completion_tokens":1}}`)
+	}))
+	defer up.Close()
+	if err := provider.Save(provider.Provider{ID: "zen", Name: "Zen", Key: "k", Models: []string{"glm-4.6"}, Chat: up.URL + "/v1"}); err != nil {
+		t.Fatal(err)
+	}
+	if err := settings.Save(settings.Settings{Redact: true, RedactPersonal: true}); err != nil {
+		t.Fatal(err)
+	}
+	req := `{"model":"zen/glm-4.6","messages":[{"role":"user","content":"list files"},` +
+		`{"role":"assistant","content":"","tool_calls":[{"id":"` + id + `","type":"function","function":{"name":"bash","arguments":"{\"command\":\"ls\"}"}}]},` +
+		`{"role":"tool","tool_call_id":"` + id + `","content":"a.go"}]}`
+	if code, body := post(t, "/v1/chat/completions", req); code != 200 {
+		t.Fatalf("%d %s\nsent: %s", code, body, sent)
+	}
+}
+
+func itoa(i int) string { b, _ := json.Marshal(i); return string(b) }

internal/redact/json.go

@@ -3,6 +3,7 @@ package redact
 import (
 	"bytes"
 	"encoding/json"
+	"sort"
 	"strconv"
 	"strings"
 )
@@ -19,7 +20,71 @@ var kept = map[string]bool{
 }
 
 func keep(key, s string) bool {
-	return kept[key] || strings.HasPrefix(s, "data:")
+	return kept[key] || isID(key) || strings.HasPrefix(s, "data:")
+}
+
+// isID says a key holds an identifier the vendor matches against another:
+// a tool result's tool_call_id against its call's id, an approval's
+// approval_request_id against its request. A vendor's ids are its own
+// letters and digits, and GLM's call_ and 19 digits can read as a bank
+// card number; masked on one side only, the two no longer match.
+func isID(key string) bool {
+	return key == "id" || key == "ids" || strings.HasSuffix(key, "_id") || strings.HasSuffix(key, "_ids") ||
+		strings.HasSuffix(key, "Id") || strings.HasSuffix(key, "Ids")
+}
+
+// signatureKeys are the keys of what a vendor seals the content beside it
+// with: Anthropic's thinking signature, Gemini's thought signature,
+// Responses' encrypted reasoning.
+var signatureKeys = []string{"signature", "thoughtSignature", "thought_signature", "encrypted_content"}
+
+// signedPaths are the paths of the objects in body that carry a signature
+// with a value: what is in them the vendor wrote and checks, so it goes
+// back exactly as the vendor wrote it.
+func signedPaths(body []byte) []string {
+	found := false
+	for _, k := range signatureKeys {
+		found = found || bytes.Contains(body, []byte(`"`+k+`"`))
+	}
+	if !found {
+		return nil
+	}
+	var v any
+	if json.Unmarshal(body, &v) != nil {
+		return nil
+	}
+	var out []string
+	var visit func(path string, v any)
+	visit = func(path string, v any) {
+		switch x := v.(type) {
+		case map[string]any:
+			for _, k := range signatureKeys {
+				if s, _ := x[k].(string); s != "" {
+					out = append(out, path)
+					return
+				}
+			}
+			for k, c := range x {
+				visit(join(path, k), c)
+			}
+		case []any:
+			for i, c := range x {
+				visit(join(path, strconv.Itoa(i)), c)
+			}
+		}
+	}
+	visit("", v)
+	return out
+}
+
+// under says path is in one of the objects at roots.
+func under(path string, roots []string) bool {
+	for _, r := range roots {
+		if r == "" || path == r || strings.HasPrefix(path, r+".") {
+			return true
+		}
+	}
+	return false
 }
 
 // walk calls fn with every string value in the JSON document b — its path
@@ -177,13 +242,32 @@ func jsonEscape(s string) string {
 
 // MaskJSON masks the strings of a request body, and says how many values
 // it masked. A body that isn't JSON is masked as text.
+//
+// What a vendor sealed with a signature (a thinking block, Gemini's
+// thoughts, Responses' encrypted reasoning) is checked by the vendor
+// against the text it wrote, so it goes back as written: the values
+// masked elsewhere in the request become their placeholders again, which
+// is what the vendor wrote where the agent reads the value (Restore), and
+// nothing else in it is touched. Masked by the rules instead, a value the
+// vendor wrote as a placeholder in other words around it ("the password
+// is {{SECRET_…}}") went back as the value itself, and what the vendor
+// wrote of its own that looks like an email or a phone number went as a
+// placeholder: either way the text no longer matched its signature, and
+// Anthropic answers 400 "Invalid `signature` in `thinking` block".
 func MaskJSON(body []byte, o Options) ([]byte, int) {
+	signed := signedPaths(body)
 	total := 0
-	out, ok := walk(body, func(_, key, s string) string {
-		if keep(key, s) {
+	var seen []string // value, placeholder, value, placeholder…
+	put := func(kind, v string) string {
+		p := placeholder(kind, v)
+		seen = append(seen, v, p)
+		return p
+	}
+	out, ok := walk(body, func(path, key, s string) string {
+		if keep(key, s) || signed != nil && under(path, signed) {
 			return s
 		}
-		t, n := Mask(s, o)
+		t, n := mask(s, o, put)
 		total += n
 		return t
 	})
@@ -191,9 +275,43 @@ func MaskJSON(body []byte, o Options) ([]byte, int) {
 		t, n := Mask(string(body), o)
 		return []byte(t), n
 	}
+	if len(signed) == 0 || len(seen) == 0 {
+		return out, total
+	}
+	remask := replacerOf(seen)
+	out, _ = walk(out, func(path, key, s string) string {
+		if keep(key, s) || !under(path, signed) {
+			return s
+		}
+		t := remask.Replace(s)
+		if t != s {
+			total++
+		}
+		return t
+	})
 	return out, total
 }
 
+// replacerOf swaps each value of pairs (value, placeholder, …) for its
+// placeholder, the longest value first where two start at the same place.
+func replacerOf(pairs []string) *strings.Replacer {
+	type pair struct{ v, p string }
+	var ps []pair
+	have := map[string]bool{}
+	for i := 0; i+1 < len(pairs); i += 2 {
+		if !have[pairs[i]] {
+			have[pairs[i]] = true
+			ps = append(ps, pair{pairs[i], pairs[i+1]})
+		}
+	}
+	sort.SliceStable(ps, func(i, j int) bool { return len(ps[i].v) > len(ps[j].v) })
+	args := make([]string, 0, 2*len(ps))
+	for _, p := range ps {
+		args = append(args, p.v, p.p)
+	}
+	return strings.NewReplacer(args...)
+}
+
 // asJSON says a string under key holds JSON text of its own, so a value
 // put back in it goes in escaped: a tool call's arguments, as a whole or
 // streamed in pieces.

internal/redact/redact_test.go

@@ -166,3 +166,52 @@ func TestWriterWhole(t *testing.T) {
 		t.Fatalf("%v %s", rec.Header(), rec.Body.String())
 	}
 }
+
+// What a vendor sealed goes back as the vendor wrote it: the values masked
+// elsewhere in the request as their placeholders, its own words as they
+// are; the same text unsealed is masked as any other.
+func TestMaskJSONSignedGoesBackAsWritten(t *testing.T) {
+	o := Options{Secrets: true, Personal: true}
+	const email, key = "jeremy.zhou@gmail.com", "sk-proj-abcdEFGH1234ijklMNOP5678qrst"
+	pe, _ := Mask(email, o)
+	pk, _ := Mask(key, o)
+	// what the agent has back of what the vendor wrote: the values in it
+	// in other words around them, and the vendor's own number
+	agent := "Order 13800138000 is for " + email + ". Its key is the one ending " + key + "."
+	vendor := "Order 13800138000 is for " + pe + ". Its key is the one ending " + pk + "."
+	user := `{"role":"user","content":"I am ` + email + `, key ` + key + `"}`
+	for _, c := range []struct{ name, body, want string }{
+		{"anthropic", `{"messages":[` + user + `,{"role":"assistant","content":[{"type":"thinking","thinking":"` + agent + `","signature":"Eqk1"}]}]}`, vendor},
+		{"gemini", `{"contents":[{"role":"user","parts":[{"text":"I am ` + email + `, key ` + key + `"}]},{"role":"model","parts":[{"text":"` + agent + `","thought":true,"thoughtSignature":"Cs4B"}]}]}`, vendor},
+		{"gemini call", `{"contents":[{"role":"user","parts":[{"text":"I am ` + email + `, key ` + key + `"}]},{"role":"model","parts":[{"functionCall":{"name":"note","args":{"text":"` + agent + `"}},"thoughtSignature":"Cs4B"}]}]}`, vendor},
+		{"responses", `{"input":[` + user + `,{"type":"reasoning","summary":[{"type":"summary_text","text":"` + agent + `"}],"encrypted_content":"gAAAA"}]}`, vendor},
+	} {
+		out, _ := MaskJSON([]byte(c.body), o)
+		if !strings.Contains(string(out), `"`+c.want+`"`) {
+			t.Errorf("%s: sealed text not as the vendor wrote it:\n%s", c.name, out)
+		}
+		if strings.Contains(string(out), email) || strings.Contains(string(out), key) {
+			t.Errorf("%s: a value went out: %s", c.name, out)
+		}
+	}
+	// no signature, nothing to match: masked by the rules
+	out, _ := MaskJSON([]byte(`{"messages":[{"role":"assistant","content":[{"type":"thinking","thinking":"call 13800138000 now","signature":""}]}]}`), o)
+	if strings.Contains(string(out), "13800138000") {
+		t.Errorf("unsigned thinking left unmasked: %s", out)
+	}
+}
+
+// An id goes as the vendor gave it, on both sides of what it matches.
+func TestMaskJSONKeepsIDs(t *testing.T) {
+	o := Options{Secrets: true, Personal: true}
+	const id = "call_-6090648166611204696" // 19 digits that pass Luhn
+	for _, k := range []string{"id", "tool_call_id", "tool_use_id", "call_id", "approval_request_id", "toolCallId"} {
+		body := `{"` + k + `":"` + id + `"}`
+		if out, _ := MaskJSON([]byte(body), o); string(out) != body {
+			t.Errorf("%s masked: %s", k, out)
+		}
+	}
+	if out, _ := MaskJSON([]byte(`{"content":"card `+id[6:]+`"}`), o); strings.Contains(string(out), id[6:]) {
+		t.Errorf("the same digits in text should be masked: %s", out)
+	}
+}