delete
This commit is contained in:
Von committed
1 parent
fe2c24c7d0
commit
ea5ca42c2c
1 file changed
-460
@@ -1,460 +0,0 @@
|
||||
# LLM_Proxy Refactor Plan
|
||||
|
||||
Last updated: 2026-03-19
|
||||
|
||||
## 1. Background
|
||||
|
||||
The current proxy is centered around one assumption:
|
||||
|
||||
- upstream streaming responses are mostly SSE
|
||||
- downstream output should be OpenAI-compatible
|
||||
- request/response format adaptation is implemented by user hooks
|
||||
|
||||
This worked for simple integrations, but it creates several structural limits:
|
||||
|
||||
- streaming parsing is tightly coupled to `data:` SSE chunks
|
||||
- HTTP and WebSocket adaptation are normalized too early into one output shape
|
||||
- provider protocol adaptation and user policy logic are mixed in `input_body_hook` / `output_body_hook`
|
||||
- it is hard to support multiple upstream stream formats cleanly
|
||||
- hook behavior is powerful but underspecified, making testing and evolution difficult
|
||||
|
||||
The next refactor should be guided by the architectural direction used in `CLIProxyAPI`:
|
||||
|
||||
- separate transport execution from format translation
|
||||
- process streaming responses incrementally instead of assuming one global parser
|
||||
- keep provider adaptation as first-class internal components
|
||||
- keep user customization as an explicit extension boundary
|
||||
|
||||
## 2. Hard Constraints
|
||||
|
||||
These constraints are part of the refactor baseline:
|
||||
|
||||
- database tables must remain unchanged
|
||||
- compatibility for provider configuration, hook API, and internal module layout is not required
|
||||
- documentation must be updated together with implementation
|
||||
- README and `docs/architecture-4plus1.md` must be treated as required deliverables, not optional follow-up work
|
||||
- bold refactoring is allowed if it makes the core pipeline simpler and more extensible
|
||||
|
||||
## 3. Refactor Goals
|
||||
|
||||
### 3.1 Core goals
|
||||
|
||||
- support multiple upstream response formats, not only `data:` SSE
|
||||
- make streaming processing event-driven and incremental
|
||||
- separate provider protocol adaptation from user-defined logic
|
||||
- keep user customization for:
|
||||
- request header mutation
|
||||
- request guard
|
||||
- response guard
|
||||
- make HTTP and WebSocket go through one conceptual pipeline, while preserving protocol-specific boundaries where needed
|
||||
- make tests target stream decoding, translation, and guards as separate concerns
|
||||
|
||||
### 3.2 Non-goals for phase 1
|
||||
|
||||
- preserving the old `input_body_hook` / `output_body_hook` user API
|
||||
- preserving exact current provider config field semantics
|
||||
- preserving current internal package boundaries
|
||||
- adding more database-backed features
|
||||
|
||||
## 4. Target Architecture
|
||||
|
||||
The refactor should replace the current "controller -> proxy service -> hook -> response adapter" shape with the following pipeline.
|
||||
|
||||
```text
|
||||
Client Request
|
||||
-> Request Normalizer
|
||||
-> User Header Hook
|
||||
-> User Request Guard
|
||||
-> Provider Request Translator
|
||||
-> Upstream Executor (HTTP / WebSocket)
|
||||
-> Stream Decoder
|
||||
-> Provider Response Translator
|
||||
-> User Response Guard
|
||||
-> Downstream Encoder
|
||||
-> Client Response
|
||||
```
|
||||
|
||||
### 4.1 New responsibility split
|
||||
|
||||
- `User Header Hook`
|
||||
- user-owned
|
||||
- mutates outbound headers only
|
||||
- no protocol conversion responsibility
|
||||
|
||||
- `User Request Guard`
|
||||
- user-owned
|
||||
- runs on normalized request objects
|
||||
- for compliance checks, blocklists, prompt rewriting, guardrail injection
|
||||
|
||||
- `Provider Request Translator`
|
||||
- built-in, provider-owned
|
||||
- converts normalized downstream request into upstream provider request shape
|
||||
|
||||
- `Upstream Executor`
|
||||
- transport-owned
|
||||
- sends HTTP or WebSocket requests
|
||||
- yields raw response body or incremental stream units
|
||||
|
||||
- `Stream Decoder`
|
||||
- transport/format-owned
|
||||
- decodes upstream stream units into typed events
|
||||
- must support at least:
|
||||
- SSE JSON
|
||||
- SSE text
|
||||
- NDJSON
|
||||
- raw text chunk
|
||||
- WebSocket JSON message
|
||||
- WebSocket text message
|
||||
|
||||
- `Provider Response Translator`
|
||||
- built-in, provider-owned
|
||||
- converts typed upstream events or full bodies into OpenAI-compatible downstream objects
|
||||
|
||||
- `User Response Guard`
|
||||
- user-owned
|
||||
- runs after provider translation, before downstream encoding
|
||||
- for filtering, redaction, blocking, policy rewriting
|
||||
|
||||
- `Downstream Encoder`
|
||||
- protocol-owned
|
||||
- emits final downstream HTTP response
|
||||
- supports:
|
||||
- non-stream JSON
|
||||
- stream response compatible with downstream OpenAI clients
|
||||
|
||||
### 4.2 Key design principle
|
||||
|
||||
Provider adaptation and user policy must never be the same abstraction again.
|
||||
|
||||
That means:
|
||||
|
||||
- provider translation is internal product logic
|
||||
- user guard/header extensions are external customization logic
|
||||
- stream decoding is infrastructure logic
|
||||
|
||||
## 5. Proposed Module Layout
|
||||
|
||||
This is a target shape, not a strict final tree:
|
||||
|
||||
```text
|
||||
src/
|
||||
proxy_core/
|
||||
contracts/
|
||||
pipeline/
|
||||
errors/
|
||||
proxy_transport/
|
||||
http_executor.py
|
||||
websocket_executor.py
|
||||
proxy_stream/
|
||||
decoders/
|
||||
events/
|
||||
encoders/
|
||||
proxy_providers/
|
||||
registry.py
|
||||
adapters/
|
||||
proxy_extensions/
|
||||
loader.py
|
||||
contracts.py
|
||||
sandbox.py
|
||||
presentation/
|
||||
services/
|
||||
repositories/
|
||||
```
|
||||
|
||||
### 5.1 Suggested core contracts
|
||||
|
||||
- `NormalizedRequest`
|
||||
- `UpstreamRequest`
|
||||
- `UpstreamResponse`
|
||||
- `StreamEvent`
|
||||
- `TranslatedEvent`
|
||||
- `GuardDecision`
|
||||
- `ProxyError`
|
||||
|
||||
### 5.2 Suggested extension contracts
|
||||
|
||||
User extension API should become explicit and small:
|
||||
|
||||
- `header_hook(ctx, headers) -> headers`
|
||||
- `request_guard(ctx, request) -> request | GuardAbort | GuardDecision`
|
||||
- `response_guard(ctx, response_or_event) -> response_or_event | GuardAbort | GuardDecision`
|
||||
|
||||
Notes:
|
||||
|
||||
- user extension API should not directly own provider translation
|
||||
- request and response guards must work for both streaming and non-streaming modes
|
||||
- stream mode should support guard execution per event, not only after the full response completes
|
||||
|
||||
## 6. Streaming Strategy
|
||||
|
||||
This is the most important architectural change.
|
||||
|
||||
### 6.1 Current limitation
|
||||
|
||||
The current implementation assumes one main path:
|
||||
|
||||
- detect SSE
|
||||
- split by event boundary
|
||||
- parse `data:`
|
||||
- optionally mutate JSON payload
|
||||
|
||||
This is not enough for multi-vendor compatibility.
|
||||
|
||||
### 6.2 New strategy
|
||||
|
||||
Each executor should emit raw upstream units, then a decoder should transform them into typed events.
|
||||
|
||||
For HTTP:
|
||||
|
||||
- non-stream: return full body
|
||||
- stream: yield raw byte chunks, then decode according to configured or detected format
|
||||
|
||||
For WebSocket:
|
||||
|
||||
- yield one message at a time
|
||||
- preserve message boundaries at the decoder layer
|
||||
|
||||
### 6.3 Decoder modes to support first
|
||||
|
||||
- `sse_json`
|
||||
- `sse_text`
|
||||
- `ndjson`
|
||||
- `raw_text`
|
||||
- `ws_json`
|
||||
- `ws_text`
|
||||
|
||||
### 6.4 Translator behavior
|
||||
|
||||
The provider translator should be allowed to:
|
||||
|
||||
- ignore some upstream events
|
||||
- map one upstream event to one downstream event
|
||||
- map one upstream event to multiple downstream events
|
||||
- accumulate limited local state across stream events
|
||||
|
||||
This is the part most directly inspired by `CLIProxyAPI`.
|
||||
|
||||
## 7. Provider Model
|
||||
|
||||
The new provider configuration model should be simplified around explicit strategy selection.
|
||||
|
||||
Suggested fields:
|
||||
|
||||
- `name`
|
||||
- `models`
|
||||
- `transport`
|
||||
- `api`
|
||||
- `auth`
|
||||
- `request_format`
|
||||
- `response_format`
|
||||
- `decoder`
|
||||
- `adapter`
|
||||
- `extension`
|
||||
- `timeout_seconds`
|
||||
- `max_retries`
|
||||
- `verify_ssl`
|
||||
- `proxy`
|
||||
|
||||
Notes:
|
||||
|
||||
- this may replace the current "one hook does everything" mental model
|
||||
- provider compatibility with old config is not required
|
||||
- built-in adapters should be the default path
|
||||
|
||||
## 8. Migration Plan
|
||||
|
||||
### Phase 0. Freeze and planning
|
||||
|
||||
- document the new architecture and extension boundaries
|
||||
- define the first supported decoder and adapter matrix
|
||||
- confirm which downstream API surface remains in scope for this iteration
|
||||
|
||||
Deliverables:
|
||||
|
||||
- this plan document
|
||||
- updated implementation backlog
|
||||
|
||||
### Phase 1. Core contracts and pipeline skeleton
|
||||
|
||||
- create core proxy contracts and error model
|
||||
- introduce executor / decoder / translator / guard interfaces
|
||||
- keep existing routes, but route them through the new pipeline shell
|
||||
|
||||
Deliverables:
|
||||
|
||||
- new core packages
|
||||
- typed contracts
|
||||
- pipeline orchestration tests
|
||||
|
||||
### Phase 2. Transport executors
|
||||
|
||||
- extract current HTTP upstream logic into an HTTP executor
|
||||
- extract current WebSocket upstream logic into a WebSocket executor
|
||||
- standardize retry, timeout, and upstream error reporting
|
||||
|
||||
Deliverables:
|
||||
|
||||
- executor tests
|
||||
- unified upstream error model
|
||||
|
||||
### Phase 3. Stream decoders
|
||||
|
||||
- implement SSE JSON decoder
|
||||
- implement SSE text decoder
|
||||
- implement NDJSON decoder
|
||||
- implement raw text chunk decoder
|
||||
- implement WebSocket JSON/text decoders
|
||||
|
||||
Deliverables:
|
||||
|
||||
- decoder fixtures
|
||||
- chunk-boundary tests
|
||||
- UTF-8 split tests
|
||||
|
||||
### Phase 4. Provider translators
|
||||
|
||||
- replace current `input_body_hook` / `output_body_hook` adaptation path with provider adapters
|
||||
- implement one built-in OpenAI-compatible adapter first
|
||||
- add one non-trivial adapter path to validate the architecture
|
||||
|
||||
Deliverables:
|
||||
|
||||
- provider adapter registry
|
||||
- stream and non-stream translator tests
|
||||
|
||||
### Phase 5. User extensions
|
||||
|
||||
- introduce new user extension contracts:
|
||||
- header hook
|
||||
- request guard
|
||||
- response guard
|
||||
- add extension loading rules, examples, and failure handling
|
||||
- remove or deprecate the old generic body hooks
|
||||
|
||||
Deliverables:
|
||||
|
||||
- extension loader
|
||||
- sample user extension files
|
||||
- extension error and logging strategy
|
||||
|
||||
### Phase 6. Presentation and config cleanup
|
||||
|
||||
- simplify controller responsibilities
|
||||
- reduce protocol knowledge in presentation layer
|
||||
- redesign provider management UI and config schema around adapters + extensions
|
||||
|
||||
Deliverables:
|
||||
|
||||
- updated admin flows
|
||||
- revised config templates
|
||||
|
||||
### Phase 7. Documentation and architecture refresh
|
||||
|
||||
- rewrite README to describe:
|
||||
- new architecture
|
||||
- supported stream formats
|
||||
- extension model
|
||||
- provider adapter model
|
||||
- rewrite `docs/architecture-4plus1.md`
|
||||
- add migration notes for old hook users
|
||||
|
||||
Deliverables:
|
||||
|
||||
- README updated
|
||||
- 4+1 architecture document updated
|
||||
- examples updated
|
||||
|
||||
### Phase 8. Cleanup and cutover
|
||||
|
||||
- remove dead compatibility layers
|
||||
- remove obsolete hook assumptions
|
||||
- tighten tests around final contracts
|
||||
|
||||
Deliverables:
|
||||
|
||||
- cleaned module tree
|
||||
- stable final tests
|
||||
|
||||
## 9. Documentation Work Required
|
||||
|
||||
Every implementation phase must update the matching documentation.
|
||||
|
||||
At minimum, the final refactor must update:
|
||||
|
||||
- `README.md`
|
||||
- `docs/architecture-4plus1.md`
|
||||
- provider examples in `config.sample.yaml`
|
||||
- hook or extension examples under `hooks/`
|
||||
|
||||
The README should stop describing the system as primarily a generic hook-based body rewrite proxy.
|
||||
|
||||
The 4+1 architecture document should reflect:
|
||||
|
||||
- executor / decoder / translator / guard split
|
||||
- the new extension boundary
|
||||
- the new stream processing model
|
||||
|
||||
## 10. Testing Plan
|
||||
|
||||
The refactor is not complete without a new test strategy.
|
||||
|
||||
Required test layers:
|
||||
|
||||
- contract tests for core pipeline
|
||||
- decoder tests with fragmented chunks
|
||||
- translator tests for stream and non-stream paths
|
||||
- guard tests for block / rewrite / pass-through
|
||||
- executor tests for retry and upstream error behavior
|
||||
- end-to-end API tests for `/v1/chat/completions`
|
||||
|
||||
Special attention:
|
||||
|
||||
- UTF-8 characters split across chunks
|
||||
- mixed SSE event fields
|
||||
- non-SSE upstream stream formats
|
||||
- WebSocket multi-message responses
|
||||
- guard interruption during streaming
|
||||
|
||||
## 11. Risks
|
||||
|
||||
- phase 1 and phase 2 may temporarily increase code duplication
|
||||
- replacing generic hooks with adapters and guards will break current custom integrations
|
||||
- stream abstraction may become overdesigned if too many formats are introduced at once
|
||||
- admin UI and provider config editing will need a coordinated redesign
|
||||
|
||||
Mitigation:
|
||||
|
||||
- implement one complete vertical slice early
|
||||
- keep the first supported decoder matrix intentionally small
|
||||
- favor explicit contracts over highly dynamic hook magic
|
||||
|
||||
## 12. Recommended First Execution Slice
|
||||
|
||||
The first real implementation slice should be:
|
||||
|
||||
- keep downstream `POST /v1/chat/completions`
|
||||
- support upstream HTTP transport
|
||||
- support `sse_json`, `ndjson`, and non-stream JSON
|
||||
- introduce:
|
||||
- `header_hook`
|
||||
- `request_guard`
|
||||
- `response_guard`
|
||||
- add one built-in provider adapter
|
||||
- remove the assumption that all streaming data is `data:`
|
||||
|
||||
Reason:
|
||||
|
||||
- this slice proves the new architecture without having to finish every protocol at once
|
||||
- it gives the highest return on the current pain point
|
||||
|
||||
## 13. Definition of Done
|
||||
|
||||
This refactor is considered done only when all of the following are true:
|
||||
|
||||
- provider adaptation no longer depends on generic user body hooks
|
||||
- user customization is explicitly limited to header hook and guards
|
||||
- multi-format streaming is supported through decoders
|
||||
- HTTP and WebSocket both use the new pipeline model
|
||||
- README and 4+1 architecture documentation are updated
|
||||
- database tables remain unchanged
|
||||
- old SSE-only assumptions are removed from the core design
|
||||
Reference in new issue
Block a user