okooooooo/Termind

0 Fork 0

Files

T

Kaede Akino bc1ffb3ac3 Document module and directory architecture

2026-09-22 12:07:14 +08:00

17 KiB

Raw Blame History

type, title, description, tags, verified, sources, generated

type

title

description

SSH Multiplexing, Channel Admission, and I/O Scheduling

What multiplexing means here

An authenticated connection owns a libssh2 session and routed byte transport. A channel is a protocol stream within that connection: a PTY shell, non-PTY exec, SFTP subsystem, or forwarded TCP stream. Termind's multiplexing reuses these in-process authenticated owners; the code described here is not OpenSSH ControlMaster/ControlPath process management. Opening another channel avoids another authentication exchange; opening another physical connection does not.

Concurrency therefore has several independent meanings:

Layer	Unit and owner	Admission or scheduling
Feature connection reuse	Terminal/session, Agent pool, forwarding Host context, or SFTP transfer context	Determines which authenticated connection an operation may use.
SFTP file admission	Logical file, across distinct endpoint Hosts	`SFTPConcurrencyScheduler` rotates hosts and work sources; it is not a byte scheduler.
Session-channel admission	PTY/exec/subsystem lease on one connection	`SSHChannelBudget` applies a FIFO ceiling, default 10.
libssh2 ownership	A single session handle	The owning dispatch queue serializes every library call.
I/O progression	Exec command, forwarding flow, or SFTP operation	Bounded nonblocking turns yield to peers; the exact wake/retry mechanism depends on the owner.

Do not conflate an SSHChannelLease with an SFTP file permit or with a pool slot. More admitted files need not mean more sockets; more forwarding sockets do not increase the channel ceiling on an existing connection.

flowchart TD
    Feature[Feature chooses transport owner] --> Conn[One authenticated SSH connection]
    Conn --> Queue[Single serial libssh2 queue]
    Policy[Shared MaxSessions preference] --> Budget[Budget for this connection]
    PTY[PTY shell] --> Budget
    Exec[Independent exec requests] --> Budget
    SFTP[Sibling SFTP subsystems] --> Budget
    Budget --> Queue
    Forward[Forwarded TCP streams] -->|no session lease| Queue
    Queue --> Buffer[Encrypted connection buffer]
    Buffer --> Route[Routed socket or jump bridge]

This is a conceptual layering diagram, not a claim that every owner supports every channel type. The PTY worker owns terminal plus sibling SFTP channels; SSHConnectionWorker owns exec plus SFTP; forwarding has its own worker.

Serial ownership is not serial completion

SSHConnection is an actor facade, but SSHConnectionWorker.queue is the actual libssh2 owner. Async callers can have several outstanding exec commands; they do not each receive a blocking thread that independently enters the shared session. The worker keeps per-command state and advances it on a serial queue. SFTP drivers submit short operations to that same queue. The PTY worker constructs the same SSHMultiplexedSFTPChannelOwner with its own context/queue instead of creating a second libssh2 owner for the terminal transport.

Metrics deliberately opts out of concurrent operations: SSHExclusiveConnectionAdapter.withConnection uses a busy flag and FIFO continuations to serialize complete operations on its owned connection. That adapter has no topology/fallback policy. Similar-looking SSH usage does not authorize merging Metrics, terminal, Agent, and forwarding lifetimes.

The Agent has a separate higher-level reuse policy: AgentConnectionPool keys connections by Host ID plus route fingerprint, joins same-fingerprint in-flight opens, and opens primary SFTP on its pooled SSH connection. Reuse survives a follow-up Agent run; default idle expiry is 180 seconds, and active exec/SFTP leases prevent whole-host idle retirement. Missing/changed routes invalidate reuse. This is an Agent-owned pool, not the terminal or forwarding pool.

Session-channel budget: local FIFO, not server discovery

SSHMaxSessionsPolicy shares the configured ceiling across weakly tracked budgets, but makeBudget() creates independent accounting for each connection. SSHPreferences normalizes the ceiling to 1...100, default 10. This is a client-side ceiling, not a discovered or negotiated value of the remote server's MaxSessions; the server can still reject a channel below the local limit.

SSHChannelBudget.acquire immediately grants only when capacity is available and no waiter is ahead. Otherwise it queues a continuation. Lease release is idempotent, and deinit also releases. Increasing the ceiling drains FIFO waiters; decreasing it does not revoke active channels. Task/token cancellation removes a queued waiter, and closing a budget fails all remaining waiters with cancellation. A waiter does not busy-spin inside libssh2 while it lacks a lease.

PTY setup retains one lease for the shell; exec retains one in its command object; each SFTP subsystem retains one beside its native pointer. Direct/forwarded TCP channels and remote listeners are outside this session-channel accounting. Consequently a terminal with ceiling 10 already uses one slot, leaving at most nine additional session channels on that transport under this local policy. Creating a sibling SFTP channel at capacity waits for admission; it does not itself trigger an extra connection.

Exec: opening → starting → running → closing → freeing

SSHConnectionWorker.execute rejects an empty command, gets the current budget, acquires a lease, and checks on the queue that the connection still uses that same budget. It then creates SSHExecChannel and enqueues its ID in pendingOpens. The post-acquire identity check prevents admission on an old budget from attaching to a replacement connection.

Each command owns:

A stable, charset-encoded C command string. It executes /bin/sh -lc '<command>', optionally preceded by a shell-quoted cd -- '<cwd>' &&; there is no shared interactive shell cwd between exec channels.
Its own stdin byte queue with consumed-byte zeroization, stdout/stderr decoders and buffers, truncation counters, deadline, cancellation token, and completion continuation.
A deadline clamped to 1–900 seconds and created after channel-budget admission. Time waiting for admission is therefore not included in that command deadline.

Only one opening command is advanced through libssh2_channel_open_ex at a time; an EAGAIN leaves that pending open in place. Once opened, the command advances to exec startup and then running. A running turn writes queued stdin, sends EOF when appropriate, drains stdout/stderr within the quantum/deadline, and moves to closing when remote EOF arrives. Closing captures exit status/signal and freeing retries channel release before completing the continuation. Stream output limits count accepted remote bytes per stream, not a single combined decoded-character limit; excess is drained but marked truncated. The output callback runs on the serial owner and must return promptly.

The main pump can be read as this reduced control flow (not replacement code):

begin bounded encrypted-buffer pass; flush queued ciphertext
if encrypted high-water reached: arm socket write, stop this pass
advance one pending channel open
while time remains and ready IDs exist: advance each command state
advance pending open again; service keepalive; flush again
detect socket loss; update read/write interest
if anything progressed or receive budget yielded: wake another sweep

SSHReadyQueue deduplicates IDs, pops FIFO, and places reinserted hot work behind peers. Socket readiness wakes commands; a 100 ms timer also checks command deadlines and keepalive. Thus exec scheduling is readiness-assisted with periodic safety/deadline wakes, not a promise of strict equal byte bandwidth or entirely timer-free progress.

SFTP: the same owner, a different retry path

SFTPClient.connect opens an SSHConnection, creates its primary subsystem channel, and publishes both only if its generation still matches. SFTPChannel.openSiblingChannel reuses the driver's identity and open closure: a sibling gets another subsystem on the same transport, not another socket/login. Channel disconnect closes only that subsystem; client disconnect closes its primary channel and connection.

SSHMultiplexedSFTPChannelOwner centralizes this sequence:

Serialize subsystem initialization with reserveOpen/releaseOpen; acquire the connection's session-channel lease.
On the owning queue, start a bounded buffer pass and flush encrypted output. Return .again under backpressure or libssh2_sftp_init EAGAIN; on success retain pointer and lease in channels[id].
For each raw SFTP operation, enter the same queue, validate the channel/context, begin/flush a pass, perform the operation, flush again, and update socket interest.
On .again, leave the queue and asynchronously sleep for 2 ms before retrying. Shutdown uses the same retry interval. Closing releases the lease and can notify a PTY owner when its last sibling is gone.

This is important when reasoning about fairness: SFTP does not live in the exec worker's ready dictionary. Queue confinement and short retry turns allow PTY/exec and other channel calls to interleave; the SFTP path currently uses async retry sleeps rather than a single unified readiness queue for all operations. Do not describe every transport user as fully event-driven or as receiving a common weighted scheduling guarantee.

For file-level fairness, extra physical connections, and fallback after silent authentication failure, continue with SFTP Transfer Scheduling. Those choices happen above this channel owner.

Bounded turns, immutable retry bytes, and backpressure

SSHForwardingTuning.current supplies a 256 KiB quantum, 256 KiB encrypted receive allowance, 2 ms slice, 2 MiB high-water, and 1 MiB low-water. These are distinct controls: the receive allowance limits what libssh2 consumes during a pass; the deadline limits owner occupancy; the queue watermarks control accumulated bytes. SSHStreamPump resets its quantum for individual directional phases, so 256 KiB is not a hard total byte ceiling for a whole bidirectional turn.

SSHByteQueue uses separately allocated chunks so appending later data cannot move the pointer/length of a write awaiting libssh2 retry. A positive write consumes only acknowledged bytes. Forwarding-to-channel and exec-input chunks use 32,700 bytes because the pinned libssh2 write path clamps there; generic socket chunks use 32 KiB. Vectored socket flush uses up to 16 chunks without coalescing them.

There are two buffer layers:

Buffer	Mechanism	Scope/caveat
Plaintext flow queues (`SSHStreamPump.toChannel` / `toSocket`)	Stop source reads at high-water; resume at or below low-water; retain bytes across partial writes and EAGAIN.	Per direction of one forwarding flow, not a global process cap.
Encrypted connection output (`SSHConnectionBuffer`)	Send callback accepts the full libssh2 packet, queues any unsent remainder, and applies backpressure between library calls. Receive callback returns EAGAIN when backpressured, out of read allowance, or past deadline.	Per SSH connection; packet acceptance can overshoot the watermark, so it is not an exact hard 2 MiB cap.

The encrypted buffer is not merely a throughput optimization: its source comment documents a pinned-library forwarded-tcpip confirmation retry problem when the send callback returns EAGAIN. Accepting whole encrypted writes avoids losing that pending channel while later bounded flushes honor socket backpressure. Teardown switches the buffer into disposal mode: writes are discarded and reads return EOF so session freeing cannot wait for a remote peer.

Wakeups and retirement are correctness mechanisms

Any libssh2 read can decode packets for another channel. The forwarding worker therefore services opens/listeners both before and after stream I/O, checks already-buffered channel data/EOF, and schedules another sweep on progress or budget exhaustion instead of requiring a new socket edge. A zero-window writer can be rescheduled after a window update even if no payload was read. See the exact forwarding algorithm in Pool and Flow Dispatch.

Socket dispatch sources are queue-confined. SSHSocketEvents.cancel resumes suspended sources after cancellation and runs the descriptor-close completion only after both cancellation handlers have retired. The exec worker increments its generation on cleanup, clears queued work, closes SFTP and the channel budget, and rejects stale scheduled pumps/events. Transport errors report connection failure once and fail outstanding commands; one command's channel-local error follows its own close/free path instead.

The Agent's automatic replacement retry is deliberately narrow: AgentConnectionPool.execute retries SSHCommandExecutionError.connectionUnavailable once, not arbitrary errors after a remote command may have started. Multiplexing is not permission to replay a command or an established TCP stream.

Focused verification and diagnostic routing

Question	Source/tests to inspect
Why is the eleventh session channel waiting?	`SSHMaxSessionsPolicyTests`: default capacity, FIFO, live decrease/increase, cancellation, independent transport budgets.
Does EAGAIN preserve bytes and pointers?	`SSHForwardingPumpTests`: blocked write stability, partial writes, encrypted packet buffering and shared flush allowance.
Can a busy/slow flow starve or corrupt a peer?	`SSHForwardingPumpTests`: ready-queue rotation, watermarks, window updates, half-close; `SSHForwardingWireTests`: concurrent streams and slow consumer.
Why did SFTP open another socket rather than a sibling?	File permit's selected mode and `SFTPTransferContext.capability`, not the exec ready queue.
Why does closing the terminal not close SSH yet?	PTY `closeTerminalSession` and last-sibling callback; an SFTP subsystem can still own the transport.
Why was an Agent connection retained or rebuilt?	`AgentConnectionPool` route fingerprint, in-flight join, active leases and idle sweep; `AgentConnectionPoolTests`.

Tests cited here are implementation evidence; they are not a performance guarantee for arbitrary servers/routes. Keep the pinned dependency graph when validating changes, and do not turn the historical forwarding benchmarks into general WAN latency claims.

17 KiB Raw Blame History Unescape Escape