Seep-Reverse-Lab/Tool/skill/apkseep/references/protocol-reverse.md

Protocol reverse engineering

Load this when the app's traffic is not readable by the usual means: the body is protobuf with no schema, the transport is gRPC, the connection is QUIC rather than TCP, or the client refuses to trust your proxy because it validates the certificate inside a native library.

Everything in this file assumes you have already decided the traffic matters and that a proxy is allowed to see it. If a feature-scoped TLS failure has appeared while the rest of the app works, that is a different problem with its own file: read tls-and-cert.md first — it owns the trust-chain attribution and the permissive-defaults template, and this file deliberately does not repeat them.

Claim strength. measured = an exact command and output exist in references/evidence-summary.md §The capability matrix. inferred = documented behaviour or a step that follows directly from a measured one. unverified = reported by others, not reproduced here. The wire-format section is measured on a self-built fixture; every tool named below is unverified unless stated otherwise.

1. Protobuf on the wire (measured)

Protobuf has no field names on the wire and no header. A payload is a flat sequence of tag, value pairs, and a nested message is just a value of type "bytes whose content is another sequence". That is the whole format; the schema supplies meaning and nothing else.

Element	Encoding
Varint	base-128, least-significant group first, high bit set on every byte but the last
Tag	(field_number << 3) | wire_type, itself a varint
Wire type 0	varint value (int32/int64/uint/bool/enum)
Wire type 1	fixed 8 bytes (fixed64/sfixed64/double)
Wire type 2	length-delimited: length varint, then that many bytes (string, bytes, nested message, packed repeated)
Wire type 5	fixed 4 bytes (fixed32/sfixed32/float)
Wire types 3/4	group start/end, legacy; treat as unsupported when walking a modern payload

A host-side check of the rules, including the canonical 150 -> 96 01, was run independently of the shipped decoder — references/evidence-summary.md §The capability matrix records that cross-check against the official runtime. Measured output on the fixture built for it:

message (29 B): 089601120774657374696e671a0508011201782206038e029ea7052800
  field 1   varint            150
  field 2   len-delimited(7)  'testing'
  field 3   len-delimited(5)  nested message:
    field 1   varint            1
    field 2   len-delimited(1)  'x'
  field 4   len-delimited(6)  hex '038e029ea705'
  field 5   varint            0
rebuild from the walked rows equals original: True

Three consequences that decide how you interpret any decoded body:

1. Field numbers are local to their message. In the walk above, field 3 of the outer message is a nested message whose own field 1 exists independently. Nothing on the wire says which message a field number belongs to; only a schema does.
2. A packed repeated field is invisible without a schema. Field 4 above is three varints (3, 270, 86942) packed into one length-delimited value. A schema-free decode can only report it as opaque bytes — the length is known, the element boundaries are not.
3. A decoded 0 and a field that was never set are two different questions. An implicit (no-presence) proto3 field whose value is the default writes nothing at all, so field = 0 and "never touched" produce identical bytes; a field with presence (proto2 optional, proto3 optional) set to 0 does put its tag and a zero byte on the wire. Both halves were measured against the official runtime; a schema-free decoder can only flag the case, never resolve it. So a decoded 0 was emitted on purpose by some writer, and a field missing from a decode is not evidence that 0 was the value in use.

Decoding without a schema

• protoc --decode_raw < body.bin — the reference implementation's own raw decode; prints field numbers and values in exactly the shape above. Note it prints strings and nested messages without distinguishing them, which is why the fallback order matters: try a nested walk first, then printable UTF-8, then hex.
• protobuf-inspector and blackboxprotobuf are the pip-installable equivalents, and both add heuristics: guessing whether a length-delimited value is a string or a submessage, and (for blackboxprotobuf) inferring a candidate type set per field across many samples. Useful when you have traffic but no protoc; still unverified here.
• Decode many samples of the same message, not one. The number of distinct field numbers in a large sample set is what tells you which fields are optional, which repeat, and which are enums whose value set you can then enumerate.
• Anything the app stores locally is a sample source that needs no proxy at all: an OkHttp disk cache (the journal file plus its entries), a DataStore file, or a Room blob column. runtime-data.md covers reading these, and datastore_inject.py can re-encode an edited DataStore value once you know the message.

The decoder that ships here (measured)

scripts/protobuf_decode_raw.py walks a payload with no schema and prints a tree. Input is a hex string, a binary file, or stdin (hex text and raw bytes are told apart automatically); 0x prefixes, spaces, commas, newlines and \xNN escapes are all accepted, and | splits independent frames. --json emits the same tree as data, --max-depth bounds the nesting, and --reencode writes a decoded (possibly hand-edited) tree back to bytes and can diff the result against the original.

$ python skills/apkseep/scripts/protobuf_decode_raw.py --hex "08 96 01 12 07 74657374696e67"
frame 0  bytes 0..12  (12 B)  status=end_of_buffer
  f1  varint  150
  f2  len-delimited(7)  view=utf8_string
      candidate utf8_string      0.50  valid UTF-8 with no control characters
      candidate packed_varint    0.50  the payload is exactly a sequence of 7 varint(s) ...
      candidate bytes            0.20  opaque bytes: always consistent with the payload ...
      tie: packed_varint and utf8_string are equally consistent with these bytes ...
      string: 'testing'

The point of the tool is the candidate list. Fed a nested message that google.protobuf 6.33.6 itself produced (inner payload 0801120178), it answers:

nested_message 0.50 | packed_varint 0.50 | utf8_string 0.35 | bytes 0.20
tie: nested_message and packed_varint are equally consistent ... the view is a display default

Both readings really are legal protobuf for those six bytes, and no heuristic closes the gap. The confidence numbers are a documented ordering, not probabilities, and the view exists only so the tree can be expanded and re-encoded -- it is not a claim about what the field is.

Round trip, which is the strongest evidence available without a schema. Decode, re-encode, and compare bytes: the message serialized by the official runtime re-encodes byte-identically to SerializeToString(); a real AndroidX DataStore file (81 B, written by scripts/datastore_inject.py) round-trips byte-identically; and editing one value in the JSON tree and re-encoding produced a file that the other tool read back as the new value. Two of the built-in fixture families are exceptions by design and are reported as such rather than hidden: a non-canonical varint re-encodes to the minimal form, and a --split varint-length run compares frame bodies only, since the length prefixes are framing and not message data.

What a schema-free decode cannot conclude -- write none of these down:

• that a length-delimited field is a string, a submessage or a packed array, rather than that it could be any of them;
• the element boundaries of a packed array, or its element type;
• that a field's value was 0 because it is absent from the decode, or that a sender "set" a field because a 0 appears (see the two traps above);
• which message a field number belongs to, or that the same field number in two payloads means the same thing;
• that a payload which parses cleanly is a real message at all -- §6 has that failure mode, and the fixes are a second sample and a round trip, not a longer look at the first one.

The measured record for this tool -- every command, both input forms, the round trip, and the framing mistake it does not protect you from -- is in references/evidence-summary.md §The capability matrix.

2. Recovering the schema from the APK

What generated protobuf code looks like

Marker in the decompiled output	What it tells you
extends GeneratedMessageLite / GeneratedMessageV3	protobuf class; Lite means the reflection/schema runtime, not the descriptor-based one
static final int FIELD_NUMBER = 3; (or int <name>_ = 3)	the field number, usually right next to the field's getter
switch (tag >>> 3) with case constants	the parse loop's field dispatch — each case is a field number, in declaration order
input.readInt32(), readInt64(), readBool(), readStringRequireUtf8(), readBytes(), readMessage(...), readEnum()	the field's wire type and semantic type, read after the tag
input.readTag() returning 0 as the loop exit	the standard parsePartialFrom skeleton; the loop body is where all field knowledge lives
writeTo(CodedOutputStream) / getSerializedSize() / dynamicMethod(...)	the class really is a generated message, not a hand-written DTO
newMessageInfo(instance, "<escaped string>", ...)	protobuf-lite's compact schema blob (see below)

The generator keeps those framework calls even after R8, so an obfuscated app usually still carries readable protobuf structure: class and field names are gone, field numbers and wire types are not.

Two routes to a .proto

Route A — read the field table out of the bytecode (inferred). Decompile the class, walk parsePartialFrom, and record case <n> → the read* call that follows. Field name unknown? Name it field_<n>; a valid .proto does not need the original identifiers, and the app's own getter names often survive anyway. Then verify immediately against captured bytes with protoc --decode=<message> <schema.proto>: a decode that produces a coherent structure for every sample is the proof that your field table is right, and it costs seconds. A field that decodes into nonsense means a wrong wire type, not a wrong field number — check the read* call before moving on.

Route B — read protobuf-lite's own schema blob (inferred). protobuf-java-lite 3.x embeds a compact schema string (rawMessageInfo) per message class, consumed by MessageSchema at runtime. Recovering it means locating that blob and feeding it to a matching runtime build. It is a strictly larger win when it works — it carries field numbers, types, and labels in one place — and it is version-sensitive, so treat Route A as the default and Route B as the acceleration. unverified here: no target in this repository was analysed this way.

Tooling that automates this (unverified): pbtk/protodump-style scanners walk dex/JAR inputs looking for the generated-code shape above and emit .proto files, and a protobuf-aware decompiler plugin can do the same from a loaded APK. Use them to produce a first draft, then verify it with protoc --decode against real captured bytes — that verification is the step that turns a guess into a schema, and it is the step worth spending time on.

3. gRPC

gRPC is HTTP/2 with a message framing convention, so the first decision is whether you can see HTTP/2 at all.

• Recognise it. A request carries :method: POST, content-type: application/grpc[+proto] (also +json, +thrift), and :path: /<package>.<Service>/<Method> — that path shape is the reliable fingerprint; the package in it usually matches a package name visible in the APK, which is how you connect traffic to a client class.
• Message framing. Each message is: one compression-flag byte, then a 4-byte big-endian length, then that many bytes of (possibly compressed) protobuf. Several messages can share one DATA frame and one message can span frames, so reassemble by length, not by frame. The call's outcome is in trailers (grpc-status, grpc-message), which many proxies display separately from the body — a 200 with no body and a nonzero grpc-status is a normal gRPC error.
• Read it after capture. Extract the reassembled message bytes and hand them to protoc --decode_raw or your recovered schema. grpcurl -plaintext -protoset file.protoset replays against an endpoint once you have a schema, which is also the fastest way to confirm that a recovered .proto has the right shape.
• Capture it. A transparent TCP proxy sees HTTP/2 fine; a tool that only understands HTTP/1.1 may show the tunnel without the frames, which looks like "the app is not making requests". Interception tooling for this comes in two shapes: a proxy add-on that understands the gRPC framing and prints messages (grpc-dump and similar mitmproxy add-ons, or a gRPC-Web decoder when the client is a browser-style stack), and a command-line client (grpcurl) for replay against a live endpoint. If the app uses TLS (gRPC-Java does by default), the certificate question comes first — see tls-and-cert.md, and the native-material trust chain below.

4. QUIC / HTTP/3

The failure to expect: a working HTTP proxy, a working CA, and an empty capture.

• Why. An HTTP proxy is a TCP endpoint, and a client reaches it with CONNECT. QUIC is a UDP-based transport with TLS 1.3 built into the handshake; there is no TCP connection for a CONNECT to carry and no separate TLS layer for a proxy to terminate.
• Recognise it before debugging anything else. Alt-Svc: h3=":443" in an earlier HTTP response announced the upgrade; on the wire you see UDP traffic to port 443 whose first packets are QUIC Initials (long header, version field, and a TLS ClientHello inside), and the client's ALPN is h3. Check for UDP activity when a TCP proxy stays idle while the app plainly works.
• Options, in order of how little they change:
  1. Force a fallback to TCP. Block or blackhole UDP 443 for the device. A client that cannot reach an origin over QUIC falls back to TCP/HTTP2 if the server also offers it, and then your existing proxy works unchanged. This is the cheapest option and the one to try first, but it is a network-condition change: state it in the report, because "the app failed to load over QUIC" and "the app failed" look identical in a screenshot.
  2. Decrypt with keys, not with a proxy. TLS 1.3 key logging (SSLKEYLOGFILE-style) works for QUIC exactly as it does for TLS-over-TCP, and a packet capture plus the key log can be decrypted in Wireshark. It requires the client to support a key-log callback — Chromium-based stacks and BoringSSL generally do, Java/OkHttp-based stacks generally do not — so this is a per-client property, not a general solution.
  3. Turn HTTP/3 off in the client by hooking whatever enables it (a library version check, a remote-config flag, or a build-time constant). This is a client-side patch and therefore belongs to the repack decision, not to the capture step.
• Everything in this section is inferred/unverified here: no HTTP/3 target was exercised in this repository. Treat the ordering as a plan whose first step (block UDP 443) needs no tool support to test.

5. SSL pinning that lives inside a native library

Three different mechanisms are routinely called "pinning", and only the last one survives a system certificate module.

Mechanism	Where it runs	What defeats it
OkHttp CertificatePinner (or a custom TrustManager) in Java/Kotlin	app's own client	Client-level, not the system path: tls-and-cert.md §Step 4 and §"If OkHttp is the failing path" own this case — do not apply the permissive-defaults template to it
System trust decisions via HttpsURLConnection / platform stack	platform	System store policy — see tls-and-cert.md §Step 5
Verification performed by a library the app ships, e.g. a Flutter app's own BoringSSL, or a bundled libcurl/mbedtls	libflutter.so, libapp.so, another .so	Only an in-memory change to that library's verification path (below), because the library does not consult the platform trust manager you can influence from Java

Flutter, specifically. A Flutter app's dart:io HTTP client runs on the engine's BoringSSL inside libflutter.so; the app's own Dart code can additionally supply a custom trusted-certificate context or a bad-certificate callback. Two consequences that decide which direction to work in:

• If the app is not pinning and simply fails to trust your CA, getting your CA into the system store is the normal fix. MoveCertificate-style Magisk modules exist precisely because Android 7+ ignores user-store CAs for app traffic; moving the CA into the system store is a device change, not an artifact, and it must be labelled as a fallback rather than the deliverable. Its boundary is exactly this: it changes what the platform trust store contains, so it can only help a client that reads that store.
• If the app is pinning, or supplies its own verification context, the system store is irrelevant and the change has to land on the library's verification function.

Locating the verifier inside libflutter.so (unverified — reported method). The published approach in universal-flutter-ssl-pinning (github.com/vichhka-git/universal-flutter-ssl-pinning) automates the anchor search with PyGhidra:

1. load libflutter.so headlessly in Ghidra and scan its defined strings for ssl_client;
2. resolve cross-references from that string to the containing functions;
3. decompile each candidate to count its parameters;
4. take the 3-parameter function — that is BoringSSL's ssl_crypto_x509_session_verify_cert_chain, the certificate-chain verification entry point;
5. emit the discovered RVA into two ready-to-run artifacts: a Frida script that attaches to that address and replaces the return value with 1 (success), and a Renef script that patches the function entry with the ARM64 sequence MOV X0, #1 ; RET.

The repository reports it working on Google Flutter arm64-v8a and Shorebird-patched builds with Frida 16.x/17.x, and notes the RVA is identical across build types for the same engine version. The same project also ships an HTTP monitor that auto-locates BoringSSL's SSL_write/SSL_read in libflutter.so and prints reassembled, de-chunked plaintext — which is the route to take when the traffic cannot be proxied at all, because it never needs a certificate the app will accept.

Two things to verify before trusting that recipe on your target. First, the string anchor is a build artifact: a stripped or repacked engine can lose ssl_client while keeping the function, in which case fall back to the structural search — find the X509_verify_cert-shaped call sequence in the handshake code, or identify the function called with the handshake structure during certificate_verify. Second, the failure mode of a wrong anchor is a silent success: replacing a return value in a function that merely looks like the verifier makes the certificate check pass in a process that then fails somewhere else, which is easy to misread as "the app detected my hook". Always confirm the hook fires on a connection that would otherwise fail, and confirm it is SSL_VERIFY-shaped semantics (the same value the library itself uses for success), rather than assuming 1 is right.

When the string anchor is gone (the session_creator route). A second family of anchors does not depend on the ssl_client literal at all: instead of finding the verifier through a string it references, find it through the call chain that reaches it. BoringSSL's handshake creates the session and then verifies the peer certificate chain, so the target is the function invoked with the handshake/session structure on that path — this is the shape that Flutter-unpinning write-ups call the session-creator route. It survives string and symbol stripping, because the call graph survives, and it costs more: you must identify the handshake structure first, which is why the string anchor is the fast path and this is the fallback when an engine build has dropped the string. unverified here — no engine binary was analysed in this repository.

When the system store is genuinely enough. If the client is Java-side and non-pinning, moving the CA into the system store is the least invasive change and needs no per-library work. The decision is therefore: identify the client first (tls-and-cert.md §Step 4 attribution), then choose the mechanism. Patching a library that was never consulting the store is work with no effect; installing a system CA when a native verifier ignores it produces the same empty capture you started with.

6. Failure modes

A decode that looks valid but is not. protobuf is self-synchronising enough that random bytes often decode into a plausible-looking field list. Guard against it by (a) decoding several samples of the same message and expecting the same field set, (b) checking that nested lengths exactly consume their parents, and (c) when you have a schema, re-encoding the decoded structure and comparing bytes with the input — the same round-trip check used in, which caught a decoder bug in this repository's own fixture.

A framing choice that is wrong without being refused. --split varint-length means protobuf's own writeDelimitedTo framing: a varint length, then the message. gRPC frames a message differently — one compression-flag byte, then a four-byte big-endian length (). Handed a real gRPC frame, the varint splitter reported five frames rather than failing, while the same bytes decoded with no splitting reported stray_end_group. Strip the 5-byte gRPC prefix yourself, and check the frame count either way: a wrong split still produces a decode, which is what makes it expensive.

Assuming the proxy sees everything. The proxy sees what the client sends through it. An app that uses QUIC (), a raw socket, a native client bypassing Java's HTTP stack, or a certificate-pinned connection shows up as silence. Silence is evidence about the transport, not about the app: check UDP activity, check whether a native library is doing the I/O (lib_map.py, dynamic-frida.md), and only then conclude that a feature is client-side.

Patching the wrong verifier. Covered in §5: the failure is silent, and the resulting state is worse than a clean failure because it looks like progress. Verify the hook against a connection known to fail.

Treating a recovered schema as ground truth for the server contract. A schema recovered from the client tells you what the client expects to send and parse — it does not prove the server honours those fields, and it says nothing about authorization. server-api.md owns that question. The recurring expensive mistake is to assume that because a field exists and is settable client-side, the server will act on it.