339 lines
23 KiB
Markdown
339 lines
23 KiB
Markdown
# Protocol reverse engineering
|
|
|
|
**Load this when the app's traffic is not readable by the usual means**: the body is protobuf with no
|
|
schema, the transport is gRPC, the connection is QUIC rather than TCP, or the client refuses to trust
|
|
your proxy because it validates the certificate inside a native library.
|
|
|
|
Everything in this file assumes you have already decided the traffic matters and that a proxy is
|
|
allowed to see it. If a *feature-scoped* TLS failure has appeared while the rest of the app works,
|
|
that is a different problem with its own file: read `tls-and-cert.md` first — it owns the trust-chain
|
|
attribution and the permissive-defaults template, and this file deliberately does not repeat them.
|
|
|
|
**Claim strength.** `measured` = an exact command and output exist in
|
|
`references/evidence-summary.md` §The capability matrix. `inferred` = documented behaviour or a step that
|
|
follows directly from a measured one. `unverified` = reported by others, not reproduced here. The
|
|
wire-format section is `measured` on a self-built fixture; every tool named below is `unverified`
|
|
unless stated otherwise.
|
|
|
|
## 1. Protobuf on the wire (measured)
|
|
|
|
Protobuf has no field names on the wire and no header. A payload is a flat sequence of
|
|
`tag, value` pairs, and a nested message is just a value of type "bytes whose content is another
|
|
sequence". That is the whole format; the schema supplies meaning and nothing else.
|
|
|
|
| Element | Encoding |
|
|
|---|---|
|
|
| Varint | base-128, least-significant group first, high bit set on every byte but the last |
|
|
| Tag | `(field_number << 3) \| wire_type`, itself a varint |
|
|
| Wire type 0 | varint value (int32/int64/uint/bool/enum) |
|
|
| Wire type 1 | fixed 8 bytes (fixed64/sfixed64/double) |
|
|
| Wire type 2 | length-delimited: length varint, then that many bytes (string, bytes, nested message, packed repeated) |
|
|
| Wire type 5 | fixed 4 bytes (fixed32/sfixed32/float) |
|
|
| Wire types 3/4 | group start/end, legacy; treat as unsupported when walking a modern payload |
|
|
|
|
A host-side check of the rules, including the canonical `150 -> 96 01`, was run independently of the
|
|
shipped decoder — `references/evidence-summary.md` §The capability matrix records that cross-check against
|
|
the official runtime. Measured output on the fixture built for it:
|
|
|
|
```
|
|
message (29 B): 089601120774657374696e671a0508011201782206038e029ea7052800
|
|
field 1 varint 150
|
|
field 2 len-delimited(7) 'testing'
|
|
field 3 len-delimited(5) nested message:
|
|
field 1 varint 1
|
|
field 2 len-delimited(1) 'x'
|
|
field 4 len-delimited(6) hex '038e029ea705'
|
|
field 5 varint 0
|
|
rebuild from the walked rows equals original: True
|
|
```
|
|
|
|
Three consequences that decide how you interpret any decoded body:
|
|
|
|
1. **Field numbers are local to their message.** In the walk above, field 3 of the outer message is a
|
|
nested message whose own field 1 exists independently. Nothing on the wire says which message a
|
|
field number belongs to; only a schema does.
|
|
2. **A packed repeated field is invisible without a schema.** Field 4 above is three varints
|
|
(`3, 270, 86942`) packed into one length-delimited value. A schema-free decode can only report it
|
|
as opaque bytes — the length is known, the element boundaries are not.
|
|
3. **A decoded `0` and a field that was never set are two different questions.** An implicit
|
|
(no-presence) proto3 field whose value is the default writes **nothing at all**, so `field = 0`
|
|
and "never touched" produce identical bytes; a field **with** presence (`proto2 optional`,
|
|
`proto3 optional`) set to `0` does put its tag and a zero byte on the wire. Both halves were
|
|
measured against the official runtime; a schema-free decoder can only flag the case, never
|
|
resolve it. So a decoded `0` was emitted on purpose by some writer, and a field *missing* from a
|
|
decode is not evidence that 0 was the value in use.
|
|
|
|
### Decoding without a schema
|
|
|
|
- `protoc --decode_raw < body.bin` — the reference implementation's own raw decode; prints field
|
|
numbers and values in exactly the shape above. Note it prints strings and nested messages without
|
|
distinguishing them, which is why the fallback order matters: try a nested walk first, then
|
|
printable UTF-8, then hex.
|
|
- `protobuf-inspector` and `blackboxprotobuf` are the pip-installable equivalents, and both add
|
|
heuristics: guessing whether a length-delimited value is a string or a submessage, and (for
|
|
`blackboxprotobuf`) inferring a candidate type set per field across many samples. Useful when you
|
|
have traffic but no `protoc`; still unverified here.
|
|
- Decode **many samples of the same message**, not one. The number of distinct field numbers in a
|
|
large sample set is what tells you which fields are optional, which repeat, and which are enums
|
|
whose value set you can then enumerate.
|
|
- Anything the app stores locally is a sample source that needs no proxy at all: an OkHttp disk cache
|
|
(the `journal` file plus its entries), a DataStore file, or a Room blob column. `runtime-data.md`
|
|
covers reading these, and `datastore_inject.py` can re-encode an edited DataStore value once you
|
|
know the message.
|
|
|
|
### The decoder that ships here (measured)
|
|
|
|
`scripts/protobuf_decode_raw.py` walks a payload with no schema and prints a tree. Input is a hex
|
|
string, a binary file, or stdin (hex text and raw bytes are told apart automatically); `0x` prefixes,
|
|
spaces, commas, newlines and `\xNN` escapes are all accepted, and `|` splits independent frames.
|
|
`--json` emits the same tree as data, `--max-depth` bounds the nesting, and `--reencode` writes a
|
|
decoded (possibly hand-edited) tree back to bytes and can diff the result against the original.
|
|
|
|
```
|
|
$ python skills/apkseep/scripts/protobuf_decode_raw.py --hex "08 96 01 12 07 74657374696e67"
|
|
frame 0 bytes 0..12 (12 B) status=end_of_buffer
|
|
f1 varint 150
|
|
f2 len-delimited(7) view=utf8_string
|
|
candidate utf8_string 0.50 valid UTF-8 with no control characters
|
|
candidate packed_varint 0.50 the payload is exactly a sequence of 7 varint(s) ...
|
|
candidate bytes 0.20 opaque bytes: always consistent with the payload ...
|
|
tie: packed_varint and utf8_string are equally consistent with these bytes ...
|
|
string: 'testing'
|
|
```
|
|
|
|
The point of the tool is the candidate list. Fed a nested message that `google.protobuf` 6.33.6
|
|
itself produced (inner payload `0801120178`), it answers:
|
|
|
|
```
|
|
nested_message 0.50 | packed_varint 0.50 | utf8_string 0.35 | bytes 0.20
|
|
tie: nested_message and packed_varint are equally consistent ... the view is a display default
|
|
```
|
|
|
|
Both readings really are legal protobuf for those six bytes, and no heuristic closes the gap. The
|
|
confidence numbers are a documented ordering, not probabilities, and the `view` exists only so the
|
|
tree can be expanded and re-encoded -- it is not a claim about what the field is.
|
|
|
|
**Round trip, which is the strongest evidence available without a schema.** Decode, re-encode, and
|
|
compare bytes: the message serialized by the official runtime re-encodes byte-identically to
|
|
`SerializeToString()`; a real AndroidX DataStore file (81 B, written by `scripts/datastore_inject.py`)
|
|
round-trips byte-identically; and editing one value in the JSON tree and re-encoding produced a file
|
|
that the *other* tool read back as the new value. Two of the built-in fixture families are exceptions
|
|
by design and are reported as such rather than hidden: a non-canonical varint re-encodes to the
|
|
minimal form, and a `--split varint-length` run compares frame bodies only, since the length prefixes
|
|
are framing and not message data.
|
|
|
|
**What a schema-free decode cannot conclude -- write none of these down:**
|
|
|
|
- that a length-delimited field *is* a string, a submessage or a packed array, rather than that it
|
|
could be any of them;
|
|
- the element boundaries of a packed array, or its element type;
|
|
- that a field's value was 0 because it is absent from the decode, or that a sender "set" a field
|
|
because a 0 appears (see the two traps above);
|
|
- which message a field number belongs to, or that the same field number in two payloads means the
|
|
same thing;
|
|
- that a payload which parses cleanly is a real message at all -- §6 has that failure mode, and the
|
|
fixes are a second sample and a round trip, not a longer look at the first one.
|
|
|
|
The measured record for this tool -- every command, both input forms, the round trip, and the
|
|
framing mistake it does *not* protect you from -- is in
|
|
`references/evidence-summary.md` §The capability matrix.
|
|
|
|
## 2. Recovering the schema from the APK
|
|
|
|
### What generated protobuf code looks like
|
|
|
|
| Marker in the decompiled output | What it tells you |
|
|
|---|---|
|
|
| `extends GeneratedMessageLite` / `GeneratedMessageV3` | protobuf class; `Lite` means the reflection/schema runtime, not the descriptor-based one |
|
|
| `static final int FIELD_NUMBER = 3;` (or `int <name>_ = 3`) | the field number, usually right next to the field's getter |
|
|
| `switch (tag >>> 3)` with `case` constants | the parse loop's field dispatch — each `case` is a field number, in declaration order |
|
|
| `input.readInt32()`, `readInt64()`, `readBool()`, `readStringRequireUtf8()`, `readBytes()`, `readMessage(...)`, `readEnum()` | the field's wire type and semantic type, read *after* the tag |
|
|
| `input.readTag()` returning 0 as the loop exit | the standard `parsePartialFrom` skeleton; the loop body is where all field knowledge lives |
|
|
| `writeTo(CodedOutputStream)` / `getSerializedSize()` / `dynamicMethod(...)` | the class really is a generated message, not a hand-written DTO |
|
|
| `newMessageInfo(instance, "<escaped string>", ...)` | protobuf-lite's compact schema blob (see below) |
|
|
|
|
The generator keeps those framework calls even after R8, so an obfuscated app usually still carries
|
|
readable protobuf structure: class and field *names* are gone, field *numbers* and wire types are
|
|
not.
|
|
|
|
### Two routes to a `.proto`
|
|
|
|
**Route A — read the field table out of the bytecode (inferred).** Decompile the class, walk
|
|
`parsePartialFrom`, and record `case <n>` → the `read*` call that follows. Field name unknown? Name
|
|
it `field_<n>`; a valid `.proto` does not need the original identifiers, and the app's own getter
|
|
names often survive anyway. Then verify immediately against captured bytes with
|
|
`protoc --decode=<message> <schema.proto>`: a decode that produces a coherent structure for **every**
|
|
sample is the proof that your field table is right, and it costs seconds. A field that decodes into
|
|
nonsense means a wrong wire type, not a wrong field number — check the `read*` call before moving on.
|
|
|
|
**Route B — read protobuf-lite's own schema blob (inferred).** protobuf-java-lite 3.x embeds a
|
|
compact schema string (`rawMessageInfo`) per message class, consumed by `MessageSchema` at runtime.
|
|
Recovering it means locating that blob and feeding it to a matching runtime build. It is a strictly
|
|
larger win when it works — it carries field numbers, types, and labels in one place — and it is
|
|
version-sensitive, so treat Route A as the default and Route B as the acceleration. `unverified`
|
|
here: no target in this repository was analysed this way.
|
|
|
|
**Tooling that automates this (unverified):** `pbtk`/`protodump`-style scanners walk dex/JAR inputs
|
|
looking for the generated-code shape above and emit `.proto` files, and a protobuf-aware decompiler
|
|
plugin can do the same from a loaded APK. Use them to produce a first draft, then verify it with
|
|
`protoc --decode` against real captured bytes — that verification is the step that turns a guess into
|
|
a schema, and it is the step worth spending time on.
|
|
|
|
## 3. gRPC
|
|
|
|
gRPC is HTTP/2 with a message framing convention, so the first decision is whether you can see
|
|
HTTP/2 at all.
|
|
|
|
- **Recognise it.** A request carries `:method: POST`, `content-type: application/grpc[+proto]` (also
|
|
`+json`, `+thrift`), and `:path: /<package>.<Service>/<Method>` — that path shape is the reliable
|
|
fingerprint; the package in it usually matches a package name visible in the APK, which is how you
|
|
connect traffic to a client class.
|
|
- **Message framing.** Each message is: one compression-flag byte, then a 4-byte **big-endian**
|
|
length, then that many bytes of (possibly compressed) protobuf. Several messages can share one
|
|
DATA frame and one message can span frames, so reassemble by length, not by frame. The call's
|
|
outcome is in **trailers** (`grpc-status`, `grpc-message`), which many proxies display separately
|
|
from the body — a `200` with no body and a nonzero `grpc-status` is a normal gRPC error.
|
|
- **Read it after capture.** Extract the reassembled message bytes and hand them to
|
|
`protoc --decode_raw` or your recovered schema. `grpcurl -plaintext -protoset file.protoset`
|
|
replays against an endpoint once you have a schema, which is also the fastest way to confirm that
|
|
a recovered `.proto` has the right shape.
|
|
- **Capture it.** A transparent TCP proxy sees HTTP/2 fine; a tool that only understands HTTP/1.1 may
|
|
show the tunnel without the frames, which looks like "the app is not making requests". Interception
|
|
tooling for this comes in two shapes: a proxy add-on that understands the gRPC framing and prints
|
|
messages (`grpc-dump` and similar mitmproxy add-ons, or a gRPC-Web decoder when the client is a
|
|
browser-style stack), and a command-line client (`grpcurl`) for replay against a live endpoint. If
|
|
the app uses TLS (gRPC-Java does by default), the certificate question comes first — see
|
|
`tls-and-cert.md`, and the native-material trust chain below.
|
|
|
|
## 4. QUIC / HTTP/3
|
|
|
|
The failure to expect: a working HTTP proxy, a working CA, and **an empty capture**.
|
|
|
|
- **Why.** An HTTP proxy is a TCP endpoint, and a client reaches it with `CONNECT`. QUIC is a
|
|
UDP-based transport with TLS 1.3 built into the handshake; there is no TCP connection for a
|
|
`CONNECT` to carry and no separate TLS layer for a proxy to terminate.
|
|
- **Recognise it before debugging anything else.** `Alt-Svc: h3=":443"` in an earlier HTTP response
|
|
announced the upgrade; on the wire you see UDP traffic to port 443 whose first packets are QUIC
|
|
Initials (long header, version field, and a TLS ClientHello inside), and the client's ALPN is `h3`.
|
|
Check for UDP activity when a TCP proxy stays idle while the app plainly works.
|
|
- **Options, in order of how little they change:**
|
|
1. **Force a fallback to TCP.** Block or blackhole UDP 443 for the device. A client that cannot
|
|
reach an origin over QUIC falls back to TCP/HTTP2 if the server also offers it, and then your
|
|
existing proxy works unchanged. This is the cheapest option and the one to try first, but it is
|
|
a **network-condition change**: state it in the report, because "the app failed to load over
|
|
QUIC" and "the app failed" look identical in a screenshot.
|
|
2. **Decrypt with keys, not with a proxy.** TLS 1.3 key logging (`SSLKEYLOGFILE`-style) works for
|
|
QUIC exactly as it does for TLS-over-TCP, and a packet capture plus the key log can be
|
|
decrypted in Wireshark. It requires the client to support a key-log callback — Chromium-based
|
|
stacks and BoringSSL generally do, Java/OkHttp-based stacks generally do not — so this is a
|
|
per-client property, not a general solution.
|
|
3. **Turn HTTP/3 off in the client** by hooking whatever enables it (a library version check, a
|
|
remote-config flag, or a build-time constant). This is a client-side patch and therefore
|
|
belongs to the repack decision, not to the capture step.
|
|
- Everything in this section is `inferred`/`unverified` here: no HTTP/3 target was exercised in this
|
|
repository. Treat the ordering as a plan whose first step (block UDP 443) needs no tool support to
|
|
test.
|
|
|
|
## 5. SSL pinning that lives inside a native library
|
|
|
|
Three different mechanisms are routinely called "pinning", and only the last one survives a system
|
|
certificate module.
|
|
|
|
| Mechanism | Where it runs | What defeats it |
|
|
|---|---|---|
|
|
| OkHttp `CertificatePinner` (or a custom `TrustManager`) in Java/Kotlin | app's own client | Client-level, **not** the system path: `tls-and-cert.md` §Step 4 and §"If OkHttp is the failing path" own this case — do not apply the permissive-defaults template to it |
|
|
| System trust decisions via `HttpsURLConnection` / platform stack | platform | System store policy — see `tls-and-cert.md` §Step 5 |
|
|
| Verification performed by a **library the app ships**, e.g. a Flutter app's own BoringSSL, or a bundled `libcurl`/`mbedtls` | `libflutter.so`, `libapp.so`, another `.so` | Only an in-memory change to that library's verification path (below), because the library does not consult the platform trust manager you can influence from Java |
|
|
|
|
**Flutter, specifically.** A Flutter app's `dart:io` HTTP client runs on the engine's BoringSSL
|
|
inside `libflutter.so`; the app's own Dart code can additionally supply a custom trusted-certificate
|
|
context or a bad-certificate callback. Two consequences that decide which direction to work in:
|
|
|
|
- If the app is **not** pinning and simply fails to trust your CA, getting your CA into the **system**
|
|
store is the normal fix. `MoveCertificate`-style Magisk modules exist precisely because Android 7+
|
|
ignores user-store CAs for app traffic; moving the CA into the system store is a **device change**,
|
|
not an artifact, and it must be labelled as a fallback rather than the deliverable. Its boundary is
|
|
exactly this: it changes what the platform trust store contains, so it can only help a client that
|
|
reads that store.
|
|
- If the app **is** pinning, or supplies its own verification context, the system store is irrelevant
|
|
and the change has to land on the library's verification function.
|
|
|
|
**Locating the verifier inside `libflutter.so` (unverified — reported method).** The published
|
|
approach in
|
|
[universal-flutter-ssl-pinning](https://github.com/vichhka-git/universal-flutter-ssl-pinning)
|
|
(`github.com/vichhka-git/universal-flutter-ssl-pinning`) automates the anchor search with PyGhidra:
|
|
|
|
1. load `libflutter.so` headlessly in Ghidra and scan its defined strings for `ssl_client`;
|
|
2. resolve cross-references from that string to the **containing functions**;
|
|
3. decompile each candidate to count its parameters;
|
|
4. take the **3-parameter** function — that is BoringSSL's
|
|
`ssl_crypto_x509_session_verify_cert_chain`, the certificate-chain verification entry point;
|
|
5. emit the discovered RVA into two ready-to-run artifacts: a Frida script that attaches to that
|
|
address and replaces the return value with `1` (success), and a Renef script that patches the
|
|
function entry with the ARM64 sequence `MOV X0, #1 ; RET`.
|
|
|
|
The repository reports it working on Google Flutter arm64-v8a and Shorebird-patched builds with
|
|
Frida 16.x/17.x, and notes the RVA is identical across build types for the same engine version. The
|
|
same project also ships an HTTP monitor that auto-locates BoringSSL's `SSL_write`/`SSL_read` in
|
|
`libflutter.so` and prints reassembled, de-chunked plaintext — which is the route to take when the
|
|
traffic cannot be proxied at all, because it never needs a certificate the app will accept.
|
|
|
|
**Two things to verify before trusting that recipe on your target.** First, the string anchor is a
|
|
build artifact: a stripped or repacked engine can lose `ssl_client` while keeping the function, in
|
|
which case fall back to the structural search — find the `X509_verify_cert`-shaped call sequence in
|
|
the handshake code, or identify the function called with the handshake structure during
|
|
`certificate_verify`. Second, the failure mode of a wrong anchor is a **silent** success: replacing a
|
|
return value in a function that merely *looks* like the verifier makes the certificate check pass
|
|
in a process that then fails somewhere else, which is easy to misread as "the app detected my hook".
|
|
Always confirm the hook fires on a connection that would otherwise fail, and confirm it is
|
|
`SSL_VERIFY`-shaped semantics (the same value the library itself uses for success), rather than
|
|
assuming `1` is right.
|
|
|
|
**When the string anchor is gone (the `session_creator` route).** A second family of anchors does not
|
|
depend on the `ssl_client` literal at all: instead of finding the verifier through a string it
|
|
references, find it through the call chain that reaches it. BoringSSL's handshake creates the session
|
|
and then verifies the peer certificate chain, so the target is the function invoked with the
|
|
handshake/session structure on that path — this is the shape that Flutter-unpinning write-ups call the
|
|
session-creator route. It survives string and symbol stripping, because the call graph survives, and
|
|
it costs more: you must identify the handshake structure first, which is why the string anchor is the
|
|
fast path and this is the fallback when an engine build has dropped the string. `unverified` here — no
|
|
engine binary was analysed in this repository.
|
|
|
|
**When the system store is genuinely enough.** If the client is Java-side and non-pinning, moving the
|
|
CA into the system store is the least invasive change and needs no per-library work. The decision is
|
|
therefore: identify the client first (`tls-and-cert.md` §Step 4 attribution), then choose the
|
|
mechanism. Patching a library that was never consulting the store is work with no effect; installing
|
|
a system CA when a native verifier ignores it produces the same empty capture you started with.
|
|
|
|
## 6. Failure modes
|
|
|
|
**A decode that looks valid but is not.** protobuf is self-synchronising enough that random bytes
|
|
often decode into a plausible-looking field list. Guard against it by (a) decoding several samples of
|
|
the same message and expecting the same field set, (b) checking that nested lengths exactly consume
|
|
their parents, and (c) when you have a schema, re-encoding the decoded structure and comparing bytes
|
|
with the input — the same round-trip check used in, which caught a decoder bug in this
|
|
repository's own fixture.
|
|
|
|
**A framing choice that is wrong without being refused.** `--split varint-length` means protobuf's own
|
|
`writeDelimitedTo` framing: a varint length, then the message. gRPC frames a message differently — one
|
|
compression-flag byte, then a **four-byte big-endian** length (). Handed a real gRPC frame, the
|
|
varint splitter reported **five** frames rather than failing, while the same bytes decoded with no
|
|
splitting reported `stray_end_group`. Strip the 5-byte gRPC prefix yourself, and check the frame count
|
|
either way: a wrong split still produces a decode, which is what makes it expensive.
|
|
|
|
**Assuming the proxy sees everything.** The proxy sees what the client sends through it. An app that
|
|
uses QUIC (), a raw socket, a native client bypassing Java's HTTP stack, or a certificate-pinned
|
|
connection shows up as silence. Silence is evidence about the transport, not about the app: check
|
|
UDP activity, check whether a native library is doing the I/O (`lib_map.py`, `dynamic-frida.md`),
|
|
and only then conclude that a feature is client-side.
|
|
|
|
**Patching the wrong verifier.** Covered in §5: the failure is silent, and the resulting state is
|
|
worse than a clean failure because it looks like progress. Verify the hook against a connection
|
|
known to fail.
|
|
|
|
**Treating a recovered schema as ground truth for the server contract.** A schema recovered from the
|
|
client tells you what the client expects to send and parse — it does not prove the server honours
|
|
those fields, and it says nothing about authorization. `server-api.md` owns that question. The
|
|
recurring expensive mistake is to assume that because a field exists and is settable client-side, the
|
|
server will act on it.
|