vpsmgr/docs/ipv6.md

# IPv6 pass-through

Two modes, chosen at install:

- **Prefix mode** (classic): no NAT, each container owns a global /112 block
  cut from the provider-routed prefix for its account (see below).
- **Pool mode**: no routable whole prefix, but the host has multiple global
  addresses on its NIC (the "discrete whitelist" providers). Each container is
  assigned one address from a confirmed pool; the host keeps the first address
  for itself.

## Prefix mode (classic)

Optional, **no NAT**: each container owns a global /112 block and the outside
can reach any address it binds directly. Enabled by setting
`net.ipv6_subnet` (asked at install; see [configuration.md](configuration.md)).

## Per-container /112 block

A block is built from a **32-bit index stored with the account** (`users.
ipv6_index`), stable across reinstalls and independent of the username:

```
block = [configured prefix][32-bit index][16 host bits]
                            bits 80-111    bits 112-127
```

- The index is **random**, picked when the account is created (and retried
  internally if the value is somehow taken — the `UNIQUE` index on the column is
  the backstop). It used to be `sha256(username)[0:4]`, which made the suffix of
  every address a global constant for a given name: the same `alice` produced
  the same block on every host running the panel, so an address identified its
  owner to anyone who knew the scheme and could enumerate names. Accounts that
  predate the column were seeded with the value they had been using, so their
  addresses never moved.
- Example (`2602:fada:6::/64`, index `0x2bd806c9`): block
  `2602:fada:6::2bd8:6c9:0/112`, primary address `2602:fada:6::2bd8:6c9:1`.
- The primary address is **byte-identical to the pre-/112 scheme** for an
  account seeded with its old value, so upgrading never changes an existing
  container's address.
- The block index is per-host: it is **not** carried across machines by
  `vps transfer`, because the receiving host numbers its own containers from its
  own prefix anyway. Deleting an account and creating one with the same name
  yields a fresh random index.
- A block never contains the bridge gateway address: an index whose block does
  is skipped when picking (the gateway lives in the all-zero block, which is why
  `0` also means "no index" — every pool-mode and V4-only account).

## Supported prefixes

`/48` .. `/80` (an explicit length is required in `ipv6_subnet`).

| Prefix | Bridge uses | Notes |
|---|---|---|
| `/48` `/56` `/60` | **first /64 of the prefix** | Incus's dnsmasq rejects non-/64 networks, and every block falls inside the first /64 anyway (bits `[prefixlen:79]` are zero-filled). |
| `/64` | the /64 itself | |
| `/80` | the /80 itself | Common provider slice (e.g. AWS ENI /80). |

The bridge prefix length is clamped with `min(ones, 64)`-equivalent logic
(`bridgePrefixLen`); `SetupIPv6Bridge` (run by `vps install` and on boot)
applies it.

## Bridge setup (`SetupIPv6Bridge`)

The bridge (`incusbr0`) gets:

```
ipv6.address = <gw>/<len>     # len = bridgePrefixLen(ones)
ipv6.nat = false
ipv6.routing = true
ipv6.dhcp.stateful = true
```

The gateway is the first free address in the prefix (`net+1`, `net+2`, ...)
that the host can see is already taken:

1. addresses assigned to the host's external interface,
2. the upstream default gateway(s),
3. any address in the NDP neighbor table on the external interface.

The conflicting prefix route Incus auto-creates on the bridge is deleted (eth0
keeps the authoritative route), and IPv6 forwarding is enabled.

## Per-container wiring

For each container:

- The `eth0` device sets `ipv6.address=<block>::1` and
  `ipv6.routes=<block>::/112`, so Incus routes the whole block to the container.
  Any address inside the /112 that the container binds is delivered to it.
- The primary `/128` is **bound statically** inside the container (a networkd
  `[Address]` section on Debian, a boot-time service on RHEL) — it does not
  depend on DHCPv6. This matters on reinstall: Incus's dnsmasq keeps the deleted
  container's DHCPv6 lease for its primary address for up to an hour, so
  DHCPv6 would hand the recreated container a *dynamic* address instead, which
  falls outside the routed /112 and is dropped by `ipv6_filtering`. Binding the
  /128 directly makes IPv6 survive reinstalls.
- The gateway (`fe80::1`, the bridge) is **pinned as a static neighbor** in the
  guest — the bridge's MAC is baked in at provisioning time. The guest cannot
  resolve it by NDP: `security.ipv6_filtering` drops a neighbor solicitation
  whose source is outside the routed /112, and that solicitation is sourced from
  the guest's link-local.
- DHCPv6 is turned off on Debian (`DHCP=ipv4` plus `[IPv6AcceptRA] DHCPv6Client=no`
  — the RA's Managed flag would otherwise start the DHCPv6 client regardless of
  `DHCP=`), and the RA is told to generate no SLAAC address, so the container
  never ends up with a stray address outside its /112. RA
  `UseOnLinkPrefix=false` / `UseRoutePrefix=false` keep the parent prefix
  off-link, so a container reaches a peer through the host (its default
  gateway) instead of direct L2 neighbour discovery.
- `ndppd` proxies Neighbor Discovery on the **external** interface for every
  /112: an upstream neighbor solicitation for an address in a block is relayed
  to the bridge, the container answers, and ndppd relays the NA back. Kernel
  `proxy_ndp` is not used for prefixes — it only answers single addresses
  (route-covered or prefix queries are ignored).

vpsmgr renders `/etc/ndppd.conf` (one `rule <block>::/112` per container) and
restarts the daemon on `add`/`del`; the config is rebuilt from the DB at boot
by `vps-ipv6.service` / `vps ipv6-reapply` and by `vps install`, so
rules survive reboots. `vps ipv6-reapply` also re-applies the per-container
routed-IPv6 config (self-healing: containers created before the host-routed
scheme, or whose networkd config was corrupted, are repaired on every boot).

## Upstream gateway neighbour pin (`net.ipv6_pin_gateway`, optional)

Off by default. A container's IPv6 rides on the host's own IPv6: the host reaches
the internet through a default route whose nexthop is the provider's link-local
gateway, and a container's packets are forwarded through it. Some providers'
routers answer neighbour solicitations unreliably, so the host's neighbour entry
for that gateway drops to `INCOMPLETE` — and until it recovers, the host answers
every forwarded packet, the containers' included, with *address unreachable*.

The symptom is distinctive: the container keeps its address, can ping the
bridge, but **nothing off-link works, intermittently** (and the host's own IPv6
drops at the same moments). No container-side config fixes it, because the
container is not at fault.

`net.ipv6_pin_gateway: true` makes that gateway's neighbour entry **permanent**,
so the kernel never probes it again and the failure cannot occur.
`RewireAllIPv6` applies it — so it is re-applied at boot by
`vps-ipv6.service` and by `vps install` — and it only ever touches the host's own
**link-local default gateway**, and only while that gateway is resolved (the MAC
is read from the live neighbour table; an unresolved gateway is left alone).
Turning the key off again drops the permanent entry on the next pass.

The catch, and the reason it is opt-in: a permanent entry is never re-resolved,
so a gateway that later moves to a **different MAC** would be missed and the
host's upstream would black-hole silently. On a provider that behaves, the
default (off) is the right setting.

## Installer flow

- `00-ip-ask.sh` — the install-time network asks: whether to enable IPv6
  (captures the prefix) and the container subnet's second octet (default 115).
  The prefix length is **required** (no silent `/64` default). On reinstall it
  reuses an existing config's `ipv6_subnet` / `subnet` instead of re-asking.
- `install.sh` — installs `ndppd` (only when IPv6 is enabled; it is not part
  of the default small install otherwise).
- `10-incus.sh` — creates `incusbr0` without an IPv6 address (the address is
  chosen clash-free by `SetupIPv6Bridge` at `vps install`).
- `20-network.sh` — enables IPv6 forwarding.
- `50-image.sh` — bakes the Debian networkd IPv6 config (`DHCP=ipv4`,
  `[IPv6AcceptRA]` off-link/no-SLAAC/no-DHCPv6) into the published image.
- `60-rhel-image.sh` — bakes the RHEL kernel-managed IPv6 plumbing (sysctls for
  the RA default route without the on-link prefix, plus the `vpsmgr-ipv6`
  helper and boot unit). The runtime script installs these idempotently if an
  older image lacks them, so a pre-fix image still gets working IPv6.
- `vps install` / `add` / `reinstall` — apply the per-container config
  (`ConfigureContainerIPv6`); `ipv6-reapply` covers existing containers.
- `check-ipv6-support.sh` — probe before install: reports the host's global
  addresses, derives a candidate prefix from the on-link routed block when the
  kernel route table shows one (e.g. an AWS /80), falling back to the
  address's own configured length, and verifies from the outside (Globalping,
  free) that the provider actually routes the prefix to the host.

## Isolation interplay

Container isolation (see [architecture.md](architecture.md)) is unaffected:
`security.ipv6_filtering` whitelists the whole routed /112 (observed on Incus
5.21), so a container may source packets from any address in its block, and
nothing else. Containers cannot reach each other on the private bridge —
v6 included — so inter-container traffic must go via public addresses; the
host does not proxy the private subnet.

## Uninstall cleanup

`uninstall.sh` reads the prefix from the config before removal, then: stops
and disables `ndppd` / `npd6` and removes `/etc/ndppd.conf`, removes any leftover
kernel `proxy_ndp` entries and `/128` routes matching the prefix, resets
`incusbr0` IPv6 to disabled, and restores forwarding sysctls.

## Pool mode (per-address pool)

For providers that hand out **discrete global addresses** (a whitelist in the
control panel) instead of a routable whole prefix — e.g. 15 addresses in one
/64 that is itself NOT routed as a whole (verified externally: a random
address inside the /64 gets no reply, while every whitelisted address does).

### Installer flow

`00-ip-ask.sh` runs the unchanged `check-ipv6-support.sh`:

1. Whole prefix **VERIFIED** → classic prefix mode (as before).
2. Not verified → the installer asks: **pool mode** (empty pool; addresses are
   added later in the admin panel), **manual prefix** (user types a prefix
   they know is routed — trusted as-is), or **disable** (pure IPv4).
   Whitelist providers typically only bind one address on the NIC at boot, so
   there is nothing to auto-collect at install time — the pool starts empty by
   design and the admin fills it from the provider's control-panel list.

The mode is fixed at install (`net.ipv6_mode`, immutable like `net.subnet`);
the **pool itself is editable** (`net.ipv6_pool`, via the admin panel's IPv6
Pool page or `vps config set`).

### Configuration

```yaml
net:
  ipv6_mode: pool
  ipv6_pool:
    - "2001:db8:1::9c4"
    - "2001:db8:1::9c5"
    ...
```

`ipv6_pool` entries are global addresses, bare or with an explicit `/128`
(any other prefix length is rejected). The host keeps its own address for
itself (not in the pool).

### Admin panel pool management

The admin panel's **IPv6 Pool** page (visible in pool mode) lets the operator:

- **Batch-add** addresses from a multi-line textarea (one per line, bare or
  `/128`). Invalid entries (bad prefix, ULA, private, duplicates) reject the
  whole batch.
- **List** the pool with per-address state: **free** or **used** (by which
  user).
- **Remove** a free address (an address assigned to a user is refused — the
  user keeps it for life).

Adding addresses re-applies the host plumbing (any address the provider
bound on the external interface is detached), so newly added addresses are
immediately usable.

### Assignment

- Each container is assigned **one address from the pool**, stored in the
  `users.ipv6_address` column (UNIQUE index = one address can never be given
  to two users; the reservation and the user row are written in one
  transaction).
- The address **belongs to the user for life** (reinstalls keep it); it is
  released only when the user is deleted (the row dies, the address is free
  again).
- The admin panel's create form has a dropdown: auto (first free), a specific
  free address, or **no IPv6** (a V4-only container). `vps add` without extra
  flags always auto-assigns the first free address; when the pool is
  exhausted, further creates simply have no IPv6 (never an error).

### Routing (empirically verified)

Each pool container gets **two NICs**:

- **eth0** — `nictype: routed`, `parent: <ext_if>`, `ipv6.address=<pool /128>`.
  Incus creates a veth pair, sets the host side to `fe80::1` (the container's
  default gateway), adds a `/128` route to the veth for the address, and
  installs a `proxy_ndp` entry on the external interface. This is the same
  mechanism `ipvlan`/`routed` NICs use to join an external network without a
  bridge.
- **eth1** — `nictype: bridged` on `incusbr0`, `ipv4.address=<private v4>`.
  Carries the shared IPv4 (SSH DNAT, user ports, NAT4 outbound) exactly like
  prefix mode.

The container's systemd-networkd binds the `/128` statically on eth0 with
`fe80::1` as the default route (no RA, no DHCPv6), and runs DHCPv4 on eth1.

**Critical prerequisite — the pool address must NOT be bound on the host's
external interface.** The whitelist provider assigns every address to the host
at boot; while the host holds an address, the kernel treats it as a LOCAL
address and drops the container's packets that use it as a source
(source-address validation) — outbound routing fails while inbound may work
by luck. Pool mode therefore removes each pool address from the external
interface (`vps ip6 addr-del`), on add/reinstall and on every
`ipv6-reapply`/boot (self-healing: a reboot that re-binds the addresses on
eth0 is corrected on the next reapply). The host keeps its own first address.

Verified on a whitelist provider host (address NOT bound on eth0, container
on a routed NIC): external Globalping probes reach the container's /128 with
**0% loss** (5/5 probes), and the container reaches out over IPv6 (curl -6)
and IPv4 (eth1 NAT) simultaneously. No ndppd, no manual per-address routes —
Incus programs everything.

### Interaction with IPv4

Pool mode works with either `v4_forward` setting. When the pool is exhausted
(or the admin picks "no IPv6"), containers are plain V4-only boxes — same as
the pre-IPv6 behavior.

## Whole /64 blocks from an extra prefix (optional)

A provider that delegates a prefix **shorter than /64** (a /60, /56 or /48) makes
one thing possible that a plain /64 host cannot do: hand a container a *whole
/64*. The /112 block a prefix-mode account normally owns has 16 host bits, so
nothing inside the container can use SLAAC (which needs exactly 64 bits) or be
delegated any further.

The feature is **off by default and stays off on every upgrade**: it exists only
while `net.ipv6_extra_prefix` is set (see [configuration.md](configuration.md)).
It works in both **prefix mode** and **IPv4-only (none) mode**. Nothing is
allocated and no panel UI appears otherwise, and the installer never asks for it
— an operator turns it on deliberately.

### What a container gets

- In **prefix mode**: its `/112` primary address and everything around it, plus a whole `/64`.
- In **none mode**: a standalone whole `/64` (e.g. `2001:1c00:b1b:7f0::/64`), delivered alongside its private IPv4. No `/112` is assigned.
- In both modes, the block is bound inside the container as `<block>::1/64` — with its real length, so the prefix is on-link there, any address in it is usable, and the customer can carve sub-prefixes out of it for internal networks. It is deliberately **not** a local route: that would claim every address in the block for the container itself and make delegation impossible.
- The host routes the block to the container, and the in-tree NDP responder covers it for an upstream that resolves prefixes by NDP.

### How the route is wired

- **Prefix mode (single bridged NIC)**:
  The block is declared on the container's own NIC — `ipv6.routes = <block>/64, <account's /112>` — and the device keeps its `ipv6.address`.
- **None mode (routed IPv6 + bridged IPv4)**:
  Because the bridge has no IPv6 subnet, the container receives two NICs (the same layout as pool mode):
  - `eth0` — `nictype: routed`, `parent: <ext_if>`, `ipv6.address=<block>::1`, `ipv6.routes=<block>/64`.
  - `eth1` — `nictype: bridged` on `incusbr0`, `ipv4.address=<private v4>`.
  Inside the guest, `eth0` binds `<block>::1/64` with default gateway `fe80::1`, and `eth1` runs DHCPv4.

In prefix mode, three things about the eth0 declaration are load-bearing:

- Incus builds `security.ipv6_filtering` from the addresses and routes a NIC
  declares, and that filter drops everything else. A block that is not declared
  is a block the bridge throws away: the container binds its addresses and gets
  replies from the host, but every packet it sources from the block dies at the
  veth.
- `ipv6.address` stays declared, because that is what keeps the account's
  `/112` usable. Incus programs a declared route as `via <ipv6.address>`, i.e.
  straight to the container's MAC, and the guest makes its whole `/112` local —
  so every address in it answers, but only as long as the host hands the packet
  to the container directly. Without the address the routes become direct
  (`dev <bridge>`), and the host has to resolve each address by NDP on the
  bridge, where the guest only answers for the ones it has bound: `::1` works
  and the rest of the `/112` goes silent.
- The `/64` is listed **first**. The kernel refuses a via-address route whose
  gateway is itself reached by a via-address route, and the `/112` route is
  exactly that (`RTNETLINK answers: No route to host`), so the `/64` only
  installs while it is declared ahead of the `/112`. Installed, it hands the
  whole block to the container's MAC, which is what lets a customer route
  sub-prefixes out of it instead of running a responder for every address.

Because the wiring lives in the container's own network configuration, Incus
restores it on every start — a container restart or a host reboot brings the
block back with no panel-side route plumbing. What the guest needs (the address
binding) is written by the provider script, so a power start, the boot unit and
`vps ipv6-reapply` re-apply it idempotently; `EnsureExtraBlockRoutes` also
re-declares the NIC of every account that owns a block, healing a container that
was recreated or downgraded out of band.

### The host's own /64 is never handed out

The pool excludes every `/64` the host itself uses: the one holding
`net.ipv6_subnet` (bridge address, gateway, every `/112`) and any `/64` the host
holds a global address in. Routing one of those to a container would collide
with the host's own on-link route and cut its upstream connectivity — which is
why an operator may safely point the key at a prefix that also contains the
host's own /64: it is carved out, not handed out.

### Capacity and degradation

`2^(64-len)` blocks, minus the reserved ones: a `/60` yields 15 or 16, a `/56`
255 or 256, a `/48` many thousands. Blocks are handed out lowest-first, so the
assigned set reads in order. An exhausted pool is **not** an error: the
container is created without a block, the create form greys its checkbox out,
and the label shows how many are left.

An assignment belongs to the account for life. Editing a container's quotas can
never take the block back — only deleting the account releases it, after which
the block is free for the next container.

### Where it shows up

- **Admin → IPv6** (prefix or none mode): the prefix editor, the capacity readout
  (`total / kept for this host / assigned / free`) and the list of assigned
  blocks with their owners (only the assigned ones — a /48 has 65536).
- **Admin → create user**: a ticked-by-default checkbox `Assign a whole /64
  (N left)`, disabled once the pool is empty; the batch dialog has the same one.
  `vps add --extra64` is the CLI equivalent.
- **Admin → quota**: `Assign a whole /64` for a container that has none; a
  container that already owns one shows it read-only. `vps quota <name>
  --extra64` is the CLI equivalent (the same dialog's other fields keep
  working).
- **User panel**: the block is listed next to the container's IPv6 address
  (`IPv6 prefix: <block>/64`), presented plainly — nothing advertises it as
  something to ask for.