* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): resolve the real cgroup on LXC for memory and CPU (#85)
* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): resolve the real cgroup on LXC for memory and CPU (#85)
* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): resolve the real cgroup on LXC for memory and CPU (#85)
* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): resolve the real cgroup on LXC for memory and CPU (#85)
* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): resolve the real cgroup on LXC for memory and CPU (#85)
* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): resolve the real cgroup on LXC for memory and CPU (#85)
* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): resolve the real cgroup on LXC for memory and CPU (#85)
* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): resolve the real cgroup on LXC for memory and CPU (#85)
* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): resolve the real cgroup on LXC for memory and CPU (#85)
* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): resolve the real cgroup on LXC for memory and CPU (#85)
* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): resolve the real cgroup on LXC for memory and CPU (#85)
* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
