Files
風雪and風雪 232a4432d8 fix(metrics): resolve the real cgroup on LXC for memory and CPU (#85)
* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): resolve the real cgroup on LXC for memory and CPU (#85)
* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): resolve the real cgroup on LXC for memory and CPU (#85)
* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): resolve the real cgroup on LXC for memory and CPU (#85)
* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): resolve the real cgroup on LXC for memory and CPU (#85)
* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): resolve the real cgroup on LXC for memory and CPU (#85)
* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): resolve the real cgroup on LXC for memory and CPU (#85)
* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): resolve the real cgroup on LXC for memory and CPU (#85)
* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): resolve the real cgroup on LXC for memory and CPU (#85)
* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): resolve the real cgroup on LXC for memory and CPU (#85)
* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): resolve the real cgroup on LXC for memory and CPU (#85)
* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): resolve the real cgroup on LXC for memory and CPU (#85)
* fix(metrics): resolve the real cgroup on LXC, not just the hierarchy root
On LXC the memory cards still disagreed. Path resolution assumed
/sys/fs/cgroup is the whole hierarchy and simply appended the path from
/proc/self/cgroup to it. LXC's default (cgroup:mixed) mounts a tmpfs there and
bind-mounts only the container's own cgroup into a directory below it, so that
path names directories relative to the host hierarchy and does not exist from
inside the container. Resolution then failed, readServiceCgroupMemory() had no
fallback, and the per-process card silently degraded to raw RSS while the
system card read a container-scoped cgroup — two different sources, which is
exactly the inconsistency being reported.
The path is now translated through the mount's root from /proc/self/mountinfo:
if /sys/fs/cgroup exposes only a subtree, the part of the reported path that
the mount hides is dropped. That resolves both the container's own cgroup and
any unit nested inside it, so the two cards land on one accounting domain
instead of "working by accident" on hosts where the mount happens to be the
root.
readServiceCgroupMemory() also gained the root fallback it was missing, so a
layout that cannot be resolved at all still reads the domain the process is
accounted in rather than falling back to RSS. Adopting the root is gated on a
finite cap being enforced there: on a bare-metal host the root is unbounded and
attributing the machine's whole memory to two processes would be a
fabrication, so those hosts keep the previous RSS behaviour.
Both readers now share cgroupMemoryFiles(), so the v2/v1 field probing exists
once instead of being copied per caller — the duplication is how the missing
fallback went unnoticed. Memory path constants are dropped with it, and the
v1 PAGE_COUNTER_MAX sentinel is named rather than open-coded.
Tests cover the layouts directly, with /proc and /sys pointed at fixtures
through named path variables: plain hierarchy root, bind-mounted container
cgroup, a unit nested below it, cgroup v1, the bounded and unbounded fallback,
sentinel rejection, and the invariant that both cards resolve to the same
usage. A malformed mountinfo degrades to "unresolved", never to a wrong path.
* fix(metrics): read CPU accounting from the resolved cgroup
cpu.stat, cpu.max and cpuset.cpus.effective were read from hard-coded
hierarchy-root paths, so they had the same flaw the memory files had: in a
container the accounting for this process is not at the root. On LXC the root
is a tmpfs holding nothing readable, so the CPU card silently fell back to
host-wide /proc/stat and reported the machine instead of the container — the
container could be fully throttled while the bar sat near idle.
They now go through cpuAccountingDir(), which picks the process' own cgroup
when the kernel enforces a cap there (a memory limit or a CPU quota) and
otherwise keeps the hierarchy root. That preserves bare-metal semantics, where
the card is meant to describe the machine rather than the panel's own unit,
while a container — the case where the root is simply the wrong place to look —
is read from the cgroup that actually confines it.
Usage is taken from the first directory that has it, never mixed across two:
a delta between two different cgroups would be meaningless. Caps are probed
across the directories instead, since a quota on a broader domain still applies
to us, and cpuCapacity() already narrows to the tightest limit found.
The three fixed CPU path constants go away with the hard-coded reads; every
accounting file is now read relative to a resolved directory.
* test(system): stop comparing memory samples taken at different times
Two tests guarded the invariant that the system and process cards read the same
memory pool, and both failed intermittently — reproducibly under -race, and
occasionally in a plain run, which the backend job executes:
system memory used (2874142720) is smaller than this process' RSS (2875170816)
panel memory 2874384384 exceeds system used 2874142720
They compared a possibly 500ms-stale cached system reading against a live
process reading of the test binary itself. Where the panel is most of its own
accounting domain — a container, or a sandbox running the test alone — those two
numbers are nearly identical, so the comparison has no margin of its own and any
allocation in between flips it.
The process is now sampled first, so the reference reading is the newer one, and
a tolerance absorbs the remaining skew. What these tests guard is a mismatch of
sources, which was out by multiples (host /proc/meminfo against the cgroup), so
a percent-wide tolerance keeps the assertion meaningful.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
2026-09-27 15:58:47 +08:00

187 lines
5.8 KiB
Go
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
package system_test
import (
"os"
"runtime"
"sync"
"testing"
"time"
"github.com/kazeyukiro/3m-ui/backend/internal/system"
)
// procAvailable reports whether this host exposes /proc, which every Linux
// counter here (and the cgroup fallbacks) ultimately read from.
func procAvailable() bool {
_, err := os.Stat("/proc/self/stat")
return err == nil
}
// spinLoad keeps `threads` OS threads busy until stop is closed.
func spinLoad(threads int, stop <-chan struct{}) *sync.WaitGroup {
var wg sync.WaitGroup
for i := 0; i < threads; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for {
select {
case <-stop:
return
default:
// Explicit no-op keeps the loop from being optimised away
// while staying well inside one CPU.
runtime.Gosched()
}
}
}()
}
return &wg
}
// TestProcessCPUTracksCurrentLoad is the regression test for the process card
// reporting something other than what the process is doing right now.
//
// The previous implementation leaned on gopsutil's Process.CPUPercent(), which
// returns 100 * lifetimeCPUTime / uptime — an average over the process' whole
// history. Two symptoms followed: a long-lived panel settled on a flat,
// unchanging figure, and hosts where the start time cannot be resolved (the
// container's BootTime disagrees with /proc/stat) got a permanent 0.
//
// This test asserts the numbers actually move with the load: high while busy,
// back down once idle.
func TestProcessCPUTracksCurrentLoad(t *testing.T) {
if !procAvailable() {
t.Skip("process CPU sampling requires /proc")
}
pid := os.Getpid()
if n := runtime.NumCPU(); n < 2 {
t.Skipf("needs at least 2 CPUs, got %d", n)
}
threads := runtime.NumCPU()
if threads > 8 {
threads = 8
}
// Prime the sampler so its first baseline is established while idle.
idle := sampleStable(t, pid)
stop := make(chan struct{})
wg := spinLoad(threads, stop)
time.Sleep(400 * time.Millisecond) // let the load reach steady state
busy := sampleStable(t, pid)
close(stop)
wg.Wait()
// Wait out the load, then re-prime so the next window contains no leftover
// busy time from the spin-down.
time.Sleep(400 * time.Millisecond)
after := sampleStable(t, pid)
t.Logf("idle=%.1f%% busy=%.1f%% after=%.1f%% (threads=%d)", idle, busy, after, threads)
if busy <= idle+5 {
t.Fatalf("process CPU did not react to load: idle=%.1f%% while busy=%.1f%%", idle, busy)
}
if after >= busy/2 {
t.Fatalf("process CPU did not fall back after load stopped: busy=%.1f%% after=%.1f%%", busy, after)
}
}
// sampleStable takes two readings and returns the second one, whose window lies
// entirely inside the current load phase. The first reading bridges whatever
// happened since the previous sample.
func sampleStable(t *testing.T, pid int) float64 {
t.Helper()
first := system.SampleProcessUsage(pid)
if first.PID != pid {
t.Fatalf("SampleProcessUsage returned pid %d, want %d", first.PID, pid)
}
time.Sleep(600 * time.Millisecond)
return system.SampleProcessUsage(pid).CPUPercent
}
// TestProcessUsageSampleShape covers the fields the dashboard renders for both
// cards, including the "no PID" degenerate call the handler makes when the core
// is stopped.
func TestProcessUsageSampleShape(t *testing.T) {
if !procAvailable() {
t.Skip("process sampling requires /proc")
}
u := system.SampleProcessUsage(0)
if u.PID != 0 || u.CPUPercent != 0 || u.MemoryUsed != 0 || u.MemoryPercent != 0 {
t.Fatalf("zero PID should produce an empty usage, got %+v", u)
}
self := system.SampleProcessUsage(os.Getpid())
if self.PID != os.Getpid() {
t.Fatalf("pid = %d, want %d", self.PID, os.Getpid())
}
if self.MemoryUsed <= 0 {
t.Fatal("own RSS should be measurable")
}
if self.MemoryPercent < 0 || self.MemoryPercent > 100 {
t.Fatalf("memory percent out of range: %f", self.MemoryPercent)
}
if self.CPUPercent < 0 || self.CPUPercent > 100 {
t.Fatalf("cpu percent out of range: %f", self.CPUPercent)
}
}
// TestSystemCPUSampleReacts asserts the system card reads sensible CPU figures
// and stays within 0–100 while the machine is loaded.
func TestSystemCPUSampleReacts(t *testing.T) {
if !procAvailable() {
t.Skip("CPU sampling requires /proc")
}
stats := system.GetSystemStats()
if stats == nil {
t.Fatal("nil stats")
}
if stats.CPU.Percent < 0 || stats.CPU.Percent > 100 {
t.Fatalf("cpu percent out of range: %f", stats.CPU.Percent)
}
stop := make(chan struct{})
wg := spinLoad(runtime.NumCPU(), stop)
defer func() { close(stop); wg.Wait() }()
time.Sleep(500 * time.Millisecond)
loaded := system.GetSystemStats()
if loaded.CPU.Percent < 1 {
t.Fatalf("system CPU did not register load: %.1f%%", loaded.CPU.Percent)
}
if loaded.CPU.Percent > 100 {
t.Fatalf("system CPU above 100%%: %.1f%%", loaded.CPU.Percent)
}
t.Logf("system cpu: idle-ish=%.1f%% loaded=%.1f%%", stats.CPU.Percent, loaded.CPU.Percent)
}
// TestProcessMemoryUsesCgroupShare asserts panel+core memory is split from the
// same cgroup working set as the system card, so the rows cannot sum above
// system Memory.Used (the RSS double-count failure mode).
func TestProcessMemoryUsesCgroupShare(t *testing.T) {
if !procAvailable() {
t.Skip("requires /proc")
}
// Process first, system second: see TestSystemMemoryMatchesProcessMemory.
panel, core := system.SampleProcessUsagePair(os.Getpid(), 0)
stats := system.GetSystemStats()
if stats == nil || stats.Memory.Total <= 0 {
t.Skip("memory stats unavailable")
}
if panel.MemoryUsed <= 0 {
t.Fatal("panel memory should be measurable")
}
ceiling := stats.Memory.Used + memorySkewTolerance(stats.Memory.Total)
if panel.MemoryUsed > ceiling {
t.Fatalf("panel memory %.0f exceeds system used %.0f", panel.MemoryUsed, stats.Memory.Used)
}
sum := panel.MemoryUsed + core.MemoryUsed
if sum > ceiling {
t.Fatalf("panel+core memory %.0f exceeds system used %.0f", sum, stats.Memory.Used)
}
}