* fix(metrics): measure process and system CPU by delta, honour cgroup limits
The process card called gopsutil's Process.CPUPercent(), which returns
100 * lifetimeCPUTime / uptime -- an average over the process' entire
history rather than what it is doing now. Two failure modes followed:
- a long-lived panel settles on a near-constant figure that lags real
spikes by hours;
- when the start time cannot be resolved (BootTime from sysinfo
disagrees with /proc/stat inside some containers) totalTime goes
negative and the call silently yields 0. Measured here: a permanent
0.0% regardless of load.
The system card had the opposite bias inside a container: cpu.Percent()
reads /proc/stat, so a container capped at 4 of 32 host cores topped out
around 21% while actually being throttled. It also only described 200ms
of each second and blocked the request thread doing it.
Both cards now difference cumulative counters across two samples:
- CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback
- process: /proc/<pid>/stat user+system (Iowait excluded: it is block
I/O delay, waiting rather than consuming)
- shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs)
so the process and system cards finally speak one unit.
Measured on a 4-of-32-core container:
system CPU busy 21% -> 100%
process CPU busy 0% -> 89% (returns to ~0% once idle)
Also fixed along the way:
- GetSystemStats sampled outside its lock, so two concurrent requests
differenced the network counters over ~0s and reported huge spikes.
Sampling now serialises behind a barrier with a minimum interval.
- The disk card hard-coded "/" even when the SQLite database lives on a
mounted volume; it now anchors on the database directory, which
disk.Usage resolves to its containing filesystem via statfs.
- The Telegram resource alerts used the same broken samplers, so a CPU
threshold inside a limited container could never be reached and the
alert would never fire.
New tests cover load tracking, cgroup and quota arithmetic, cpuset
parsing and the already-guarded process memory denominator invariant.
Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile.
* fix(memory): report cgroup working set instead of gross usage
The previous commit moved the memory card onto cgroup accounting, which
fixed the two cards disagreeing about their source but swapped in a new
error on the other side: it used memory.current verbatim, and that is a
gross figure billing the cgroup for reclaimable page cache.
Measured on this host against an 8GB cap:
memory.current 4.44 GB
anon 0.90 GB <- real process memory
inactive_file 3.05 GB <- reclaimable by the kernel on demand
current / limit 51.7% <- what the card showed
working set / limit 16.2% <- actual pressure
So the bar sat in the orange band (>=50%) while 84% of the budget was
free. Reading a large file once -- mihomo's geoip database, a rule
provider download, SQLite growth -- is enough to trigger it, because the
kernel keeps those pages cached and simply hands them back under demand.
"Used" is now the kubelet working-set figure, usage minus inactive_file,
which is what container runtimes report as a percentage. Only *inactive*
cache is discounted, so pages actively mapped and in use stay charged and
the bar can still hold every process' RSS. Three related points:
- The label finally means the same thing on both branches. The
/proc/meminfo fallback already excluded buffers and cache, so gross
cgroup usage and host "used" were two different quantities behind one
label depending on how the panel was deployed.
- The denominator no longer requires a cap to use cgroup accounting. An
unlimited container previously fell back to host-wide figures and so
tracked other tenants' activity, while the CPU card (which always
uses the container's own counter) did not. It falls back to the host
total now only because that genuinely is the cap.
- PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit".
New tests pin the arithmetic against the real numbers above, the three
ways the limit resolves, and the clamp when cache exceeds usage.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): measure process and system CPU by delta, honour cgroup limits (#82)
* fix(metrics): measure process and system CPU by delta, honour cgroup limits
The process card called gopsutil's Process.CPUPercent(), which returns
100 * lifetimeCPUTime / uptime -- an average over the process' entire
history rather than what it is doing now. Two failure modes followed:
- a long-lived panel settles on a near-constant figure that lags real
spikes by hours;
- when the start time cannot be resolved (BootTime from sysinfo
disagrees with /proc/stat inside some containers) totalTime goes
negative and the call silently yields 0. Measured here: a permanent
0.0% regardless of load.
The system card had the opposite bias inside a container: cpu.Percent()
reads /proc/stat, so a container capped at 4 of 32 host cores topped out
around 21% while actually being throttled. It also only described 200ms
of each second and blocked the request thread doing it.
Both cards now difference cumulative counters across two samples:
- CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback
- process: /proc/<pid>/stat user+system (Iowait excluded: it is block
I/O delay, waiting rather than consuming)
- shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs)
so the process and system cards finally speak one unit.
Measured on a 4-of-32-core container:
system CPU busy 21% -> 100%
process CPU busy 0% -> 89% (returns to ~0% once idle)
Also fixed along the way:
- GetSystemStats sampled outside its lock, so two concurrent requests
differenced the network counters over ~0s and reported huge spikes.
Sampling now serialises behind a barrier with a minimum interval.
- The disk card hard-coded "/" even when the SQLite database lives on a
mounted volume; it now anchors on the database directory, which
disk.Usage resolves to its containing filesystem via statfs.
- The Telegram resource alerts used the same broken samplers, so a CPU
threshold inside a limited container could never be reached and the
alert would never fire.
New tests cover load tracking, cgroup and quota arithmetic, cpuset
parsing and the already-guarded process memory denominator invariant.
Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile.
* fix(memory): report cgroup working set instead of gross usage
The previous commit moved the memory card onto cgroup accounting, which
fixed the two cards disagreeing about their source but swapped in a new
error on the other side: it used memory.current verbatim, and that is a
gross figure billing the cgroup for reclaimable page cache.
Measured on this host against an 8GB cap:
memory.current 4.44 GB
anon 0.90 GB <- real process memory
inactive_file 3.05 GB <- reclaimable by the kernel on demand
current / limit 51.7% <- what the card showed
working set / limit 16.2% <- actual pressure
So the bar sat in the orange band (>=50%) while 84% of the budget was
free. Reading a large file once -- mihomo's geoip database, a rule
provider download, SQLite growth -- is enough to trigger it, because the
kernel keeps those pages cached and simply hands them back under demand.
"Used" is now the kubelet working-set figure, usage minus inactive_file,
which is what container runtimes report as a percentage. Only *inactive*
cache is discounted, so pages actively mapped and in use stay charged and
the bar can still hold every process' RSS. Three related points:
- The label finally means the same thing on both branches. The
/proc/meminfo fallback already excluded buffers and cache, so gross
cgroup usage and host "used" were two different quantities behind one
label depending on how the panel was deployed.
- The denominator no longer requires a cap to use cgroup accounting. An
unlimited container previously fell back to host-wide figures and so
tracked other tenants' activity, while the CPU card (which always
uses the container's own counter) did not. It falls back to the host
total now only because that genuinely is the cap.
- PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit".
New tests pin the arithmetic against the real numbers above, the three
ways the limit resolves, and the clamp when cache exceeds usage.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): measure process and system CPU by delta, honour cgroup limits (#82)
* fix(metrics): measure process and system CPU by delta, honour cgroup limits
The process card called gopsutil's Process.CPUPercent(), which returns
100 * lifetimeCPUTime / uptime -- an average over the process' entire
history rather than what it is doing now. Two failure modes followed:
- a long-lived panel settles on a near-constant figure that lags real
spikes by hours;
- when the start time cannot be resolved (BootTime from sysinfo
disagrees with /proc/stat inside some containers) totalTime goes
negative and the call silently yields 0. Measured here: a permanent
0.0% regardless of load.
The system card had the opposite bias inside a container: cpu.Percent()
reads /proc/stat, so a container capped at 4 of 32 host cores topped out
around 21% while actually being throttled. It also only described 200ms
of each second and blocked the request thread doing it.
Both cards now difference cumulative counters across two samples:
- CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback
- process: /proc/<pid>/stat user+system (Iowait excluded: it is block
I/O delay, waiting rather than consuming)
- shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs)
so the process and system cards finally speak one unit.
Measured on a 4-of-32-core container:
system CPU busy 21% -> 100%
process CPU busy 0% -> 89% (returns to ~0% once idle)
Also fixed along the way:
- GetSystemStats sampled outside its lock, so two concurrent requests
differenced the network counters over ~0s and reported huge spikes.
Sampling now serialises behind a barrier with a minimum interval.
- The disk card hard-coded "/" even when the SQLite database lives on a
mounted volume; it now anchors on the database directory, which
disk.Usage resolves to its containing filesystem via statfs.
- The Telegram resource alerts used the same broken samplers, so a CPU
threshold inside a limited container could never be reached and the
alert would never fire.
New tests cover load tracking, cgroup and quota arithmetic, cpuset
parsing and the already-guarded process memory denominator invariant.
Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile.
* fix(memory): report cgroup working set instead of gross usage
The previous commit moved the memory card onto cgroup accounting, which
fixed the two cards disagreeing about their source but swapped in a new
error on the other side: it used memory.current verbatim, and that is a
gross figure billing the cgroup for reclaimable page cache.
Measured on this host against an 8GB cap:
memory.current 4.44 GB
anon 0.90 GB <- real process memory
inactive_file 3.05 GB <- reclaimable by the kernel on demand
current / limit 51.7% <- what the card showed
working set / limit 16.2% <- actual pressure
So the bar sat in the orange band (>=50%) while 84% of the budget was
free. Reading a large file once -- mihomo's geoip database, a rule
provider download, SQLite growth -- is enough to trigger it, because the
kernel keeps those pages cached and simply hands them back under demand.
"Used" is now the kubelet working-set figure, usage minus inactive_file,
which is what container runtimes report as a percentage. Only *inactive*
cache is discounted, so pages actively mapped and in use stay charged and
the bar can still hold every process' RSS. Three related points:
- The label finally means the same thing on both branches. The
/proc/meminfo fallback already excluded buffers and cache, so gross
cgroup usage and host "used" were two different quantities behind one
label depending on how the panel was deployed.
- The denominator no longer requires a cap to use cgroup accounting. An
unlimited container previously fell back to host-wide figures and so
tracked other tenants' activity, while the CPU card (which always
uses the container's own counter) did not. It falls back to the host
total now only because that genuinely is the cap.
- PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit".
New tests pin the arithmetic against the real numbers above, the three
ways the limit resolves, and the clamp when cache exceeds usage.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): measure process and system CPU by delta, honour cgroup limits (#82)
* fix(metrics): measure process and system CPU by delta, honour cgroup limits
The process card called gopsutil's Process.CPUPercent(), which returns
100 * lifetimeCPUTime / uptime -- an average over the process' entire
history rather than what it is doing now. Two failure modes followed:
- a long-lived panel settles on a near-constant figure that lags real
spikes by hours;
- when the start time cannot be resolved (BootTime from sysinfo
disagrees with /proc/stat inside some containers) totalTime goes
negative and the call silently yields 0. Measured here: a permanent
0.0% regardless of load.
The system card had the opposite bias inside a container: cpu.Percent()
reads /proc/stat, so a container capped at 4 of 32 host cores topped out
around 21% while actually being throttled. It also only described 200ms
of each second and blocked the request thread doing it.
Both cards now difference cumulative counters across two samples:
- CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback
- process: /proc/<pid>/stat user+system (Iowait excluded: it is block
I/O delay, waiting rather than consuming)
- shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs)
so the process and system cards finally speak one unit.
Measured on a 4-of-32-core container:
system CPU busy 21% -> 100%
process CPU busy 0% -> 89% (returns to ~0% once idle)
Also fixed along the way:
- GetSystemStats sampled outside its lock, so two concurrent requests
differenced the network counters over ~0s and reported huge spikes.
Sampling now serialises behind a barrier with a minimum interval.
- The disk card hard-coded "/" even when the SQLite database lives on a
mounted volume; it now anchors on the database directory, which
disk.Usage resolves to its containing filesystem via statfs.
- The Telegram resource alerts used the same broken samplers, so a CPU
threshold inside a limited container could never be reached and the
alert would never fire.
New tests cover load tracking, cgroup and quota arithmetic, cpuset
parsing and the already-guarded process memory denominator invariant.
Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile.
* fix(memory): report cgroup working set instead of gross usage
The previous commit moved the memory card onto cgroup accounting, which
fixed the two cards disagreeing about their source but swapped in a new
error on the other side: it used memory.current verbatim, and that is a
gross figure billing the cgroup for reclaimable page cache.
Measured on this host against an 8GB cap:
memory.current 4.44 GB
anon 0.90 GB <- real process memory
inactive_file 3.05 GB <- reclaimable by the kernel on demand
current / limit 51.7% <- what the card showed
working set / limit 16.2% <- actual pressure
So the bar sat in the orange band (>=50%) while 84% of the budget was
free. Reading a large file once -- mihomo's geoip database, a rule
provider download, SQLite growth -- is enough to trigger it, because the
kernel keeps those pages cached and simply hands them back under demand.
"Used" is now the kubelet working-set figure, usage minus inactive_file,
which is what container runtimes report as a percentage. Only *inactive*
cache is discounted, so pages actively mapped and in use stay charged and
the bar can still hold every process' RSS. Three related points:
- The label finally means the same thing on both branches. The
/proc/meminfo fallback already excluded buffers and cache, so gross
cgroup usage and host "used" were two different quantities behind one
label depending on how the panel was deployed.
- The denominator no longer requires a cap to use cgroup accounting. An
unlimited container previously fell back to host-wide figures and so
tracked other tenants' activity, while the CPU card (which always
uses the container's own counter) did not. It falls back to the host
total now only because that genuinely is the cap.
- PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit".
New tests pin the arithmetic against the real numbers above, the three
ways the limit resolves, and the clamp when cache exceeds usage.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): measure process and system CPU by delta, honour cgroup limits (#82)
* fix(metrics): measure process and system CPU by delta, honour cgroup limits
The process card called gopsutil's Process.CPUPercent(), which returns
100 * lifetimeCPUTime / uptime -- an average over the process' entire
history rather than what it is doing now. Two failure modes followed:
- a long-lived panel settles on a near-constant figure that lags real
spikes by hours;
- when the start time cannot be resolved (BootTime from sysinfo
disagrees with /proc/stat inside some containers) totalTime goes
negative and the call silently yields 0. Measured here: a permanent
0.0% regardless of load.
The system card had the opposite bias inside a container: cpu.Percent()
reads /proc/stat, so a container capped at 4 of 32 host cores topped out
around 21% while actually being throttled. It also only described 200ms
of each second and blocked the request thread doing it.
Both cards now difference cumulative counters across two samples:
- CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback
- process: /proc/<pid>/stat user+system (Iowait excluded: it is block
I/O delay, waiting rather than consuming)
- shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs)
so the process and system cards finally speak one unit.
Measured on a 4-of-32-core container:
system CPU busy 21% -> 100%
process CPU busy 0% -> 89% (returns to ~0% once idle)
Also fixed along the way:
- GetSystemStats sampled outside its lock, so two concurrent requests
differenced the network counters over ~0s and reported huge spikes.
Sampling now serialises behind a barrier with a minimum interval.
- The disk card hard-coded "/" even when the SQLite database lives on a
mounted volume; it now anchors on the database directory, which
disk.Usage resolves to its containing filesystem via statfs.
- The Telegram resource alerts used the same broken samplers, so a CPU
threshold inside a limited container could never be reached and the
alert would never fire.
New tests cover load tracking, cgroup and quota arithmetic, cpuset
parsing and the already-guarded process memory denominator invariant.
Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile.
* fix(memory): report cgroup working set instead of gross usage
The previous commit moved the memory card onto cgroup accounting, which
fixed the two cards disagreeing about their source but swapped in a new
error on the other side: it used memory.current verbatim, and that is a
gross figure billing the cgroup for reclaimable page cache.
Measured on this host against an 8GB cap:
memory.current 4.44 GB
anon 0.90 GB <- real process memory
inactive_file 3.05 GB <- reclaimable by the kernel on demand
current / limit 51.7% <- what the card showed
working set / limit 16.2% <- actual pressure
So the bar sat in the orange band (>=50%) while 84% of the budget was
free. Reading a large file once -- mihomo's geoip database, a rule
provider download, SQLite growth -- is enough to trigger it, because the
kernel keeps those pages cached and simply hands them back under demand.
"Used" is now the kubelet working-set figure, usage minus inactive_file,
which is what container runtimes report as a percentage. Only *inactive*
cache is discounted, so pages actively mapped and in use stay charged and
the bar can still hold every process' RSS. Three related points:
- The label finally means the same thing on both branches. The
/proc/meminfo fallback already excluded buffers and cache, so gross
cgroup usage and host "used" were two different quantities behind one
label depending on how the panel was deployed.
- The denominator no longer requires a cap to use cgroup accounting. An
unlimited container previously fell back to host-wide figures and so
tracked other tenants' activity, while the CPU card (which always
uses the container's own counter) did not. It falls back to the host
total now only because that genuinely is the cap.
- PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit".
New tests pin the arithmetic against the real numbers above, the three
ways the limit resolves, and the clamp when cache exceeds usage.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): measure process and system CPU by delta, honour cgroup limits (#82)
* fix(metrics): measure process and system CPU by delta, honour cgroup limits
The process card called gopsutil's Process.CPUPercent(), which returns
100 * lifetimeCPUTime / uptime -- an average over the process' entire
history rather than what it is doing now. Two failure modes followed:
- a long-lived panel settles on a near-constant figure that lags real
spikes by hours;
- when the start time cannot be resolved (BootTime from sysinfo
disagrees with /proc/stat inside some containers) totalTime goes
negative and the call silently yields 0. Measured here: a permanent
0.0% regardless of load.
The system card had the opposite bias inside a container: cpu.Percent()
reads /proc/stat, so a container capped at 4 of 32 host cores topped out
around 21% while actually being throttled. It also only described 200ms
of each second and blocked the request thread doing it.
Both cards now difference cumulative counters across two samples:
- CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback
- process: /proc/<pid>/stat user+system (Iowait excluded: it is block
I/O delay, waiting rather than consuming)
- shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs)
so the process and system cards finally speak one unit.
Measured on a 4-of-32-core container:
system CPU busy 21% -> 100%
process CPU busy 0% -> 89% (returns to ~0% once idle)
Also fixed along the way:
- GetSystemStats sampled outside its lock, so two concurrent requests
differenced the network counters over ~0s and reported huge spikes.
Sampling now serialises behind a barrier with a minimum interval.
- The disk card hard-coded "/" even when the SQLite database lives on a
mounted volume; it now anchors on the database directory, which
disk.Usage resolves to its containing filesystem via statfs.
- The Telegram resource alerts used the same broken samplers, so a CPU
threshold inside a limited container could never be reached and the
alert would never fire.
New tests cover load tracking, cgroup and quota arithmetic, cpuset
parsing and the already-guarded process memory denominator invariant.
Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile.
* fix(memory): report cgroup working set instead of gross usage
The previous commit moved the memory card onto cgroup accounting, which
fixed the two cards disagreeing about their source but swapped in a new
error on the other side: it used memory.current verbatim, and that is a
gross figure billing the cgroup for reclaimable page cache.
Measured on this host against an 8GB cap:
memory.current 4.44 GB
anon 0.90 GB <- real process memory
inactive_file 3.05 GB <- reclaimable by the kernel on demand
current / limit 51.7% <- what the card showed
working set / limit 16.2% <- actual pressure
So the bar sat in the orange band (>=50%) while 84% of the budget was
free. Reading a large file once -- mihomo's geoip database, a rule
provider download, SQLite growth -- is enough to trigger it, because the
kernel keeps those pages cached and simply hands them back under demand.
"Used" is now the kubelet working-set figure, usage minus inactive_file,
which is what container runtimes report as a percentage. Only *inactive*
cache is discounted, so pages actively mapped and in use stay charged and
the bar can still hold every process' RSS. Three related points:
- The label finally means the same thing on both branches. The
/proc/meminfo fallback already excluded buffers and cache, so gross
cgroup usage and host "used" were two different quantities behind one
label depending on how the panel was deployed.
- The denominator no longer requires a cap to use cgroup accounting. An
unlimited container previously fell back to host-wide figures and so
tracked other tenants' activity, while the CPU card (which always
uses the container's own counter) did not. It falls back to the host
total now only because that genuinely is the cap.
- PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit".
New tests pin the arithmetic against the real numbers above, the three
ways the limit resolves, and the clamp when cache exceeds usage.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): measure process and system CPU by delta, honour cgroup limits (#82)
* fix(metrics): measure process and system CPU by delta, honour cgroup limits
The process card called gopsutil's Process.CPUPercent(), which returns
100 * lifetimeCPUTime / uptime -- an average over the process' entire
history rather than what it is doing now. Two failure modes followed:
- a long-lived panel settles on a near-constant figure that lags real
spikes by hours;
- when the start time cannot be resolved (BootTime from sysinfo
disagrees with /proc/stat inside some containers) totalTime goes
negative and the call silently yields 0. Measured here: a permanent
0.0% regardless of load.
The system card had the opposite bias inside a container: cpu.Percent()
reads /proc/stat, so a container capped at 4 of 32 host cores topped out
around 21% while actually being throttled. It also only described 200ms
of each second and blocked the request thread doing it.
Both cards now difference cumulative counters across two samples:
- CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback
- process: /proc/<pid>/stat user+system (Iowait excluded: it is block
I/O delay, waiting rather than consuming)
- shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs)
so the process and system cards finally speak one unit.
Measured on a 4-of-32-core container:
system CPU busy 21% -> 100%
process CPU busy 0% -> 89% (returns to ~0% once idle)
Also fixed along the way:
- GetSystemStats sampled outside its lock, so two concurrent requests
differenced the network counters over ~0s and reported huge spikes.
Sampling now serialises behind a barrier with a minimum interval.
- The disk card hard-coded "/" even when the SQLite database lives on a
mounted volume; it now anchors on the database directory, which
disk.Usage resolves to its containing filesystem via statfs.
- The Telegram resource alerts used the same broken samplers, so a CPU
threshold inside a limited container could never be reached and the
alert would never fire.
New tests cover load tracking, cgroup and quota arithmetic, cpuset
parsing and the already-guarded process memory denominator invariant.
Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile.
* fix(memory): report cgroup working set instead of gross usage
The previous commit moved the memory card onto cgroup accounting, which
fixed the two cards disagreeing about their source but swapped in a new
error on the other side: it used memory.current verbatim, and that is a
gross figure billing the cgroup for reclaimable page cache.
Measured on this host against an 8GB cap:
memory.current 4.44 GB
anon 0.90 GB <- real process memory
inactive_file 3.05 GB <- reclaimable by the kernel on demand
current / limit 51.7% <- what the card showed
working set / limit 16.2% <- actual pressure
So the bar sat in the orange band (>=50%) while 84% of the budget was
free. Reading a large file once -- mihomo's geoip database, a rule
provider download, SQLite growth -- is enough to trigger it, because the
kernel keeps those pages cached and simply hands them back under demand.
"Used" is now the kubelet working-set figure, usage minus inactive_file,
which is what container runtimes report as a percentage. Only *inactive*
cache is discounted, so pages actively mapped and in use stay charged and
the bar can still hold every process' RSS. Three related points:
- The label finally means the same thing on both branches. The
/proc/meminfo fallback already excluded buffers and cache, so gross
cgroup usage and host "used" were two different quantities behind one
label depending on how the panel was deployed.
- The denominator no longer requires a cap to use cgroup accounting. An
unlimited container previously fell back to host-wide figures and so
tracked other tenants' activity, while the CPU card (which always
uses the container's own counter) did not. It falls back to the host
total now only because that genuinely is the cap.
- PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit".
New tests pin the arithmetic against the real numbers above, the three
ways the limit resolves, and the clamp when cache exceeds usage.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): measure process and system CPU by delta, honour cgroup limits (#82)
* fix(metrics): measure process and system CPU by delta, honour cgroup limits
The process card called gopsutil's Process.CPUPercent(), which returns
100 * lifetimeCPUTime / uptime -- an average over the process' entire
history rather than what it is doing now. Two failure modes followed:
- a long-lived panel settles on a near-constant figure that lags real
spikes by hours;
- when the start time cannot be resolved (BootTime from sysinfo
disagrees with /proc/stat inside some containers) totalTime goes
negative and the call silently yields 0. Measured here: a permanent
0.0% regardless of load.
The system card had the opposite bias inside a container: cpu.Percent()
reads /proc/stat, so a container capped at 4 of 32 host cores topped out
around 21% while actually being throttled. It also only described 200ms
of each second and blocked the request thread doing it.
Both cards now difference cumulative counters across two samples:
- CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback
- process: /proc/<pid>/stat user+system (Iowait excluded: it is block
I/O delay, waiting rather than consuming)
- shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs)
so the process and system cards finally speak one unit.
Measured on a 4-of-32-core container:
system CPU busy 21% -> 100%
process CPU busy 0% -> 89% (returns to ~0% once idle)
Also fixed along the way:
- GetSystemStats sampled outside its lock, so two concurrent requests
differenced the network counters over ~0s and reported huge spikes.
Sampling now serialises behind a barrier with a minimum interval.
- The disk card hard-coded "/" even when the SQLite database lives on a
mounted volume; it now anchors on the database directory, which
disk.Usage resolves to its containing filesystem via statfs.
- The Telegram resource alerts used the same broken samplers, so a CPU
threshold inside a limited container could never be reached and the
alert would never fire.
New tests cover load tracking, cgroup and quota arithmetic, cpuset
parsing and the already-guarded process memory denominator invariant.
Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile.
* fix(memory): report cgroup working set instead of gross usage
The previous commit moved the memory card onto cgroup accounting, which
fixed the two cards disagreeing about their source but swapped in a new
error on the other side: it used memory.current verbatim, and that is a
gross figure billing the cgroup for reclaimable page cache.
Measured on this host against an 8GB cap:
memory.current 4.44 GB
anon 0.90 GB <- real process memory
inactive_file 3.05 GB <- reclaimable by the kernel on demand
current / limit 51.7% <- what the card showed
working set / limit 16.2% <- actual pressure
So the bar sat in the orange band (>=50%) while 84% of the budget was
free. Reading a large file once -- mihomo's geoip database, a rule
provider download, SQLite growth -- is enough to trigger it, because the
kernel keeps those pages cached and simply hands them back under demand.
"Used" is now the kubelet working-set figure, usage minus inactive_file,
which is what container runtimes report as a percentage. Only *inactive*
cache is discounted, so pages actively mapped and in use stay charged and
the bar can still hold every process' RSS. Three related points:
- The label finally means the same thing on both branches. The
/proc/meminfo fallback already excluded buffers and cache, so gross
cgroup usage and host "used" were two different quantities behind one
label depending on how the panel was deployed.
- The denominator no longer requires a cap to use cgroup accounting. An
unlimited container previously fell back to host-wide figures and so
tracked other tenants' activity, while the CPU card (which always
uses the container's own counter) did not. It falls back to the host
total now only because that genuinely is the cap.
- PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit".
New tests pin the arithmetic against the real numbers above, the three
ways the limit resolves, and the clamp when cache exceeds usage.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): measure process and system CPU by delta, honour cgroup limits (#82)
* fix(metrics): measure process and system CPU by delta, honour cgroup limits
The process card called gopsutil's Process.CPUPercent(), which returns
100 * lifetimeCPUTime / uptime -- an average over the process' entire
history rather than what it is doing now. Two failure modes followed:
- a long-lived panel settles on a near-constant figure that lags real
spikes by hours;
- when the start time cannot be resolved (BootTime from sysinfo
disagrees with /proc/stat inside some containers) totalTime goes
negative and the call silently yields 0. Measured here: a permanent
0.0% regardless of load.
The system card had the opposite bias inside a container: cpu.Percent()
reads /proc/stat, so a container capped at 4 of 32 host cores topped out
around 21% while actually being throttled. It also only described 200ms
of each second and blocked the request thread doing it.
Both cards now difference cumulative counters across two samples:
- CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback
- process: /proc/<pid>/stat user+system (Iowait excluded: it is block
I/O delay, waiting rather than consuming)
- shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs)
so the process and system cards finally speak one unit.
Measured on a 4-of-32-core container:
system CPU busy 21% -> 100%
process CPU busy 0% -> 89% (returns to ~0% once idle)
Also fixed along the way:
- GetSystemStats sampled outside its lock, so two concurrent requests
differenced the network counters over ~0s and reported huge spikes.
Sampling now serialises behind a barrier with a minimum interval.
- The disk card hard-coded "/" even when the SQLite database lives on a
mounted volume; it now anchors on the database directory, which
disk.Usage resolves to its containing filesystem via statfs.
- The Telegram resource alerts used the same broken samplers, so a CPU
threshold inside a limited container could never be reached and the
alert would never fire.
New tests cover load tracking, cgroup and quota arithmetic, cpuset
parsing and the already-guarded process memory denominator invariant.
Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile.
* fix(memory): report cgroup working set instead of gross usage
The previous commit moved the memory card onto cgroup accounting, which
fixed the two cards disagreeing about their source but swapped in a new
error on the other side: it used memory.current verbatim, and that is a
gross figure billing the cgroup for reclaimable page cache.
Measured on this host against an 8GB cap:
memory.current 4.44 GB
anon 0.90 GB <- real process memory
inactive_file 3.05 GB <- reclaimable by the kernel on demand
current / limit 51.7% <- what the card showed
working set / limit 16.2% <- actual pressure
So the bar sat in the orange band (>=50%) while 84% of the budget was
free. Reading a large file once -- mihomo's geoip database, a rule
provider download, SQLite growth -- is enough to trigger it, because the
kernel keeps those pages cached and simply hands them back under demand.
"Used" is now the kubelet working-set figure, usage minus inactive_file,
which is what container runtimes report as a percentage. Only *inactive*
cache is discounted, so pages actively mapped and in use stay charged and
the bar can still hold every process' RSS. Three related points:
- The label finally means the same thing on both branches. The
/proc/meminfo fallback already excluded buffers and cache, so gross
cgroup usage and host "used" were two different quantities behind one
label depending on how the panel was deployed.
- The denominator no longer requires a cap to use cgroup accounting. An
unlimited container previously fell back to host-wide figures and so
tracked other tenants' activity, while the CPU card (which always
uses the container's own counter) did not. It falls back to the host
total now only because that genuinely is the cap.
- PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit".
New tests pin the arithmetic against the real numbers above, the three
ways the limit resolves, and the clamp when cache exceeds usage.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): measure process and system CPU by delta, honour cgroup limits (#82)
* fix(metrics): measure process and system CPU by delta, honour cgroup limits
The process card called gopsutil's Process.CPUPercent(), which returns
100 * lifetimeCPUTime / uptime -- an average over the process' entire
history rather than what it is doing now. Two failure modes followed:
- a long-lived panel settles on a near-constant figure that lags real
spikes by hours;
- when the start time cannot be resolved (BootTime from sysinfo
disagrees with /proc/stat inside some containers) totalTime goes
negative and the call silently yields 0. Measured here: a permanent
0.0% regardless of load.
The system card had the opposite bias inside a container: cpu.Percent()
reads /proc/stat, so a container capped at 4 of 32 host cores topped out
around 21% while actually being throttled. It also only described 200ms
of each second and blocked the request thread doing it.
Both cards now difference cumulative counters across two samples:
- CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback
- process: /proc/<pid>/stat user+system (Iowait excluded: it is block
I/O delay, waiting rather than consuming)
- shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs)
so the process and system cards finally speak one unit.
Measured on a 4-of-32-core container:
system CPU busy 21% -> 100%
process CPU busy 0% -> 89% (returns to ~0% once idle)
Also fixed along the way:
- GetSystemStats sampled outside its lock, so two concurrent requests
differenced the network counters over ~0s and reported huge spikes.
Sampling now serialises behind a barrier with a minimum interval.
- The disk card hard-coded "/" even when the SQLite database lives on a
mounted volume; it now anchors on the database directory, which
disk.Usage resolves to its containing filesystem via statfs.
- The Telegram resource alerts used the same broken samplers, so a CPU
threshold inside a limited container could never be reached and the
alert would never fire.
New tests cover load tracking, cgroup and quota arithmetic, cpuset
parsing and the already-guarded process memory denominator invariant.
Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile.
* fix(memory): report cgroup working set instead of gross usage
The previous commit moved the memory card onto cgroup accounting, which
fixed the two cards disagreeing about their source but swapped in a new
error on the other side: it used memory.current verbatim, and that is a
gross figure billing the cgroup for reclaimable page cache.
Measured on this host against an 8GB cap:
memory.current 4.44 GB
anon 0.90 GB <- real process memory
inactive_file 3.05 GB <- reclaimable by the kernel on demand
current / limit 51.7% <- what the card showed
working set / limit 16.2% <- actual pressure
So the bar sat in the orange band (>=50%) while 84% of the budget was
free. Reading a large file once -- mihomo's geoip database, a rule
provider download, SQLite growth -- is enough to trigger it, because the
kernel keeps those pages cached and simply hands them back under demand.
"Used" is now the kubelet working-set figure, usage minus inactive_file,
which is what container runtimes report as a percentage. Only *inactive*
cache is discounted, so pages actively mapped and in use stay charged and
the bar can still hold every process' RSS. Three related points:
- The label finally means the same thing on both branches. The
/proc/meminfo fallback already excluded buffers and cache, so gross
cgroup usage and host "used" were two different quantities behind one
label depending on how the panel was deployed.
- The denominator no longer requires a cap to use cgroup accounting. An
unlimited container previously fell back to host-wide figures and so
tracked other tenants' activity, while the CPU card (which always
uses the container's own counter) did not. It falls back to the host
total now only because that genuinely is the cap.
- PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit".
New tests pin the arithmetic against the real numbers above, the three
ways the limit resolves, and the clamp when cache exceeds usage.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): measure process and system CPU by delta, honour cgroup limits (#82)
* fix(metrics): measure process and system CPU by delta, honour cgroup limits
The process card called gopsutil's Process.CPUPercent(), which returns
100 * lifetimeCPUTime / uptime -- an average over the process' entire
history rather than what it is doing now. Two failure modes followed:
- a long-lived panel settles on a near-constant figure that lags real
spikes by hours;
- when the start time cannot be resolved (BootTime from sysinfo
disagrees with /proc/stat inside some containers) totalTime goes
negative and the call silently yields 0. Measured here: a permanent
0.0% regardless of load.
The system card had the opposite bias inside a container: cpu.Percent()
reads /proc/stat, so a container capped at 4 of 32 host cores topped out
around 21% while actually being throttled. It also only described 200ms
of each second and blocked the request thread doing it.
Both cards now difference cumulative counters across two samples:
- CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback
- process: /proc/<pid>/stat user+system (Iowait excluded: it is block
I/O delay, waiting rather than consuming)
- shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs)
so the process and system cards finally speak one unit.
Measured on a 4-of-32-core container:
system CPU busy 21% -> 100%
process CPU busy 0% -> 89% (returns to ~0% once idle)
Also fixed along the way:
- GetSystemStats sampled outside its lock, so two concurrent requests
differenced the network counters over ~0s and reported huge spikes.
Sampling now serialises behind a barrier with a minimum interval.
- The disk card hard-coded "/" even when the SQLite database lives on a
mounted volume; it now anchors on the database directory, which
disk.Usage resolves to its containing filesystem via statfs.
- The Telegram resource alerts used the same broken samplers, so a CPU
threshold inside a limited container could never be reached and the
alert would never fire.
New tests cover load tracking, cgroup and quota arithmetic, cpuset
parsing and the already-guarded process memory denominator invariant.
Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile.
* fix(memory): report cgroup working set instead of gross usage
The previous commit moved the memory card onto cgroup accounting, which
fixed the two cards disagreeing about their source but swapped in a new
error on the other side: it used memory.current verbatim, and that is a
gross figure billing the cgroup for reclaimable page cache.
Measured on this host against an 8GB cap:
memory.current 4.44 GB
anon 0.90 GB <- real process memory
inactive_file 3.05 GB <- reclaimable by the kernel on demand
current / limit 51.7% <- what the card showed
working set / limit 16.2% <- actual pressure
So the bar sat in the orange band (>=50%) while 84% of the budget was
free. Reading a large file once -- mihomo's geoip database, a rule
provider download, SQLite growth -- is enough to trigger it, because the
kernel keeps those pages cached and simply hands them back under demand.
"Used" is now the kubelet working-set figure, usage minus inactive_file,
which is what container runtimes report as a percentage. Only *inactive*
cache is discounted, so pages actively mapped and in use stay charged and
the bar can still hold every process' RSS. Three related points:
- The label finally means the same thing on both branches. The
/proc/meminfo fallback already excluded buffers and cache, so gross
cgroup usage and host "used" were two different quantities behind one
label depending on how the panel was deployed.
- The denominator no longer requires a cap to use cgroup accounting. An
unlimited container previously fell back to host-wide figures and so
tracked other tenants' activity, while the CPU card (which always
uses the container's own counter) did not. It falls back to the host
total now only because that genuinely is the cap.
- PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit".
New tests pin the arithmetic against the real numbers above, the three
ways the limit resolves, and the clamp when cache exceeds usage.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
fix(metrics): measure process and system CPU by delta, honour cgroup limits (#82)
* fix(metrics): measure process and system CPU by delta, honour cgroup limits
The process card called gopsutil's Process.CPUPercent(), which returns
100 * lifetimeCPUTime / uptime -- an average over the process' entire
history rather than what it is doing now. Two failure modes followed:
- a long-lived panel settles on a near-constant figure that lags real
spikes by hours;
- when the start time cannot be resolved (BootTime from sysinfo
disagrees with /proc/stat inside some containers) totalTime goes
negative and the call silently yields 0. Measured here: a permanent
0.0% regardless of load.
The system card had the opposite bias inside a container: cpu.Percent()
reads /proc/stat, so a container capped at 4 of 32 host cores topped out
around 21% while actually being throttled. It also only described 200ms
of each second and blocked the request thread doing it.
Both cards now difference cumulative counters across two samples:
- CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback
- process: /proc/<pid>/stat user+system (Iowait excluded: it is block
I/O delay, waiting rather than consuming)
- shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs)
so the process and system cards finally speak one unit.
Measured on a 4-of-32-core container:
system CPU busy 21% -> 100%
process CPU busy 0% -> 89% (returns to ~0% once idle)
Also fixed along the way:
- GetSystemStats sampled outside its lock, so two concurrent requests
differenced the network counters over ~0s and reported huge spikes.
Sampling now serialises behind a barrier with a minimum interval.
- The disk card hard-coded "/" even when the SQLite database lives on a
mounted volume; it now anchors on the database directory, which
disk.Usage resolves to its containing filesystem via statfs.
- The Telegram resource alerts used the same broken samplers, so a CPU
threshold inside a limited container could never be reached and the
alert would never fire.
New tests cover load tracking, cgroup and quota arithmetic, cpuset
parsing and the already-guarded process memory denominator invariant.
Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile.
* fix(memory): report cgroup working set instead of gross usage
The previous commit moved the memory card onto cgroup accounting, which
fixed the two cards disagreeing about their source but swapped in a new
error on the other side: it used memory.current verbatim, and that is a
gross figure billing the cgroup for reclaimable page cache.
Measured on this host against an 8GB cap:
memory.current 4.44 GB
anon 0.90 GB <- real process memory
inactive_file 3.05 GB <- reclaimable by the kernel on demand
current / limit 51.7% <- what the card showed
working set / limit 16.2% <- actual pressure
So the bar sat in the orange band (>=50%) while 84% of the budget was
free. Reading a large file once -- mihomo's geoip database, a rule
provider download, SQLite growth -- is enough to trigger it, because the
kernel keeps those pages cached and simply hands them back under demand.
"Used" is now the kubelet working-set figure, usage minus inactive_file,
which is what container runtimes report as a percentage. Only *inactive*
cache is discounted, so pages actively mapped and in use stay charged and
the bar can still hold every process' RSS. Three related points:
- The label finally means the same thing on both branches. The
/proc/meminfo fallback already excluded buffers and cache, so gross
cgroup usage and host "used" were two different quantities behind one
label depending on how the panel was deployed.
- The denominator no longer requires a cap to use cgroup accounting. An
unlimited container previously fell back to host-wide figures and so
tracked other tenants' activity, while the CPU card (which always
uses the container's own counter) did not. It falls back to the host
total now only because that genuinely is the cap.
- PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit".
New tests pin the arithmetic against the real numbers above, the three
ways the limit resolves, and the clamp when cache exceeds usage.
---------
Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
