風雪and風雪
a089b14908
* fix(metrics): measure process and system CPU by delta, honour cgroup limits The process card called gopsutil's Process.CPUPercent(), which returns 100 * lifetimeCPUTime / uptime -- an average over the process' entire history rather than what it is doing now. Two failure modes followed: - a long-lived panel settles on a near-constant figure that lags real spikes by hours; - when the start time cannot be resolved (BootTime from sysinfo disagrees with /proc/stat inside some containers) totalTime goes negative and the call silently yields 0. Measured here: a permanent 0.0% regardless of load. The system card had the opposite bias inside a container: cpu.Percent() reads /proc/stat, so a container capped at 4 of 32 host cores topped out around 21% while actually being throttled. It also only described 200ms of each second and blocked the request thread doing it. Both cards now difference cumulative counters across two samples: - CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback - process: /proc/<pid>/stat user+system (Iowait excluded: it is block I/O delay, waiting rather than consuming) - shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs) so the process and system cards finally speak one unit. Measured on a 4-of-32-core container: system CPU busy 21% -> 100% process CPU busy 0% -> 89% (returns to ~0% once idle) Also fixed along the way: - GetSystemStats sampled outside its lock, so two concurrent requests differenced the network counters over ~0s and reported huge spikes. Sampling now serialises behind a barrier with a minimum interval. - The disk card hard-coded "/" even when the SQLite database lives on a mounted volume; it now anchors on the database directory, which disk.Usage resolves to its containing filesystem via statfs. - The Telegram resource alerts used the same broken samplers, so a CPU threshold inside a limited container could never be reached and the alert would never fire. New tests cover load tracking, cgroup and quota arithmetic, cpuset parsing and the already-guarded process memory denominator invariant. Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile. * fix(memory): report cgroup working set instead of gross usage The previous commit moved the memory card onto cgroup accounting, which fixed the two cards disagreeing about their source but swapped in a new error on the other side: it used memory.current verbatim, and that is a gross figure billing the cgroup for reclaimable page cache. Measured on this host against an 8GB cap: memory.current 4.44 GB anon 0.90 GB <- real process memory inactive_file 3.05 GB <- reclaimable by the kernel on demand current / limit 51.7% <- what the card showed working set / limit 16.2% <- actual pressure So the bar sat in the orange band (>=50%) while 84% of the budget was free. Reading a large file once -- mihomo's geoip database, a rule provider download, SQLite growth -- is enough to trigger it, because the kernel keeps those pages cached and simply hands them back under demand. "Used" is now the kubelet working-set figure, usage minus inactive_file, which is what container runtimes report as a percentage. Only *inactive* cache is discounted, so pages actively mapped and in use stay charged and the bar can still hold every process' RSS. Three related points: - The label finally means the same thing on both branches. The /proc/meminfo fallback already excluded buffers and cache, so gross cgroup usage and host "used" were two different quantities behind one label depending on how the panel was deployed. - The denominator no longer requires a cap to use cgroup accounting. An unlimited container previously fell back to host-wide figures and so tracked other tenants' activity, while the CPU card (which always uses the container's own counter) did not. It falls back to the host total now only because that genuinely is the cap. - PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit". New tests pin the arithmetic against the real numbers above, the three ways the limit resolves, and the clamp when cache exceeds usage. --------- Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com> fix(metrics): measure process and system CPU by delta, honour cgroup limits (#82) * fix(metrics): measure process and system CPU by delta, honour cgroup limits The process card called gopsutil's Process.CPUPercent(), which returns 100 * lifetimeCPUTime / uptime -- an average over the process' entire history rather than what it is doing now. Two failure modes followed: - a long-lived panel settles on a near-constant figure that lags real spikes by hours; - when the start time cannot be resolved (BootTime from sysinfo disagrees with /proc/stat inside some containers) totalTime goes negative and the call silently yields 0. Measured here: a permanent 0.0% regardless of load. The system card had the opposite bias inside a container: cpu.Percent() reads /proc/stat, so a container capped at 4 of 32 host cores topped out around 21% while actually being throttled. It also only described 200ms of each second and blocked the request thread doing it. Both cards now difference cumulative counters across two samples: - CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback - process: /proc/<pid>/stat user+system (Iowait excluded: it is block I/O delay, waiting rather than consuming) - shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs) so the process and system cards finally speak one unit. Measured on a 4-of-32-core container: system CPU busy 21% -> 100% process CPU busy 0% -> 89% (returns to ~0% once idle) Also fixed along the way: - GetSystemStats sampled outside its lock, so two concurrent requests differenced the network counters over ~0s and reported huge spikes. Sampling now serialises behind a barrier with a minimum interval. - The disk card hard-coded "/" even when the SQLite database lives on a mounted volume; it now anchors on the database directory, which disk.Usage resolves to its containing filesystem via statfs. - The Telegram resource alerts used the same broken samplers, so a CPU threshold inside a limited container could never be reached and the alert would never fire. New tests cover load tracking, cgroup and quota arithmetic, cpuset parsing and the already-guarded process memory denominator invariant. Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile. * fix(memory): report cgroup working set instead of gross usage The previous commit moved the memory card onto cgroup accounting, which fixed the two cards disagreeing about their source but swapped in a new error on the other side: it used memory.current verbatim, and that is a gross figure billing the cgroup for reclaimable page cache. Measured on this host against an 8GB cap: memory.current 4.44 GB anon 0.90 GB <- real process memory inactive_file 3.05 GB <- reclaimable by the kernel on demand current / limit 51.7% <- what the card showed working set / limit 16.2% <- actual pressure So the bar sat in the orange band (>=50%) while 84% of the budget was free. Reading a large file once -- mihomo's geoip database, a rule provider download, SQLite growth -- is enough to trigger it, because the kernel keeps those pages cached and simply hands them back under demand. "Used" is now the kubelet working-set figure, usage minus inactive_file, which is what container runtimes report as a percentage. Only *inactive* cache is discounted, so pages actively mapped and in use stay charged and the bar can still hold every process' RSS. Three related points: - The label finally means the same thing on both branches. The /proc/meminfo fallback already excluded buffers and cache, so gross cgroup usage and host "used" were two different quantities behind one label depending on how the panel was deployed. - The denominator no longer requires a cap to use cgroup accounting. An unlimited container previously fell back to host-wide figures and so tracked other tenants' activity, while the CPU card (which always uses the container's own counter) did not. It falls back to the host total now only because that genuinely is the cap. - PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit". New tests pin the arithmetic against the real numbers above, the three ways the limit resolves, and the clamp when cache exceeds usage. --------- Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com> fix(metrics): measure process and system CPU by delta, honour cgroup limits (#82) * fix(metrics): measure process and system CPU by delta, honour cgroup limits The process card called gopsutil's Process.CPUPercent(), which returns 100 * lifetimeCPUTime / uptime -- an average over the process' entire history rather than what it is doing now. Two failure modes followed: - a long-lived panel settles on a near-constant figure that lags real spikes by hours; - when the start time cannot be resolved (BootTime from sysinfo disagrees with /proc/stat inside some containers) totalTime goes negative and the call silently yields 0. Measured here: a permanent 0.0% regardless of load. The system card had the opposite bias inside a container: cpu.Percent() reads /proc/stat, so a container capped at 4 of 32 host cores topped out around 21% while actually being throttled. It also only described 200ms of each second and blocked the request thread doing it. Both cards now difference cumulative counters across two samples: - CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback - process: /proc/<pid>/stat user+system (Iowait excluded: it is block I/O delay, waiting rather than consuming) - shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs) so the process and system cards finally speak one unit. Measured on a 4-of-32-core container: system CPU busy 21% -> 100% process CPU busy 0% -> 89% (returns to ~0% once idle) Also fixed along the way: - GetSystemStats sampled outside its lock, so two concurrent requests differenced the network counters over ~0s and reported huge spikes. Sampling now serialises behind a barrier with a minimum interval. - The disk card hard-coded "/" even when the SQLite database lives on a mounted volume; it now anchors on the database directory, which disk.Usage resolves to its containing filesystem via statfs. - The Telegram resource alerts used the same broken samplers, so a CPU threshold inside a limited container could never be reached and the alert would never fire. New tests cover load tracking, cgroup and quota arithmetic, cpuset parsing and the already-guarded process memory denominator invariant. Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile. * fix(memory): report cgroup working set instead of gross usage The previous commit moved the memory card onto cgroup accounting, which fixed the two cards disagreeing about their source but swapped in a new error on the other side: it used memory.current verbatim, and that is a gross figure billing the cgroup for reclaimable page cache. Measured on this host against an 8GB cap: memory.current 4.44 GB anon 0.90 GB <- real process memory inactive_file 3.05 GB <- reclaimable by the kernel on demand current / limit 51.7% <- what the card showed working set / limit 16.2% <- actual pressure So the bar sat in the orange band (>=50%) while 84% of the budget was free. Reading a large file once -- mihomo's geoip database, a rule provider download, SQLite growth -- is enough to trigger it, because the kernel keeps those pages cached and simply hands them back under demand. "Used" is now the kubelet working-set figure, usage minus inactive_file, which is what container runtimes report as a percentage. Only *inactive* cache is discounted, so pages actively mapped and in use stay charged and the bar can still hold every process' RSS. Three related points: - The label finally means the same thing on both branches. The /proc/meminfo fallback already excluded buffers and cache, so gross cgroup usage and host "used" were two different quantities behind one label depending on how the panel was deployed. - The denominator no longer requires a cap to use cgroup accounting. An unlimited container previously fell back to host-wide figures and so tracked other tenants' activity, while the CPU card (which always uses the container's own counter) did not. It falls back to the host total now only because that genuinely is the cap. - PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit". New tests pin the arithmetic against the real numbers above, the three ways the limit resolves, and the clamp when cache exceeds usage. --------- Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com> fix(metrics): measure process and system CPU by delta, honour cgroup limits (#82) * fix(metrics): measure process and system CPU by delta, honour cgroup limits The process card called gopsutil's Process.CPUPercent(), which returns 100 * lifetimeCPUTime / uptime -- an average over the process' entire history rather than what it is doing now. Two failure modes followed: - a long-lived panel settles on a near-constant figure that lags real spikes by hours; - when the start time cannot be resolved (BootTime from sysinfo disagrees with /proc/stat inside some containers) totalTime goes negative and the call silently yields 0. Measured here: a permanent 0.0% regardless of load. The system card had the opposite bias inside a container: cpu.Percent() reads /proc/stat, so a container capped at 4 of 32 host cores topped out around 21% while actually being throttled. It also only described 200ms of each second and blocked the request thread doing it. Both cards now difference cumulative counters across two samples: - CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback - process: /proc/<pid>/stat user+system (Iowait excluded: it is block I/O delay, waiting rather than consuming) - shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs) so the process and system cards finally speak one unit. Measured on a 4-of-32-core container: system CPU busy 21% -> 100% process CPU busy 0% -> 89% (returns to ~0% once idle) Also fixed along the way: - GetSystemStats sampled outside its lock, so two concurrent requests differenced the network counters over ~0s and reported huge spikes. Sampling now serialises behind a barrier with a minimum interval. - The disk card hard-coded "/" even when the SQLite database lives on a mounted volume; it now anchors on the database directory, which disk.Usage resolves to its containing filesystem via statfs. - The Telegram resource alerts used the same broken samplers, so a CPU threshold inside a limited container could never be reached and the alert would never fire. New tests cover load tracking, cgroup and quota arithmetic, cpuset parsing and the already-guarded process memory denominator invariant. Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile. * fix(memory): report cgroup working set instead of gross usage The previous commit moved the memory card onto cgroup accounting, which fixed the two cards disagreeing about their source but swapped in a new error on the other side: it used memory.current verbatim, and that is a gross figure billing the cgroup for reclaimable page cache. Measured on this host against an 8GB cap: memory.current 4.44 GB anon 0.90 GB <- real process memory inactive_file 3.05 GB <- reclaimable by the kernel on demand current / limit 51.7% <- what the card showed working set / limit 16.2% <- actual pressure So the bar sat in the orange band (>=50%) while 84% of the budget was free. Reading a large file once -- mihomo's geoip database, a rule provider download, SQLite growth -- is enough to trigger it, because the kernel keeps those pages cached and simply hands them back under demand. "Used" is now the kubelet working-set figure, usage minus inactive_file, which is what container runtimes report as a percentage. Only *inactive* cache is discounted, so pages actively mapped and in use stay charged and the bar can still hold every process' RSS. Three related points: - The label finally means the same thing on both branches. The /proc/meminfo fallback already excluded buffers and cache, so gross cgroup usage and host "used" were two different quantities behind one label depending on how the panel was deployed. - The denominator no longer requires a cap to use cgroup accounting. An unlimited container previously fell back to host-wide figures and so tracked other tenants' activity, while the CPU card (which always uses the container's own counter) did not. It falls back to the host total now only because that genuinely is the cap. - PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit". New tests pin the arithmetic against the real numbers above, the three ways the limit resolves, and the clamp when cache exceeds usage. --------- Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com> fix(metrics): measure process and system CPU by delta, honour cgroup limits (#82) * fix(metrics): measure process and system CPU by delta, honour cgroup limits The process card called gopsutil's Process.CPUPercent(), which returns 100 * lifetimeCPUTime / uptime -- an average over the process' entire history rather than what it is doing now. Two failure modes followed: - a long-lived panel settles on a near-constant figure that lags real spikes by hours; - when the start time cannot be resolved (BootTime from sysinfo disagrees with /proc/stat inside some containers) totalTime goes negative and the call silently yields 0. Measured here: a permanent 0.0% regardless of load. The system card had the opposite bias inside a container: cpu.Percent() reads /proc/stat, so a container capped at 4 of 32 host cores topped out around 21% while actually being throttled. It also only described 200ms of each second and blocked the request thread doing it. Both cards now difference cumulative counters across two samples: - CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback - process: /proc/<pid>/stat user+system (Iowait excluded: it is block I/O delay, waiting rather than consuming) - shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs) so the process and system cards finally speak one unit. Measured on a 4-of-32-core container: system CPU busy 21% -> 100% process CPU busy 0% -> 89% (returns to ~0% once idle) Also fixed along the way: - GetSystemStats sampled outside its lock, so two concurrent requests differenced the network counters over ~0s and reported huge spikes. Sampling now serialises behind a barrier with a minimum interval. - The disk card hard-coded "/" even when the SQLite database lives on a mounted volume; it now anchors on the database directory, which disk.Usage resolves to its containing filesystem via statfs. - The Telegram resource alerts used the same broken samplers, so a CPU threshold inside a limited container could never be reached and the alert would never fire. New tests cover load tracking, cgroup and quota arithmetic, cpuset parsing and the already-guarded process memory denominator invariant. Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile. * fix(memory): report cgroup working set instead of gross usage The previous commit moved the memory card onto cgroup accounting, which fixed the two cards disagreeing about their source but swapped in a new error on the other side: it used memory.current verbatim, and that is a gross figure billing the cgroup for reclaimable page cache. Measured on this host against an 8GB cap: memory.current 4.44 GB anon 0.90 GB <- real process memory inactive_file 3.05 GB <- reclaimable by the kernel on demand current / limit 51.7% <- what the card showed working set / limit 16.2% <- actual pressure So the bar sat in the orange band (>=50%) while 84% of the budget was free. Reading a large file once -- mihomo's geoip database, a rule provider download, SQLite growth -- is enough to trigger it, because the kernel keeps those pages cached and simply hands them back under demand. "Used" is now the kubelet working-set figure, usage minus inactive_file, which is what container runtimes report as a percentage. Only *inactive* cache is discounted, so pages actively mapped and in use stay charged and the bar can still hold every process' RSS. Three related points: - The label finally means the same thing on both branches. The /proc/meminfo fallback already excluded buffers and cache, so gross cgroup usage and host "used" were two different quantities behind one label depending on how the panel was deployed. - The denominator no longer requires a cap to use cgroup accounting. An unlimited container previously fell back to host-wide figures and so tracked other tenants' activity, while the CPU card (which always uses the container's own counter) did not. It falls back to the host total now only because that genuinely is the cap. - PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit". New tests pin the arithmetic against the real numbers above, the three ways the limit resolves, and the clamp when cache exceeds usage. --------- Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com> fix(metrics): measure process and system CPU by delta, honour cgroup limits (#82) * fix(metrics): measure process and system CPU by delta, honour cgroup limits The process card called gopsutil's Process.CPUPercent(), which returns 100 * lifetimeCPUTime / uptime -- an average over the process' entire history rather than what it is doing now. Two failure modes followed: - a long-lived panel settles on a near-constant figure that lags real spikes by hours; - when the start time cannot be resolved (BootTime from sysinfo disagrees with /proc/stat inside some containers) totalTime goes negative and the call silently yields 0. Measured here: a permanent 0.0% regardless of load. The system card had the opposite bias inside a container: cpu.Percent() reads /proc/stat, so a container capped at 4 of 32 host cores topped out around 21% while actually being throttled. It also only described 200ms of each second and blocked the request thread doing it. Both cards now difference cumulative counters across two samples: - CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback - process: /proc/<pid>/stat user+system (Iowait excluded: it is block I/O delay, waiting rather than consuming) - shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs) so the process and system cards finally speak one unit. Measured on a 4-of-32-core container: system CPU busy 21% -> 100% process CPU busy 0% -> 89% (returns to ~0% once idle) Also fixed along the way: - GetSystemStats sampled outside its lock, so two concurrent requests differenced the network counters over ~0s and reported huge spikes. Sampling now serialises behind a barrier with a minimum interval. - The disk card hard-coded "/" even when the SQLite database lives on a mounted volume; it now anchors on the database directory, which disk.Usage resolves to its containing filesystem via statfs. - The Telegram resource alerts used the same broken samplers, so a CPU threshold inside a limited container could never be reached and the alert would never fire. New tests cover load tracking, cgroup and quota arithmetic, cpuset parsing and the already-guarded process memory denominator invariant. Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile. * fix(memory): report cgroup working set instead of gross usage The previous commit moved the memory card onto cgroup accounting, which fixed the two cards disagreeing about their source but swapped in a new error on the other side: it used memory.current verbatim, and that is a gross figure billing the cgroup for reclaimable page cache. Measured on this host against an 8GB cap: memory.current 4.44 GB anon 0.90 GB <- real process memory inactive_file 3.05 GB <- reclaimable by the kernel on demand current / limit 51.7% <- what the card showed working set / limit 16.2% <- actual pressure So the bar sat in the orange band (>=50%) while 84% of the budget was free. Reading a large file once -- mihomo's geoip database, a rule provider download, SQLite growth -- is enough to trigger it, because the kernel keeps those pages cached and simply hands them back under demand. "Used" is now the kubelet working-set figure, usage minus inactive_file, which is what container runtimes report as a percentage. Only *inactive* cache is discounted, so pages actively mapped and in use stay charged and the bar can still hold every process' RSS. Three related points: - The label finally means the same thing on both branches. The /proc/meminfo fallback already excluded buffers and cache, so gross cgroup usage and host "used" were two different quantities behind one label depending on how the panel was deployed. - The denominator no longer requires a cap to use cgroup accounting. An unlimited container previously fell back to host-wide figures and so tracked other tenants' activity, while the CPU card (which always uses the container's own counter) did not. It falls back to the host total now only because that genuinely is the cap. - PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit". New tests pin the arithmetic against the real numbers above, the three ways the limit resolves, and the clamp when cache exceeds usage. --------- Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com> fix(metrics): measure process and system CPU by delta, honour cgroup limits (#82) * fix(metrics): measure process and system CPU by delta, honour cgroup limits The process card called gopsutil's Process.CPUPercent(), which returns 100 * lifetimeCPUTime / uptime -- an average over the process' entire history rather than what it is doing now. Two failure modes followed: - a long-lived panel settles on a near-constant figure that lags real spikes by hours; - when the start time cannot be resolved (BootTime from sysinfo disagrees with /proc/stat inside some containers) totalTime goes negative and the call silently yields 0. Measured here: a permanent 0.0% regardless of load. The system card had the opposite bias inside a container: cpu.Percent() reads /proc/stat, so a container capped at 4 of 32 host cores topped out around 21% while actually being throttled. It also only described 200ms of each second and blocked the request thread doing it. Both cards now difference cumulative counters across two samples: - CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback - process: /proc/<pid>/stat user+system (Iowait excluded: it is block I/O delay, waiting rather than consuming) - shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs) so the process and system cards finally speak one unit. Measured on a 4-of-32-core container: system CPU busy 21% -> 100% process CPU busy 0% -> 89% (returns to ~0% once idle) Also fixed along the way: - GetSystemStats sampled outside its lock, so two concurrent requests differenced the network counters over ~0s and reported huge spikes. Sampling now serialises behind a barrier with a minimum interval. - The disk card hard-coded "/" even when the SQLite database lives on a mounted volume; it now anchors on the database directory, which disk.Usage resolves to its containing filesystem via statfs. - The Telegram resource alerts used the same broken samplers, so a CPU threshold inside a limited container could never be reached and the alert would never fire. New tests cover load tracking, cgroup and quota arithmetic, cpuset parsing and the already-guarded process memory denominator invariant. Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile. * fix(memory): report cgroup working set instead of gross usage The previous commit moved the memory card onto cgroup accounting, which fixed the two cards disagreeing about their source but swapped in a new error on the other side: it used memory.current verbatim, and that is a gross figure billing the cgroup for reclaimable page cache. Measured on this host against an 8GB cap: memory.current 4.44 GB anon 0.90 GB <- real process memory inactive_file 3.05 GB <- reclaimable by the kernel on demand current / limit 51.7% <- what the card showed working set / limit 16.2% <- actual pressure So the bar sat in the orange band (>=50%) while 84% of the budget was free. Reading a large file once -- mihomo's geoip database, a rule provider download, SQLite growth -- is enough to trigger it, because the kernel keeps those pages cached and simply hands them back under demand. "Used" is now the kubelet working-set figure, usage minus inactive_file, which is what container runtimes report as a percentage. Only *inactive* cache is discounted, so pages actively mapped and in use stay charged and the bar can still hold every process' RSS. Three related points: - The label finally means the same thing on both branches. The /proc/meminfo fallback already excluded buffers and cache, so gross cgroup usage and host "used" were two different quantities behind one label depending on how the panel was deployed. - The denominator no longer requires a cap to use cgroup accounting. An unlimited container previously fell back to host-wide figures and so tracked other tenants' activity, while the CPU card (which always uses the container's own counter) did not. It falls back to the host total now only because that genuinely is the cap. - PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit". New tests pin the arithmetic against the real numbers above, the three ways the limit resolves, and the clamp when cache exceeds usage. --------- Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com> fix(metrics): measure process and system CPU by delta, honour cgroup limits (#82) * fix(metrics): measure process and system CPU by delta, honour cgroup limits The process card called gopsutil's Process.CPUPercent(), which returns 100 * lifetimeCPUTime / uptime -- an average over the process' entire history rather than what it is doing now. Two failure modes followed: - a long-lived panel settles on a near-constant figure that lags real spikes by hours; - when the start time cannot be resolved (BootTime from sysinfo disagrees with /proc/stat inside some containers) totalTime goes negative and the call silently yields 0. Measured here: a permanent 0.0% regardless of load. The system card had the opposite bias inside a container: cpu.Percent() reads /proc/stat, so a container capped at 4 of 32 host cores topped out around 21% while actually being throttled. It also only described 200ms of each second and blocked the request thread doing it. Both cards now difference cumulative counters across two samples: - CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback - process: /proc/<pid>/stat user+system (Iowait excluded: it is block I/O delay, waiting rather than consuming) - shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs) so the process and system cards finally speak one unit. Measured on a 4-of-32-core container: system CPU busy 21% -> 100% process CPU busy 0% -> 89% (returns to ~0% once idle) Also fixed along the way: - GetSystemStats sampled outside its lock, so two concurrent requests differenced the network counters over ~0s and reported huge spikes. Sampling now serialises behind a barrier with a minimum interval. - The disk card hard-coded "/" even when the SQLite database lives on a mounted volume; it now anchors on the database directory, which disk.Usage resolves to its containing filesystem via statfs. - The Telegram resource alerts used the same broken samplers, so a CPU threshold inside a limited container could never be reached and the alert would never fire. New tests cover load tracking, cgroup and quota arithmetic, cpuset parsing and the already-guarded process memory denominator invariant. Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile. * fix(memory): report cgroup working set instead of gross usage The previous commit moved the memory card onto cgroup accounting, which fixed the two cards disagreeing about their source but swapped in a new error on the other side: it used memory.current verbatim, and that is a gross figure billing the cgroup for reclaimable page cache. Measured on this host against an 8GB cap: memory.current 4.44 GB anon 0.90 GB <- real process memory inactive_file 3.05 GB <- reclaimable by the kernel on demand current / limit 51.7% <- what the card showed working set / limit 16.2% <- actual pressure So the bar sat in the orange band (>=50%) while 84% of the budget was free. Reading a large file once -- mihomo's geoip database, a rule provider download, SQLite growth -- is enough to trigger it, because the kernel keeps those pages cached and simply hands them back under demand. "Used" is now the kubelet working-set figure, usage minus inactive_file, which is what container runtimes report as a percentage. Only *inactive* cache is discounted, so pages actively mapped and in use stay charged and the bar can still hold every process' RSS. Three related points: - The label finally means the same thing on both branches. The /proc/meminfo fallback already excluded buffers and cache, so gross cgroup usage and host "used" were two different quantities behind one label depending on how the panel was deployed. - The denominator no longer requires a cap to use cgroup accounting. An unlimited container previously fell back to host-wide figures and so tracked other tenants' activity, while the CPU card (which always uses the container's own counter) did not. It falls back to the host total now only because that genuinely is the cap. - PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit". New tests pin the arithmetic against the real numbers above, the three ways the limit resolves, and the clamp when cache exceeds usage. --------- Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com> fix(metrics): measure process and system CPU by delta, honour cgroup limits (#82) * fix(metrics): measure process and system CPU by delta, honour cgroup limits The process card called gopsutil's Process.CPUPercent(), which returns 100 * lifetimeCPUTime / uptime -- an average over the process' entire history rather than what it is doing now. Two failure modes followed: - a long-lived panel settles on a near-constant figure that lags real spikes by hours; - when the start time cannot be resolved (BootTime from sysinfo disagrees with /proc/stat inside some containers) totalTime goes negative and the call silently yields 0. Measured here: a permanent 0.0% regardless of load. The system card had the opposite bias inside a container: cpu.Percent() reads /proc/stat, so a container capped at 4 of 32 host cores topped out around 21% while actually being throttled. It also only described 200ms of each second and blocked the request thread doing it. Both cards now difference cumulative counters across two samples: - CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback - process: /proc/<pid>/stat user+system (Iowait excluded: it is block I/O delay, waiting rather than consuming) - shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs) so the process and system cards finally speak one unit. Measured on a 4-of-32-core container: system CPU busy 21% -> 100% process CPU busy 0% -> 89% (returns to ~0% once idle) Also fixed along the way: - GetSystemStats sampled outside its lock, so two concurrent requests differenced the network counters over ~0s and reported huge spikes. Sampling now serialises behind a barrier with a minimum interval. - The disk card hard-coded "/" even when the SQLite database lives on a mounted volume; it now anchors on the database directory, which disk.Usage resolves to its containing filesystem via statfs. - The Telegram resource alerts used the same broken samplers, so a CPU threshold inside a limited container could never be reached and the alert would never fire. New tests cover load tracking, cgroup and quota arithmetic, cpuset parsing and the already-guarded process memory denominator invariant. Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile. * fix(memory): report cgroup working set instead of gross usage The previous commit moved the memory card onto cgroup accounting, which fixed the two cards disagreeing about their source but swapped in a new error on the other side: it used memory.current verbatim, and that is a gross figure billing the cgroup for reclaimable page cache. Measured on this host against an 8GB cap: memory.current 4.44 GB anon 0.90 GB <- real process memory inactive_file 3.05 GB <- reclaimable by the kernel on demand current / limit 51.7% <- what the card showed working set / limit 16.2% <- actual pressure So the bar sat in the orange band (>=50%) while 84% of the budget was free. Reading a large file once -- mihomo's geoip database, a rule provider download, SQLite growth -- is enough to trigger it, because the kernel keeps those pages cached and simply hands them back under demand. "Used" is now the kubelet working-set figure, usage minus inactive_file, which is what container runtimes report as a percentage. Only *inactive* cache is discounted, so pages actively mapped and in use stay charged and the bar can still hold every process' RSS. Three related points: - The label finally means the same thing on both branches. The /proc/meminfo fallback already excluded buffers and cache, so gross cgroup usage and host "used" were two different quantities behind one label depending on how the panel was deployed. - The denominator no longer requires a cap to use cgroup accounting. An unlimited container previously fell back to host-wide figures and so tracked other tenants' activity, while the CPU card (which always uses the container's own counter) did not. It falls back to the host total now only because that genuinely is the cap. - PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit". New tests pin the arithmetic against the real numbers above, the three ways the limit resolves, and the clamp when cache exceeds usage. --------- Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com> fix(metrics): measure process and system CPU by delta, honour cgroup limits (#82) * fix(metrics): measure process and system CPU by delta, honour cgroup limits The process card called gopsutil's Process.CPUPercent(), which returns 100 * lifetimeCPUTime / uptime -- an average over the process' entire history rather than what it is doing now. Two failure modes followed: - a long-lived panel settles on a near-constant figure that lags real spikes by hours; - when the start time cannot be resolved (BootTime from sysinfo disagrees with /proc/stat inside some containers) totalTime goes negative and the call silently yields 0. Measured here: a permanent 0.0% regardless of load. The system card had the opposite bias inside a container: cpu.Percent() reads /proc/stat, so a container capped at 4 of 32 host cores topped out around 21% while actually being throttled. It also only described 200ms of each second and blocked the request thread doing it. Both cards now difference cumulative counters across two samples: - CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback - process: /proc/<pid>/stat user+system (Iowait excluded: it is block I/O delay, waiting rather than consuming) - shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs) so the process and system cards finally speak one unit. Measured on a 4-of-32-core container: system CPU busy 21% -> 100% process CPU busy 0% -> 89% (returns to ~0% once idle) Also fixed along the way: - GetSystemStats sampled outside its lock, so two concurrent requests differenced the network counters over ~0s and reported huge spikes. Sampling now serialises behind a barrier with a minimum interval. - The disk card hard-coded "/" even when the SQLite database lives on a mounted volume; it now anchors on the database directory, which disk.Usage resolves to its containing filesystem via statfs. - The Telegram resource alerts used the same broken samplers, so a CPU threshold inside a limited container could never be reached and the alert would never fire. New tests cover load tracking, cgroup and quota arithmetic, cpuset parsing and the already-guarded process memory denominator invariant. Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile. * fix(memory): report cgroup working set instead of gross usage The previous commit moved the memory card onto cgroup accounting, which fixed the two cards disagreeing about their source but swapped in a new error on the other side: it used memory.current verbatim, and that is a gross figure billing the cgroup for reclaimable page cache. Measured on this host against an 8GB cap: memory.current 4.44 GB anon 0.90 GB <- real process memory inactive_file 3.05 GB <- reclaimable by the kernel on demand current / limit 51.7% <- what the card showed working set / limit 16.2% <- actual pressure So the bar sat in the orange band (>=50%) while 84% of the budget was free. Reading a large file once -- mihomo's geoip database, a rule provider download, SQLite growth -- is enough to trigger it, because the kernel keeps those pages cached and simply hands them back under demand. "Used" is now the kubelet working-set figure, usage minus inactive_file, which is what container runtimes report as a percentage. Only *inactive* cache is discounted, so pages actively mapped and in use stay charged and the bar can still hold every process' RSS. Three related points: - The label finally means the same thing on both branches. The /proc/meminfo fallback already excluded buffers and cache, so gross cgroup usage and host "used" were two different quantities behind one label depending on how the panel was deployed. - The denominator no longer requires a cap to use cgroup accounting. An unlimited container previously fell back to host-wide figures and so tracked other tenants' activity, while the CPU card (which always uses the container's own counter) did not. It falls back to the host total now only because that genuinely is the cap. - PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit". New tests pin the arithmetic against the real numbers above, the three ways the limit resolves, and the clamp when cache exceeds usage. --------- Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com> fix(metrics): measure process and system CPU by delta, honour cgroup limits (#82) * fix(metrics): measure process and system CPU by delta, honour cgroup limits The process card called gopsutil's Process.CPUPercent(), which returns 100 * lifetimeCPUTime / uptime -- an average over the process' entire history rather than what it is doing now. Two failure modes followed: - a long-lived panel settles on a near-constant figure that lags real spikes by hours; - when the start time cannot be resolved (BootTime from sysinfo disagrees with /proc/stat inside some containers) totalTime goes negative and the call silently yields 0. Measured here: a permanent 0.0% regardless of load. The system card had the opposite bias inside a container: cpu.Percent() reads /proc/stat, so a container capped at 4 of 32 host cores topped out around 21% while actually being throttled. It also only described 200ms of each second and blocked the request thread doing it. Both cards now difference cumulative counters across two samples: - CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback - process: /proc/<pid>/stat user+system (Iowait excluded: it is block I/O delay, waiting rather than consuming) - shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs) so the process and system cards finally speak one unit. Measured on a 4-of-32-core container: system CPU busy 21% -> 100% process CPU busy 0% -> 89% (returns to ~0% once idle) Also fixed along the way: - GetSystemStats sampled outside its lock, so two concurrent requests differenced the network counters over ~0s and reported huge spikes. Sampling now serialises behind a barrier with a minimum interval. - The disk card hard-coded "/" even when the SQLite database lives on a mounted volume; it now anchors on the database directory, which disk.Usage resolves to its containing filesystem via statfs. - The Telegram resource alerts used the same broken samplers, so a CPU threshold inside a limited container could never be reached and the alert would never fire. New tests cover load tracking, cgroup and quota arithmetic, cpuset parsing and the already-guarded process memory denominator invariant. Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile. * fix(memory): report cgroup working set instead of gross usage The previous commit moved the memory card onto cgroup accounting, which fixed the two cards disagreeing about their source but swapped in a new error on the other side: it used memory.current verbatim, and that is a gross figure billing the cgroup for reclaimable page cache. Measured on this host against an 8GB cap: memory.current 4.44 GB anon 0.90 GB <- real process memory inactive_file 3.05 GB <- reclaimable by the kernel on demand current / limit 51.7% <- what the card showed working set / limit 16.2% <- actual pressure So the bar sat in the orange band (>=50%) while 84% of the budget was free. Reading a large file once -- mihomo's geoip database, a rule provider download, SQLite growth -- is enough to trigger it, because the kernel keeps those pages cached and simply hands them back under demand. "Used" is now the kubelet working-set figure, usage minus inactive_file, which is what container runtimes report as a percentage. Only *inactive* cache is discounted, so pages actively mapped and in use stay charged and the bar can still hold every process' RSS. Three related points: - The label finally means the same thing on both branches. The /proc/meminfo fallback already excluded buffers and cache, so gross cgroup usage and host "used" were two different quantities behind one label depending on how the panel was deployed. - The denominator no longer requires a cap to use cgroup accounting. An unlimited container previously fell back to host-wide figures and so tracked other tenants' activity, while the CPU card (which always uses the container's own counter) did not. It falls back to the host total now only because that genuinely is the cap. - PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit". New tests pin the arithmetic against the real numbers above, the three ways the limit resolves, and the clamp when cache exceeds usage. --------- Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com> fix(metrics): measure process and system CPU by delta, honour cgroup limits (#82) * fix(metrics): measure process and system CPU by delta, honour cgroup limits The process card called gopsutil's Process.CPUPercent(), which returns 100 * lifetimeCPUTime / uptime -- an average over the process' entire history rather than what it is doing now. Two failure modes followed: - a long-lived panel settles on a near-constant figure that lags real spikes by hours; - when the start time cannot be resolved (BootTime from sysinfo disagrees with /proc/stat inside some containers) totalTime goes negative and the call silently yields 0. Measured here: a permanent 0.0% regardless of load. The system card had the opposite bias inside a container: cpu.Percent() reads /proc/stat, so a container capped at 4 of 32 host cores topped out around 21% while actually being throttled. It also only described 200ms of each second and blocked the request thread doing it. Both cards now difference cumulative counters across two samples: - CPU: cgroup v2 cpu.stat / v1 cpuacct.usage, host /proc/stat fallback - process: /proc/<pid>/stat user+system (Iowait excluded: it is block I/O delay, waiting rather than consuming) - shared denominator cpuCapacity() = min(CFS quota, cpuset, online CPUs) so the process and system cards finally speak one unit. Measured on a 4-of-32-core container: system CPU busy 21% -> 100% process CPU busy 0% -> 89% (returns to ~0% once idle) Also fixed along the way: - GetSystemStats sampled outside its lock, so two concurrent requests differenced the network counters over ~0s and reported huge spikes. Sampling now serialises behind a barrier with a minimum interval. - The disk card hard-coded "/" even when the SQLite database lives on a mounted volume; it now anchors on the database directory, which disk.Usage resolves to its containing filesystem via statfs. - The Telegram resource alerts used the same broken samplers, so a CPU threshold inside a limited container could never be reached and the alert would never fire. New tests cover load tracking, cgroup and quota arithmetic, cpuset parsing and the already-guarded process memory denominator invariant. Full suite plus -race passes; darwin/linux-arm64/linux-386 cross-compile. * fix(memory): report cgroup working set instead of gross usage The previous commit moved the memory card onto cgroup accounting, which fixed the two cards disagreeing about their source but swapped in a new error on the other side: it used memory.current verbatim, and that is a gross figure billing the cgroup for reclaimable page cache. Measured on this host against an 8GB cap: memory.current 4.44 GB anon 0.90 GB <- real process memory inactive_file 3.05 GB <- reclaimable by the kernel on demand current / limit 51.7% <- what the card showed working set / limit 16.2% <- actual pressure So the bar sat in the orange band (>=50%) while 84% of the budget was free. Reading a large file once -- mihomo's geoip database, a rule provider download, SQLite growth -- is enough to trigger it, because the kernel keeps those pages cached and simply hands them back under demand. "Used" is now the kubelet working-set figure, usage minus inactive_file, which is what container runtimes report as a percentage. Only *inactive* cache is discounted, so pages actively mapped and in use stay charged and the bar can still hold every process' RSS. Three related points: - The label finally means the same thing on both branches. The /proc/meminfo fallback already excluded buffers and cache, so gross cgroup usage and host "used" were two different quantities behind one label depending on how the panel was deployed. - The denominator no longer requires a cap to use cgroup accounting. An unlimited container previously fell back to host-wide figures and so tracked other tenants' activity, while the CPU card (which always uses the container's own counter) did not. It falls back to the host total now only because that genuinely is the cap. - PAGE_COUNTER_MAX from cgroup v1 is still treated as "no limit". New tests pin the arithmetic against the real numbers above, the three ways the limit resolves, and the clamp when cache exceeds usage. --------- Co-authored-by: 風雪 <kazeyukiro@users.noreply.github.com>
41 lines
1.4 KiB
Go
41 lines
1.4 KiB
Go
package telegram
|
|
|
|
import (
|
|
"errors"
|
|
|
|
"github.com/kazeyukiro/3m-ui/backend/internal/system"
|
|
)
|
|
|
|
// errSampleUnavailable is returned when neither the panel's own accounting nor
|
|
// /proc could produce a reading.
|
|
var errSampleUnavailable = errors.New("system metrics unavailable")
|
|
|
|
// cpuPercentSample returns a single-element slice giving overall CPU
|
|
// utilization, sourced from the same sampler that feeds the dashboard.
|
|
//
|
|
// Delegating matters because the two must agree: reading /proc/stat directly
|
|
// here meant /proc/stat's host-wide arithmetic again, so inside a container
|
|
// pinned to a fraction of the machine the alert could never reach its
|
|
// threshold even while the panel was throttled. One sampler, one answer.
|
|
//
|
|
// Unlike the old 150ms blocking window this only re-samples when the shared
|
|
// TTL has expired, so a notifier tick costs nothing when idle.
|
|
func cpuPercentSample() ([]float64, error) {
|
|
stats := system.GetSystemStats()
|
|
if stats == nil {
|
|
return nil, errSampleUnavailable
|
|
}
|
|
return []float64{stats.CPU.Percent}, nil
|
|
}
|
|
|
|
// memoryPercentSample returns overall memory used-percent, likewise shared with
|
|
// the dashboard — cgroup accounting inside a container instead of host meminfo,
|
|
// which describes a budget this process cannot use.
|
|
func memoryPercentSample() (float64, error) {
|
|
stats := system.GetSystemStats()
|
|
if stats == nil {
|
|
return 0, errSampleUnavailable
|
|
}
|
|
return stats.Memory.Percent, nil
|
|
}
|