I have a small board farm, with one of every device I have deployed, connected to many of the same sensors I have deployed, which I use to test new builds before deploying them. I've had too many cases where an update will cause a device to fail (some configuration does not align with some new software) or fail to be reachable (a network device's name will change and the interface won't come up) therefore I want to be able to test updates on devices within easy reach before deploying to locations that would be harder to reach physically.
The devices are mostly Rockchip-based deivces: rock pi e, rock pi s, and radxa zero 3e. Each one is connected to a different set of various sensors, mostly from Adafruit, that sense properties such as temperature, humidity, light, and barometric pressure. Sensors such as the bh1750, sht30 (indoor), sht30 (outdoor), shtc3, aht20, lps25, mcp3002, ds18b20 (reg temp), and ds18b20 (high temp) (plus some experiments with others).
A "server" device (rock 5b) polls each unit for its data, and this data (along with the poll latency) is recorded on the server. Recently I started adding more data to the list of values each device was reporting such as cpu temperature, uptime, loadavg, and various pieces of memory (RAM) information. Luckily I started adding RAM information just in time before one of my devices performed a kernel dump due to running out of memory.
Most of the boards can be purchased with various RAM amounts. For example the rock pi s can either have 256MB or 512MB of RAM. The boards I bought have 256MB. The rock-pi-e and zero-3e devices I use come with 1GB of RAM.
I had noticed memory issues with the latest kernel (6.17) on my rock pi s devices and decided to configure them to continue using the 6.12 kernel. systemd has a tendency to use a lot more RAM than sysvinit, therefore I had made some tweaks to the amount of memory it is allowed to use. In order to test to see if these changes had a positive effect on these devices, I updated the one in my board farm to use te 6.17 kernel again. Unfortunately the rock pi s device in my board farm still throws an out-of-memory kernel panic running the 6.17 kernel despite my systemd RAM-usage tweaks.
Here is a graph of the memory use of my test-rockpis boardfarm system (with 256MB of RAM) running the 6.12.55 kernel pretty much with the in-kernel defconfig. In this graph the system ran for almost 16 days. It didn't crash, after 16 days I simply uploaded a new RAUC bundle. The purple line is the total memory (230428), the blue line is the amount of available memory, and the green line is the amount of free memory:
Here's a graph of the same device running the next update, also running a 6.12.55 kernel, which ran for almost 22 days:
For the next update the only change I made was to switch from the 6.12.55 kernel to the 6.17.6 kernel. The total memory decreased from 230428 to 229636:
It only ran for about 4.5 days before issuing the following kernel panic:
508644.063311] Kernel panic - not syncing: System is deadlocked on memory [508644.063928] CPU: 2 UID: 0 PID: 1 Comm: systemd Not tainted 6.17.6-yocto-standard-00102-g5a817ec7a7 96-dirty #1 PREEMPT [508644.064898] Hardware name: Radxa ROCK Pi S (DT) [508644.065319] Call trace: [508644.065557] show_stack+0x18/0x30 (C) [508644.065921] dump_stack_lvl+0x60/0x80 [508644.066277] dump_stack+0x18/0x24 [508644.066602] vpanic+0x124/0x2e8 [508644.066909] abort+0x0/0x4 [508644.067176] out_of_memory+0x560/0x580 [508644.067540] __alloc_frozen_pages_noprof+0xc24/0xcf4 [508644.068007] alloc_pages_mpol+0xb4/0x1a4 [508644.068384] alloc_frozen_pages_noprof+0x44/0xc0 [508644.068822] new_slab+0x328/0x3b0 [508644.069142] ___slab_alloc+0x5dc/0x9c0 [508644.069503] __slab_alloc.isra.0+0x34/0x68 [508644.069893] __kmalloc_cache_noprof+0x168/0x2c0 [508644.070322] copy_verifier_state+0x1bc/0x1f8 [508644.070732] push_stack+0x7c/0x100 [508644.071062] check_cond_jmp_op+0x3d8/0x13cc [508644.071463] do_check_common+0x28ac/0x2cec [508644.071856] bpf_check+0x247c/0x3220 [508644.072204] bpf_prog_load+0x620/0xbe0 [508644.072564] __sys_bpf+0x7b8/0x205c [508644.072902] __arm64_sys_bpf+0x20/0x30 [508644.073264] invoke_syscall.constprop.0+0x40/0xf0 [508644.073712] el0_svc_common.constprop.0+0x38/0xd8 [508644.074160] do_el0_svc+0x1c/0x28 [508644.074485] el0_svc+0x34/0xe8 [508644.074789] el0t_64_sync_handler+0xa0/0xe4 [508644.075189] el0t_64_sync+0x198/0x19c [508644.075545] SMP: stopping secondary CPUs [508644.076001] Kernel Offset: 0x391170c00000 from 0xffff800080000000 [508644.076560] PHYS_OFFSET: 0xfff1000000000000 [508644.076952] CPU features: 0x000000,00010000,20002000,0400421b [508644.077484] Memory Limit: none [508644.077787] ---[ end Kernel panic - not syncing: System is deadlocked on memory ]---
For comparison, here is a plot of the memory usage of my rock-pi-e boardfarm device using a 6.17 kernel running for over 21 days:
[purple=total blue=available green=free]
Here's a plot of a radxa zero 3e device on a 6.17 kernel running for the same 21-22 days:
