homelab

I run quite a bit of hardware in order to facilitate all of the open-source software that runs in the background of my everyday life. I've found that giving servers discrete, memorable names that pertain to some sort of logical grouping of their function decreases cognitive load. As such, I've chosen to name all of my servers after k-pop girl groups.

For observability and monitoring, all nodes run cAdvisor and node_exporter which are collected by prometheus. Container logs are written to file and collected using promtail.

I've found that promtail is much more well-suited to my use case due to its low resource consumption and the ability to dynamically add/drop from logs via config. My default setup is to keep debug logs on, even in production, and then drop the debug lines via promtail. This allows for me to "turn on" debug logs at a moment by simply changing the promtail config on that node, without having to restart the corresponding service.