Note: It seems my original post from last week didn’t get posted on lemmy.world from kbin (I can’t seem to find it) so I’m reposting it. Apologies to those who may have already seen this.
I’m looking to deploy some form of monitoring across my selhosted servers and I’m a bit confused about the different options.
I have a small network of three machines that I would like to monitor. I am not looking for a solution that lets me monitor tens, hundreds, or thousands of nodes. Furthermore, I am more interested in being able to observe metrics for each node individually rather than in aggregate. Each of these machines performs a different task so aggregate metrics from these machines are not particularly meaningful. However, collecting all the metrics centrally so that I can have a single dashboard to view them all in one convenient place is definitely something I would like.
With that said, I have been trying to understand the different (popular) options that are available and I would like to hear what the community’s experience is with these options and if anybody has any advice on any of these in light of my requirements above.
Prometheus seems like the default go-to for monitoring. This would require deploying a node_exporter on each node, a prometheus service, and a grafana dashboard. That’s all fine, I can do that. However, from all that I’m reading it doesn’t seem like Prometheus is optimised for my use case of monitoring each node individually. I’m sure it’s possible, but I’m concerned that because this is not what it’s meant for, it would take me ages to set it up such that I’m happy with it.
Netdata seems like a comprehensive single-device monitoring solution. It also appears that it is possible to run your own registry to help with distributed monitoring. Not gonna lie, the netdata dashboard looks slick. An important additional advantage is that it comes packaged on Debian (all my machines run Debian). However, it looks like it does not store the metrics for very long. To solve that I could also set up InfluxDB and Grafana for long-term metrics. I could use Prometheus instead of InfluxDB in this arrangement, but I’m more likely to deploy a bunch of IoT devices than I am to deploy servers needing monitoring which means InfluxDB is a bit more future-proof for me as it could be reused for IoT data.
Cockpit is another single-device solution which additionally provides direct control of the system. The direct control is probably not so much of a plus as then I would never let Cockpit be accessible from outside my home network whereas I wouldn’t mind that so much for dashboards with read-only data (still behind some authentication of course). It’s also probably not built for monitoring specifically, but I included this in the list in case somebody has something interesting to say about it.
What’s everybody’s experience with the above solutions and does anybody have advice specific to my situation? I’m currently leaning to netdata with my own registry at first and later add InfluxDB and Grafana for long-term metrics.
I am more interested in being able to observe metrics for each node individually rather than in aggregate.
This requirement makes me think netdata would be a good solution. In my current setup, each host has its own netdata dashboard and manages its own health checks/alarms. I have also enabled streaming which sends metrics from all hosts to a “parent/master” netdata instance from which I can see all metrics from all hosts without checking each dashboard individually.
However, it looks like it does not store the metrics for very long.
I still have to look into this, in the past it was certainly true and you had to setup a prometheus instance to store (and downsample, who needs few-seconds resolution for one year old metrics) metrics for long-term archival - but looking at the documentation right now, it looks possible to store long-term metrics in the netdata DB itself, by moving old metrics to a lower-definition storage tier: https://learn.netdata.cloud/docs/configuring/optimizing-metrics-database/change-how-long-netdata-stores-metrics
An important additional advantage is that it comes packaged on Debian (all my machines run Debian).
Same. However I install and update it from their third-party APT repository - it’s one of the rare cases where I prefer upstream releases to Debian stable packages, the last few upstream releases have been really nice (for example I’m not sure the new tiered retention system is availabel in the v1.37.1 Debian stable package)
My automated installation procedure (ansible role) is here if you’re interested (start at tasks/main.yml and follow the import_tasks).
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !selfhosted@lemmy.world
A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don’t control.
Rules:
Be civil: we’re here to support and learn from one another. Insults won’t be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it’s not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don’t duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
Resources:
selfh.st Newsletter and index of selfhosted software and apps
This requirement makes me think netdata would be a good solution. In my current setup, each host has its own netdata dashboard and manages its own health checks/alarms. I have also enabled streaming which sends metrics from all hosts to a “parent/master” netdata instance from which I can see all metrics from all hosts without checking each dashboard individually.
I still have to look into this, in the past it was certainly true and you had to setup a prometheus instance to store (and downsample, who needs few-seconds resolution for one year old metrics) metrics for long-term archival - but looking at the documentation right now, it looks possible to store long-term metrics in the netdata DB itself, by moving old metrics to a lower-definition storage tier: https://learn.netdata.cloud/docs/configuring/optimizing-metrics-database/change-how-long-netdata-stores-metrics
Same. However I install and update it from their third-party APT repository - it’s one of the rare cases where I prefer upstream releases to Debian stable packages, the last few upstream releases have been really nice (for example I’m not sure the new tiered retention system is availabel in the v1.37.1 Debian stable package)
My automated installation procedure (ansible role) is here if you’re interested (start at
tasks/main.ymland follow theimport_tasks).Thanks a lot for these tips! Especially about using the upstream deb.