How to Install Prometheus and Node Exporter on AlmaLinux 9

Prometheus is an open-source monitoring and alerting platform.

Originally, Prometheus is created by Soundcloud in 2012. Since then the Prometheus project adopted by some famous companies abe become a bigger project with very active developers and community. And In 2016, the Prometheus project is graduated under the Cloud Native Computing Foundation (CNCF).

Basic Concepts You Must Know

Basically, Prometheus collects data and metrics through HTTP endpoints from target servers, then stores all data as times series. In Prometheus, time-series data are identified by metric name and key/value pairs.

Prometheus provides flexibility through the Prometheus Query Language (PromQL). You can use PromQL to query the Prometheus time-series database.

On the target servers, you must install the ‘exporter’ application that exposes all data and metrics to Prometheus. ‘Node Exporter’ is a commonly used exporter to monitor Linux machines.

Node exporter exposes hardware and kernel-related matrics from Linux machines. It is a single binary file that will expose data end metrics to the Prometheus server.

This guide’ll show you installing Prometheus and Node Exporter on AlmaLinux 9 servers.

Prerequisites

Ensure that you have the following before starting:

• Two AlmaLinux 9 servers – This example will use a separate server for Prometheus and Node Exporter.
• A non-root user can execute sudo to get the root privileges.

Installing and Configuring Prometheus

Prometheus is an open-source and modern system monitoring and alerting system. It can be installed on traditional Unix/Linux servers via a pre-compiled package, using container technology such as Docker and Kubernetes, or configuration management tools such as Ansible, Chef, Puppet, and Saltstack.

The following section shows you how to install Prometheus via a prebuilt binary file to an ALmaLinux 9 server.

Complete these step-by-steps for successful installation of Prometheus on your AlmaLinux server:

• Setup User and Directories
• Downloading Prometheus Binary Package
• Configuring Prometheus Server
• Running Prometheus in the Background as a Systemd Service

Setup User and Directories

Before installing prometheus, you will be creating a new user and directories on your system. This user will be used to run Prometheus, and some of the directories will be used to store Prometheus configuration files and data.

Run the following command to create a new system user prometheus.
sudo adduser -M -r -s /sbin/nologin prometheus

Run the command below to create a new directory /etc/prometheus that will be used to store Prometheus configuration files and the directory /var/lib/prometheus that will be used as the data directory for Prometheus.
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus

Now run the command below to change the ownership of both /etc/prometheus and /var/lib/prometheus directories to the user prometheus and group prometheus.
sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus

Downloading Prometheus Pre-Compiled Package

Now, you will install prometheus on your server. In this demo, you will install Prometheus via a pre-compiled binary package. You will install Prometheus on an AlmaLinux server with IP address 192.168.10.20.

Run the dnf command below to install wget to your system.
sudo dnf install wget -y

Move your working directory to /usr/src and download the pre-compiled binary file of Prometheus via wget.
cd /usr/src
wget https://github.com/prometheus/prometheus/releases/download/v2.44.0/prometheus-2.44.0.linux-amd64.tar.gz

Now extract the Prometheus binary file prometheus-2.44.0.linux-amd64.tar.gz using the tar command below. You should get a new directory prometheus-2.44.0.linux-amd64 that contains the Prometheus binary file and some of the default configurations.
tar -xf prometheus-2.44.0.linux-amd64.tar.gz

Run the command below to copy the default Prometheus configuration file prometheus.yml and some important directories to /etc/prometheus/.
sudo cp /usr/src/prometheus-*/prometheus.yml /etc/prometheus/
sudo cp -r /usr/src/prometheus-*/consoles /etc/prometheus
sudo cp -r /usr/src/prometheus-*/console_libraries /etc/prometheus

After that, change the ownership of the /etc/prometheus/ directory to the user promtheus and group prometheus using the following command.
sudo chown -R prometheus:prometheus /etc/prometheus

Next, run the following command to copy the binary file of prometheus and promtool to the /usr/bin/ directory.
sudo cp /usr/src/prometheus-*/prometheus /usr/bin/
sudo cp /usr/src/prometheus-*/promtool /usr/bin/

Then, verify the prometheus and promtool using the following command.
which prometheus
prometheus --version
which promtool
promtool --version

If the installation is successful, you should see the full path of the prometheus and promtool command, which is located in the /usr/bin/ directory. You will also see the installed version of Prometheus.

Configuring Prometheus

After prometheus is downloaded and installed, you will start the basic configuration of Prometheus.

Open the prometheus configuration file /etc/prometheus/prometheus.yml using the nano editor.
nano /etc/prometheus/prometheus.yml

At the scrape_configs parameter, add the new job prometheus with the target of the Prometheus server itself. In this example, the Prometheus server has an IP address of 192.168.10.20, and this will scrape metrics for Prometheus.
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    static_configs:
      - targets: ["192.168.10.20:9090"]

Save and close the file when finished.

Next, create a new systemd service file for prometheus to /etc/systemd/system/prometheus.service using the nano editor command.
sudo nano /etc/systemd/system/prometheus.service

Add the following configuration to run Prometheus as a systemd service.
[Unit]Description=Prometheus
Wants=network-online.target
After=network-online.target

Save the file and exit the editor when you’re finished.

Now run the following command to reload the systemd manager and apply the changes to the systemd.
sudo systemctl daemon-reload

Then, start and enable the prometheus service using the systemctl command below.
sudo systemctl start prometheus
sudo systemctl enable prometheus

Verify the prometheus service to ensure that the service is running.
sudo systemctl status prometheus

If your installation is successful, you should see the prometheus service status is active (running).

Lastly, launch your web browser and visit the server IP address followed by port 9090 (i.e http://192.168.10:20:9090/). If the installation is successful, you should get the Prometheus graph page like this:

You can also verify the monitoring endpoint status by clicking Status > Targets. You should see the job_name prometheus with state UP.

Installing and Configuring Node Exporter on the Target Monitoring

Node Exporter is an application by Prometheus that expose system metrics of target monitoring. It allows you to scrape a variety of system metrics and can be exposed for further process to Prometheus.

In this section, you will install Node Exporter on an AlmaLinux 9 server with IP address 192.168.10.21. You will install Node Exporter via a pre-compiled binary package provided by Prometheus.

Downloading and Installing Node Exporter

To install Node Exporter, you must create a dedicated system user for it and download the binary package of Node Exporter to your system.

First, run the following command to create a new system user node_exporter.
sudo adduser -M -r -s /sbin/nologin node_exporter

Install wget using the dnf command below.
sudo dnf install wget -y

Then, move to the /usr/src directory and download the pre-compiled binary file of Node Exporter via wget.
cd /usr/src/
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.0/node_exporter-1.6.0.linux-amd64.tar.gz

Once the downloaded is finished, you should see the file node_exporter-1.6.0.linux-amd64.tar.gz. Extract it using the tar command below.
tar -xf node_exporter-1.6.0.linux-amd64.tar.gz

Next, move the binary file of Node Exporter to the /usr/bin directory using the command below.
mv node_exporter-*/node_exporter /usr/bin

Now verify Node Exporter using the command below. If successful, you should get the full path binary path and the version of Node Exporter.
which node_exporter
node_exporter --version

Running Node Exporter in the Background as a Service

After the Node Exporter is installed, you will set up and run Node Exporter in the background as a systemd service.

To do that, create a new systemd service file /etc/systemd/system/node_exporter.service using the nano editor command below. In this example, you will run Node Exporter as a systemd service.
sudo nano /etc/systemd/system/node_exporter.service

Insert the following configuration into the file.
[Unit]Description=Node Exporter
After=network.target

Save and exit the file when you’re done.

Now run the systemctl command below to reload the systemd manager.
sudo systemctl daemon-reload

Then start and enable the node_exporter service using the following command.
sudo systemctl start node_exporter
sudo systemctl enable node_exporter

Once node_exporter is started and enabled, run the following command to verify it and ensure that the service is running.
sudo systemctl status node_exporter

If running, you should see the node_exporter service with the status active (running).

Lastly, run the following command to verify the port of Node Exporter. By default, it’s running on port 9100.
ss -tulpn | grep node

You should see that port 9100 is used by the node_exporter service.

You can also verify it via web browser by visiting the server IP address followed by port 9100 (i.e: http://192.168.10.21:9100/). You should see the endpoint data of server monitoring via Node Exporter.

Adding Node Exporter to Prometheus Server

In this section, you will add the Node Exporter to the Prometheus server. You have installed and exposed system metrics on the server 192.168.10.21, then you will now add it to the Prometheus server that running on 192.168.10.20.

Open the Prometheus configuration /etc/prometheus/prometheus.yml using the following nano editor.
sudo nano /etc/prometheus/prometheus.yml

Withing the scrape_config parameter, add new job_name node_exporter_metrics and input the detail endpoint of the Node Exporter. The following example will be monitoring the server node1 with IP address 192.168.10.21.
  - job_name: 'node_exporter_metrics'
    scrape_interval: 5s
    static_configs:
      - targets: ['192.168.10.21:9100']

Save the file and exit the editor after finishing.

Now run the following command to restart the prometheus service and apply the changes.
sudo systemctl restart prometheus

Now open the prometheus from your web browser and click Status > Targets to ensure that the Node Exporter is added. If successful, you should see the job node_exporter_metrics with state UP.

Next, move to the Graph menu and input the query beginning with node_ to get the list of queries provided by Node Exporter.

Input the query node_network_iface_id to get the list of network devices on the server 192.168.10.21.

You can also verify the details of OS information via node_os_info query.

Lastly, type the query node_load15 and click on the Graph tab to get the status of the system load for the last 15 minutes.

Conclusion

Excellent job! You have successfully installed Prometheus on AlmaLinux 9 server and also monitored an AlmaLinux server via Node Exporter.

You can now monitor another server or monitor your application via the Prometheus exporter. You can also integrate your Prometheus with Grafana for visualization of your monitoring system.