Kiến Thức Linux Monitoring

MongoDB Monitoring with Grafana & Prometheus

congdonglinux-mongodb-monitoring

Whether you are a database administrator, or an web application developer, it is very likely that your infrastructure relies on MongoDB in some ways.

As a consequence, monitoring MongoDB is crucial to ensure that you are not having performance or memory issues with your database.

In 2019, there are many ways to monitor your MongoDB instance.

Today, we are taking a look at how to monitor your MongoDB database with Grafana and Prometheus.

I – What You Will Learn

As always, here are the key points that you will learn if you follow this tutorial until the end:

  • How to install Prometheus, a modern time series database on your computer;
  • How to setup the MongoDB developed by Percona as well as binding it to MongoDB;
  • What your Grafana – Prometheus – MongoDB exporter will look like
  • How to configure import a MongoDB dashboard in seconds

Note : Percona’s MongoDB exporter includes MongoDB stats for sharding and replica, as an evolution of David Cuadrado’s MongoDB exporter

II – MongoDB, Grafana and Prometheus Architecture

Similarly to the architecture we described in our MySQL monitoring tutorial, here’s a complete overview of what the final monitoring architecture looks like.

As a quick reminder, Prometheus scrapes targets. Targets may be instrumented applications (like instrumented Java apps for example), the Pushgateway or exporters.

Exporters are a way to bind to an existing entity (a database, a reverse proxy server, an application server) to expose metrics to Prometheus.

The MongoDB exporter is one of them.

Prometheus will bind to the MongoDB exporters and store related metrics in its own internal storage system.

From there, Grafana will bind to Prometheus and display metrics on dashboard panels.

Easy, isn’t it?

Now that you have a great understanding of what we are trying to build, let’s install the different tools needed to monitor MongoDB.

III – Installing The Different Tools

a – Installing Prometheus

If you are still a beginner using Prometheus, you will find the complete Prometheus installation on this tutorial.

If you run the Prometheus installation entirely, you know have your Prometheus up and ready.

To verify it, head over to http://localhost:9090. If you have a web interface close to the one presented just below, it means that your installation went correctly.

No metrics are currently stored, except maybe Prometheus internal metrics.

Run a simple prometheus query, such as prometheus_http_request_duration_seconds_sum and make sure that you have some results.

Prometheus web interface

Now that your Prometheus server is running, let’s install the MongoDB exporter to start monitor our MongoDB database.

b – Installing the MongoDB exporter

As explained before, the MongoDB exporter is available on Percona’s GitHub here.

The MongoDB exporter comes as a binary file in an archive, but as always, we are going to configure it as a service.

We are also going to configure it to run on a specific authenticated user dedicated to monitoring.

First, download the MongoDB exporter release file from one of the versions available here.

$ mkdir mongodb-exporter
$ cd mongodb-exporter
$ wget https://github.com/percona/mongodb_exporter/releases/download/v0.7.1/mongodb_exporter-0.7.1.linux-amd64.tar.gz

Note : as of June 2019, the MongoDB exporter version is 0.7.1.

Next, extract the downloaded archive in your current folder.

$ tar xvzf mongodb_exporter-0.7.1.linux-amd64.tar.gz

You should now have 4 files: mongodb_exporter, LICENSE, README.md, CHANGELOG.md.

All files are pretty self explanatory, but we are only interested in the mongodb_exporter binary file here.

As we are going to configure the exporter as a service, create a dedicated user for Prometheus if not already existing.

$ sudo useradd -rs /bin/false prometheus
$ sudo mv mongodb_exporter /usr/local/bin/

c – Enabling MongoDB authentication

Every access to your MongoDB instance should be authenticated and authorized.

To ensure it, we are going to setup a basic MongoDB authentication for the MongoDB exporter.

MongoDB authentication is set using the –auth flag in the Mongo shell.

By default, mongod does not set this flag, so you should be able to connect to it via localhost.

$ ps aux | grep mongod
mongodb  13468  1.1  6.9 1490632 281728 ? Ssl  Jan05 2838:27 /usr/bin/mongod --unixSocketPrefix=/run/mongodb --config /etc/mongodb.conf

Connect to your MongoDB instance with mongo.

$ mongo --port 27017

Create an administrator account for your exporter with the cluster monitor role.

use admin
db.createUser(
  {
    user: "mongodb_exporter",
    pwd: "password",
    roles: [
        { role: "clusterMonitor", db: "admin" },
        { role: "read", db: "local" }
    ]
  }
)

You should see the following message

Successfully added user: {                        
        "user" : "mongodb_exporter",              
        "roles" : [                               
                {                                 
                        "role" : "clusterMonitor",
                        "db" : "admin"            
                },                                
                {                                 
                        "role" : "read",          
                        "db" : "local"            
                }                                 
        ]                                         
}                                                 

Before exiting, shutdown your MongoDB instance, and restart it.

$ db.adminCommand( { shutdown: 1 } )
$ exit
$ sudo mongod --auth --port 27017 --config /etc/mongodb.conf &

Set your MongoDB URI environment variable, according to the changes that you made before.

$ export MONGODB_URI=mongodb://mongodb_exporter:password@localhost:27017

d – Creating a service for the MongoDB exporter

Similarly to the MySQLd exporter, we are going to setup the MongoDB exporter as a service.

As usual, head over to /lib/systemd/system and create a new service file for the exporter.

$ cd /lib/systemd/system/
$ sudo touch mongodb_exporter.service

Parse the following configuration into your service file:

[Unit]
Description=MongoDB Exporter
User=prometheus

[Service]
Type=simple
Restart=always
ExecStart=/usr/local/bin/mongodb_exporter

[Install]
WantedBy=multi-user.target

From there, don’t forget to restart your system daemon and to start your service.

$ sudo systemctl daemon-reload
$ sudo systemctl start mongodb_exporter.service

You should always verify that your service is working.

As a quick reminder, Percona’s MongoDB exporter runs on port 9216.

To ensure that everything is working correctly, run a simple curl command on the port 9216.

$ sudo curl http://localhost:9216/metrics
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
...

Can you already see some Prometheus metrics that are being aggregated already?

Great! Your MongoDB exporter is working!

We only need to bind it to Prometheus, and we should be all set.

e – Configure the MongoDB exporter as a Prometheus target

Almost there!

As described in the schema shown in the architecture section, we are going to bind Prometheus to the new MongoDB exporter.

Head over to the location of your Prometheus configuration file (mine is located at /etc/prometheus/prometheus.yml) and edit it to add the MongoDB exporter as a target.

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    static_configs:
            - targets: ['localhost:9090', 'localhost:9216']

Restart Prometheus, and go to http://localhost:9090/targets to verify that Prometheus is now binded to your newly added exporter.

Prometheus target pages

Great! Everything is up and running now.

The last step will be to create a complete Grafana dashboard to have some insights on our metrics.

Looking for a tutorial to install Grafana? We got it covered in our last article.

III – Building Awesome MongoDB dashboards

As described before, we are going to use dashboards built by Percona in order to monitor our MongoDB instance on Grafana.

Percona provides multiple existing dashboards such as:

  • MongoDB Overview;
  • MongoDB ReplSet;
  • MongoDB RocksDB;
  • MongoDB WiredTiger;
  • MongoDB MMAPv1
  • MongoDB InMemory

For this part, we are going to focus on importing the MongoDB Overview dashboards into our Grafana instance.

a – Set Prometheus as a Grafana datasource

If you followed our previous tutorials, you are probably a master at it right now.

If that’s not the case, here’s the configuration for Prometheus on Grafana.

Configuring Prometheus in Grafana

b – Downloading Percona dashboards for Grafana

In Grafana, dashboards come as JSON files. When you create a dashboard, you can export it in this format and share it with the world.

Percona provides dozens of dashboards on its Github repository.

In this case, we are going to download the MongoDB Overview dashboard file.

Run a simple wget command to get the JSON file.

You can create a dedicated dashboards folder in your /etc/grafana folder to store your downloaded dashboards.

$ cd /etc/grafana
$ sudo mkdir dashboards
$ cd dashboards
$ sudo wget https://github.com/percona/grafana-dashboards/blob/master/dashboards/MongoDB_Overview.json

If you want all the dashboards available in the repository, simply clone the repository into your current folder.

$ sudo git clone https://github.com/percona/grafana-dashboards.git

Now that you have the JSON files available, it is time for us to import them into Grafana.

c – Importing the MongoDB dashboard in Grafana

For this example, we are going to import the MongoDB Overview dashboard for Grafana.

Head over to your Grafana instance, located at http://localhost:3000 (if you didn’t change any default ports in your configuration file)

On the left menu, hover the ‘Plus‘ icon with your mouse and click on Import.

Grafana import dashboard

From there, you should be taken to the Import page. Click on the Upload .json file option.

Grafana import dashboard window

Given the operating system you are working with, navigate to the /etc/grafana folder (where you stored your dashboard), and click on the MongoDB_Overview.json file.

Your dashboard should be imported automatically, with real time updates of your MongoDB database!

Final dashboard

IV – Common Errors

If you carefully followed this tutorial, chances are that you have a fully functional dashboard right now.

However, you might encounter some errors during the process.

Here are some clues on how to solve them:

  • Failed to get server status: not authorized on admin to execute command

This error message is fairly simple to understand.

Your mongodb_exporter user does not have the necessary credentials to perform queries on the admin database.

Failed to get server status: not authorized on admin to execute command

Clue 1

To resolve it, connect to your instance, use the admin database, and make sure that you configured correctly the mongodb_exporter user (it must have the cluster monitor right on the admin database)

$ mongo --port 27017 (or --auth if you already have an admin account on your database)
$ use admin;
$ db.getUsers()
{
        "_id" : "admin.mongodb_exporter",         
        "user" : "mongodb_exporter",              
        "db" : "admin",                           
        "roles" : [                               
                {                                 
                        "role" : "clusterMonitor",
                        "db" : "admin"            
                },                                
                {                                 
                        "role" : "read",          
                        "db" : "local"            
                }                                 
        ]                                         
}                                               

Clue 2

You didn’t not properly set the MongoDB URI environment variable.

To verify it, launch the following command:

$ env  | grep mongodb
MONGODB_URI=mongodb://mongodb_exporter:password@localhost:27017

Clue 3

If this is still not working, set the MongoDB URI directly in the service file, restart your service, as well as your MongoDB instance.

[Service]
Type=simple
Restart=always
ExecStart=/usr/local/bin/mongodb_exporter --mongodb.uri=mongodb://mongodb_exporter:password@localhost:27017

$ sudo systemctl daemon-reload
$ sudo systemctl restart mongodb.service
$ sudo systemctl restart mongodb_exporter.service
  • Could not get MongoDB BuildInfo: no reachable servers!

Clue 1

Your MongoDB database is either not launched, not it is not running on the port 27017.

For the first option, just verify that your MongoDB service is running, or that your mongod background is running.

$ sudo systemctl status mongodb.service
or
$ ps aux | grep mongod

For the second option, verify the port used by your MongoDB instance. To do so, run the following command:

$ sudo lsof -i -P -n | grep LISTEN
grafana-s   642         grafana    7u  IPv6  998256601      0t0  TCP *:3333 (LISTEN)
mysqld_ex  3136            root    3u  IPv6  998030854      0t0  TCP *:9104 (LISTEN)
mongod     3688            root   10u  IPv4 1092070442      0t0  TCP 127.0.0.1:27017 (LISTEN)

If your MongoDB instance is not running on the default 27017 port, change your mongodb_exporter file for it to bind to your custom port.

[Service]
mongodb.uri=mongodb://mongodb_exporter:password@localhost:12345
Could not get MongoDB BuildInfo: no reachable servers!

V – Going Further

Now that you have a fully operational monitoring pipeline for MongoDB, it is time for you to dig a little deeper.

Here are the best resources if you want to know more about MongoDB monitoring.

First, here’s a great talk by Vadim Tkachenko from Percona about Monitoring MySQL and MongoDB instances. You will understand how Percona builds its own monitoring architecture and their own dashboards.

Đăng ký liền tay Nhận Ngay Bài Mới

Subscribe ngay

Cám ơn bạn đã đăng ký !

Lỗi đăng ký !

Add Comment

Click here to post a comment

Đăng ký liền tay
Nhận Ngay Bài Mới

Subscribe ngay

Cám ơn bạn đã đăng ký !

Lỗi đăng ký !