Buy Percona ServicesBuy Now!
Subscribe to Latest MySQL Performance Blog posts feed
Updated: 49 min 18 sec ago

Percona XtraDB Cluster 5.6.35-26.20 is now available

March 13, 2017 - 10:32am

Percona announces the release of Percona XtraDB Cluster 5.6.35-26.20 on March 10, 2017. Binaries are available from the downloads section or our software repositories.

Percona XtraDB Cluster 5.6.35-26.20 is now the current release, based on the following:

All Percona software is open-source and free. Details of this release can be found in the 5.6.35-26.20 milestone on Launchpad.

There are no new features or bug fixes to the main components, besides upstream changes and the following fixes related to packaging:

  • BLD-593: Limited the use of rm and chown by mysqld_safe to avoid exploits of the CVE-2016-5617 vulnerability. For more information, see 1660265.
    Credit to Dawid Golunski (https://legalhackers.com).
  • BLD-610: Added version number to the dependency requirements of the full RPM package.
  • BLD-645: Fixed mysqld_safe to support options with a forward slash (/). For more information, see 1652838.

Help us improve our software quality by reporting any bugs you encounter using our bug tracking system. As always, thanks for your continued support of Percona!

Troubleshooting MySQL access privileges issues: Q & A

March 10, 2017 - 4:36pm

In this blog, I will provide answers to the Q & A for the Troubleshooting MySQL Access Privileges Issues webinar.

First, I want to thank everybody for attending the February 23 webinar. The recording and slides for the webinar are available here. Below is the list of your questions that I wasn’t able to answer during the webinar, with responses:

Q: Should the root@localhost user be given ALL privileges or Super privileges? Does All include Super privileges also?

A: Yes, you should have a user with all privileges. Better if this user has access from localhost only. ALL  includes SUPER.

Q: We have users who connect via a laptop that get dynamic IP addresses, so granting access with a server name is an easier way to manage these users. Can I grant access to a MySQL database with a hostname as opposed to an ipaddress? For example “myname@mymachine.mydomain.com” as opposed to “myname@10.10.10.10”?  Is the host cache/performance_schema required for this?

A: Yes, you can.

But it looks like I was not clear about host cache. Host cache is an internal structure that is always available and contains answers from DNS server. You cannot enable or disable it. Until version 5.6, you also could not control it. For example, if the cache got corrupted the only thing you could do is to restart the server. Version 5.6 the table HOST_CACHE was introduced to Performance Schema. With this table you can examine the content of the host cache and truncate it if needed.

Q: If there are multiple entries in the user table that match the connecting user (e.g., with wildcards, hostname, and IP), what rules does MySQL use to select which is used for authentication?  Does it try multiple ones until it gets a password match?

A: Not, mysqld does not try to hack your passwords. Instead it sorts the user table by name and host in descending order as I showed on slide #37 (page 110). Then it takes the first matching row. So if you created users foo@somehost, foo@some% and foo@1.2.3.4, and you connect as foo from somehost, mysqld first checks the user name and then chooses the first matching row foo@somehost. If you instead connect as foo from someotherhost, mysqld chooses foo@some%. An IP-based host is chosen if either mysqld started with option skip-networking or if 1.2.3.4  points to a host whose name does not start with “some”.

Mixing IP-based and name-based hosts is dangerous in situations when the same host can be resolved as somehost or 1.2.3.4. In this case, if something goes wrong with the host cache or DNS server, the wrong entry from the user table can be chosen. For example, if you initially had three hosts: uniquehost (which resolves as 1.2.3.4), somehost (which resolves as 4.3.2.1) and someothershost (which resolves as 4.3.2.2). Now you decided to re-locate uniquehost to a machine with IP 1.2.3.5 and use IP 1.2.3.4 for the host with name someyetanotherhost. In this case, the clients from the machine with IP 1.2.3.4 will be treated as foo@some%, which isn’t what you want.

To demonstrate this issue, I created two users and granted two different privileges to them:

mysql> create user sveta@Thinkie; Query OK, 0 rows affected (0,01 sec) mysql> create user sveta@'192.168.0.4'; Query OK, 0 rows affected (0,00 sec) mysql> grant all on *.* to 'sveta'@'Thinkie'; Query OK, 0 rows affected (0,00 sec) mysql> grant all on db1.* to 'sveta'@'192.168.0.4'; Query OK, 0 rows affected (0,00 sec)

Now I modified my /etc/hosts  file and pointed address 192.168.0.4  to name Thinkie:

127.0.0.1 localhost # 127.0.1.1 Thinkie 192.168.0.4 Thinkie

Now, if I connect as sveta both Thinkie and 192.168.0.4 are resolved to the same host:

sveta@Thinkie:$ mysql -hThinkie -usveta ... mysql> select user(), current_user(); +---------------+----------------+ | user() | current_user() | +---------------+----------------+ | sveta@Thinkie | sveta@thinkie | +---------------+----------------+ 1 row in set (0,00 sec) mysql> show grants; +--------------------------------------------------+ | Grants for sveta@thinkie | +--------------------------------------------------+ | GRANT ALL PRIVILEGES ON *.* TO 'sveta'@'thinkie' | +--------------------------------------------------+ 1 row in set (0,00 sec) mysql> q Bye sveta@Thinkie:$ mysql -h192.168.0.4 -usveta ... mysql> show grants; +--------------------------------------------------+ | Grants for sveta@thinkie | +--------------------------------------------------+ | GRANT ALL PRIVILEGES ON *.* TO 'sveta'@'thinkie' | +--------------------------------------------------+ 1 row in set (0,00 sec) mysql> select user(), current_user(); +---------------+----------------+ | user() | current_user() | +---------------+----------------+ | sveta@Thinkie | sveta@thinkie | +---------------+----------------+ 1 row in set (0,00 sec) mysql> q Bye

Now I modified the /etc/hosts  file and pointed Thinkie  back to 127.0.0.1  (localhost):

127.0.0.1 localhost 127.0.1.1 Thinkie # 192.168.0.4 Thinkie

But host 192.168.0.4 still resolves to Thinkie:

sveta@Thinkie:$ mysql -h192.168.0.4 -usveta ... mysql> select user(), current_user(); +---------------+----------------+ | user() | current_user() | +---------------+----------------+ | sveta@Thinkie | sveta@thinkie | +---------------+----------------+ 1 row in set (0,00 sec) mysql> show grants; +--------------------------------------------------+ | Grants for sveta@thinkie | +--------------------------------------------------+ | GRANT ALL PRIVILEGES ON *.* TO 'sveta'@'thinkie' | +--------------------------------------------------+ 1 row in set (0,00 sec) mysql> q Bye

The reason for this is a stalled host cache, which can be easily observable with Performance Schema:

sveta@Thinkie:$ mysql -uroot ... mysql> select * from performance_schema.host_cacheG *************************** 1. row *************************** IP: 192.168.0.4 HOST: Thinkie HOST_VALIDATED: YES SUM_CONNECT_ERRORS: 0 COUNT_HOST_BLOCKED_ERRORS: 0 COUNT_NAMEINFO_TRANSIENT_ERRORS: 0 COUNT_NAMEINFO_PERMANENT_ERRORS: 0 COUNT_FORMAT_ERRORS: 0 COUNT_ADDRINFO_TRANSIENT_ERRORS: 0 COUNT_ADDRINFO_PERMANENT_ERRORS: 0 COUNT_FCRDNS_ERRORS: 0 COUNT_HOST_ACL_ERRORS: 0 COUNT_NO_AUTH_PLUGIN_ERRORS: 0 COUNT_AUTH_PLUGIN_ERRORS: 0 COUNT_HANDSHAKE_ERRORS: 0 COUNT_PROXY_USER_ERRORS: 0 COUNT_PROXY_USER_ACL_ERRORS: 0 COUNT_AUTHENTICATION_ERRORS: 0 COUNT_SSL_ERRORS: 0 COUNT_MAX_USER_CONNECTIONS_ERRORS: 0 COUNT_MAX_USER_CONNECTIONS_PER_HOUR_ERRORS: 0 COUNT_DEFAULT_DATABASE_ERRORS: 0 COUNT_INIT_CONNECT_ERRORS: 0 COUNT_LOCAL_ERRORS: 0 COUNT_UNKNOWN_ERRORS: 0 FIRST_SEEN: 2017-03-02 23:19:32 LAST_SEEN: 2017-03-02 23:20:31 FIRST_ERROR_SEEN: NULL LAST_ERROR_SEEN: NULL 1 row in set (0,00 sec) mysql> truncate performance_schema.host_cache; Query OK, 0 rows affected (0,00 sec) mysql> q Bye

After I truncated table host_cache the numeric host resolves as I expect:

sveta@Thinkie:$ mysql -h192.168.0.4 -usveta ... mysql> show grants; +----------------------------------------------------------+ | Grants for sveta@192.168.0.4 | +----------------------------------------------------------+ | GRANT USAGE ON *.* TO 'sveta'@'192.168.0.4' | | GRANT ALL PRIVILEGES ON `db1`.* TO 'sveta'@'192.168.0.4' | +----------------------------------------------------------+ 2 rows in set (0,00 sec) mysql> select user(), current_user(); +-------------------+-------------------+ | user() | current_user() | +-------------------+-------------------+ | sveta@192.168.0.4 | sveta@192.168.0.4 | +-------------------+-------------------+ 1 row in set (0,00 sec) mysql> q Bye

Q: What privileges are required for a non-root or non-super user to be to use mysqldump to dump the database and then restore it on a different server?

A: Generally you should have SELECT  privilege on all objects you are going to dump. If you are dumping views, you also should have SHOW VIEW  privilege in order to run SHOW CREATE TABLE. If you want to dump stored routines/events, you need access to them as well. If you use option --lock-tables or --lock-all-tables, you should have the LOCK  privilege.

Q: If the max_connection value is reached in MySQL, can root@localhost with ALL privilege still login, or with Super privilege user can login?

A: ALL includes SUPER, so a user with ALL  privilege can login. Just note there can be only one such connection, thus do not grant SUPER  or ALL privilege to the application user.

Q: Is it possible to remove a priv at a lower level? In other words, grant select and delete at the database level, but remove delete for a specific table?  Or can privs only be added to?

A: Not, MySQL will reject such a statement:

mysql> show grants for sveta@'192.168.0.4'; +----------------------------------------------------------+ | Grants for sveta@192.168.0.4 | +----------------------------------------------------------+ | GRANT USAGE ON *.* TO 'sveta'@'192.168.0.4' | | GRANT ALL PRIVILEGES ON `db1`.* TO 'sveta'@'192.168.0.4' | +----------------------------------------------------------+ 2 rows in set (0,00 sec) mysql> revoke update on db1.t1 from sveta@'192.168.0.4'; ERROR 1147 (42000): There is no such grant defined for user 'sveta' on host '192.168.0.4' on table 't1'

Q: How can we have DB user roles… like a group of grants for a particular role?

A: You have several options.

  1. Use MariaDB 10.0.5 or newer. You can read about roles support in MariaDB here
  2. Use MySQL 8.0. You can read about roles in MySQL 8.0 here
  3. With MySQL 5.7: imitate roles as I showed on slide 19 (pages 53 – 60)
  4. With MySQL 5.5 and 5.6: use the same method as shown on slides, but use the custom authentication plugin that supports proxy users.
  5. Always: create a template with privileges, assign privileges to each user manually.

Q: How would you migrate role simulation with proxy users to actual roles in MySQL 8.x?

A: I would drop the proxied user and create a role with the same privileges instead, then grant the proxy user the newly created role instead of PROXY.

Q: Is there a plugin to integrate Active Directory and MySQL in order to use Active Directory groups?

A: There is commercial Windows Authentication Plugin, available in versions 5.5 and newer. You can also use the open source Percona PAM authentication plugin and connect it to Active Directory the same way as can be done for LDAP. There is a blog post describing how to do so, but I’ve never used this method myself.

Q: Can we use central auth with MySQL?

A: Yes, with the help of the PAM Plugin. There are tutorials for LDAP and Active Directory. You may use similar methods to setup other kinds of authentications, such as Kerberos.

Percona Monitoring and Management (PMM) Graphs Explained: Custom MongoDB Graphs and Metrics

March 10, 2017 - 4:10pm

This blog post is another in the series on the Percona Server for MongoDB 3.4 bundle release. In this blog post, we will cover how to add custom MongoDB graphs to Percona Monitoring and Management (PMM) and (for the daring Golang developers out there) how to add custom metrics to PMM’s percona/mongodb_exporter metric exporter.

To get to adding new graphs and metrics, we first need to go over how PMM gets metrics from your database nodes and how they become graphs.

Percona Monitoring and Management (PMM)

Percona Monitoring and Management (PMM) is an open-source platform for managing and monitoring MySQL and MongoDB. It was developed by Percona on top of open-source technology. Behind the scenes, the graphing features this article covers use Prometheus (a popular time-series data store), Grafana (a popular visualisation tool), mongodb_exporter (our MongoDB database metric exporter) plus other technologies to provide database and operating system metric graphs for your database instances.

Prometheus

As mentioned, Percona Monitoring and Management uses Prometheus to gather and store database and operating system metrics. Prometheus works on an HTTP(s) pull-based architecture, where Prometheus “pulls” metrics from “exporters” on a schedule.

To provide a detailed view of your database hosts, you must enable two PMM monitoring services for MongoDB graphing capabilities:

  1. linux:metrics
  2. mongodb:metrics

See this link for more details on adding monitoring services to Percona Monitoring and Management: https://www.percona.com/doc/percona-monitoring-and-management/pmm-admin.html#adding-monitoring-services

It is important to note that not all metrics gathered by Percona Monitoring and Management are graphed. This is by design. Storing more metrics vs. what is graphed is very useful when more advanced insight is necessary. We also aim for PMM to be simple and straightforward to use, explaining why we don’t graph all of the nearly 1,000 metrics we collect per MongoDB node on each polling.

My personal monitoring philosophy is “monitor until it hurts and then take one step back.” In other words, try to get a much data as you can without impacting the database or adding monitoring resources/cost. Also, Prometheus stores large volumes of metrics efficiently due to compression and highly optimized data structures on disk. This offsets a lot of the cost of collecting extra metrics.

Later in this blog, I will show how to add a graph for an example metric that is currently gathered by PMM but is not graphed in PMM as of today. To see what metrics are available on PMM’s Prometheus instance, visit “http://<pmm-server>/promtheus/graph”.

prometheus/node_exporter (linux:metrics)

PMM’s OS-level metrics are provided to Prometheus via the 3rd-party exporter: prometheus/node_exporter.

This exporter provides 712 metrics per “pull” on my test CentOS 7 host that has:

  • 2 x CPUs
  • 3 x disks with 6 x LVMs
  • 2 x network interfaces

Note: more physical or virtual devices add to the number of node_exporter metrics.

The inner workings and addition of metrics to this exporter will not be covered in this blog post. Generally, the current metrics offered by node_exporter are more than enough.

Below is a full example of a single “pull” of metrics from node_exporter on my test host:

$ curl -sk https://192.168.99.10:42000/metrics|grep node # HELP node_boot_time Node boot time, in unixtime. # TYPE node_boot_time gauge node_boot_time 1.488904144e+09 # HELP node_context_switches Total number of context switches. # TYPE node_context_switches counter node_context_switches 1.0407839e+07 # HELP node_cpu Seconds the cpus spent in each mode. # TYPE node_cpu counter node_cpu{cpu="cpu0",mode="guest"} 0 node_cpu{cpu="cpu0",mode="idle"} 6437.71 node_cpu{cpu="cpu0",mode="iowait"} 24 node_cpu{cpu="cpu0",mode="irq"} 0 node_cpu{cpu="cpu0",mode="nice"} 0 node_cpu{cpu="cpu0",mode="softirq"} 1.38 node_cpu{cpu="cpu0",mode="steal"} 0 node_cpu{cpu="cpu0",mode="system"} 117.65 node_cpu{cpu="cpu0",mode="user"} 33.6 node_cpu{cpu="cpu1",mode="guest"} 0 node_cpu{cpu="cpu1",mode="idle"} 6443.68 node_cpu{cpu="cpu1",mode="iowait"} 15.03 node_cpu{cpu="cpu1",mode="irq"} 0 node_cpu{cpu="cpu1",mode="nice"} 0 node_cpu{cpu="cpu1",mode="softirq"} 7.15 node_cpu{cpu="cpu1",mode="steal"} 0 node_cpu{cpu="cpu1",mode="system"} 118.08 node_cpu{cpu="cpu1",mode="user"} 28.84 # HELP node_disk_bytes_read The total number of bytes read successfully. # TYPE node_disk_bytes_read counter node_disk_bytes_read{device="dm-0"} 2.09581056e+08 node_disk_bytes_read{device="dm-1"} 1.114112e+06 node_disk_bytes_read{device="dm-2"} 5.500416e+06 node_disk_bytes_read{device="dm-3"} 2.98752e+06 node_disk_bytes_read{device="dm-4"} 3.480576e+06 node_disk_bytes_read{device="dm-5"} 4.0983552e+07 node_disk_bytes_read{device="dm-6"} 303104 node_disk_bytes_read{device="sda"} 2.40276992e+08 node_disk_bytes_read{device="sdb"} 4.5549568e+07 node_disk_bytes_read{device="sdc"} 8.967168e+06 node_disk_bytes_read{device="sr0"} 49152 # HELP node_disk_bytes_written The total number of bytes written successfully. # TYPE node_disk_bytes_written counter node_disk_bytes_written{device="dm-0"} 4.1160192e+07 node_disk_bytes_written{device="dm-1"} 0 node_disk_bytes_written{device="dm-2"} 2.413568e+06 node_disk_bytes_written{device="dm-3"} 9.0589184e+07 node_disk_bytes_written{device="dm-4"} 4.631552e+06 node_disk_bytes_written{device="dm-5"} 2.140672e+06 node_disk_bytes_written{device="dm-6"} 0 node_disk_bytes_written{device="sda"} 4.3257344e+07 node_disk_bytes_written{device="sdb"} 6.772224e+06 node_disk_bytes_written{device="sdc"} 9.3002752e+07 node_disk_bytes_written{device="sr0"} 0 # HELP node_disk_io_now The number of I/Os currently in progress. # TYPE node_disk_io_now gauge node_disk_io_now{device="dm-0"} 0 node_disk_io_now{device="dm-1"} 0 node_disk_io_now{device="dm-2"} 0 node_disk_io_now{device="dm-3"} 0 node_disk_io_now{device="dm-4"} 0 node_disk_io_now{device="dm-5"} 0 node_disk_io_now{device="dm-6"} 0 node_disk_io_now{device="sda"} 0 node_disk_io_now{device="sdb"} 0 node_disk_io_now{device="sdc"} 0 node_disk_io_now{device="sr0"} 0 # HELP node_disk_io_time_ms Milliseconds spent doing I/Os. # TYPE node_disk_io_time_ms counter node_disk_io_time_ms{device="dm-0"} 6443 node_disk_io_time_ms{device="dm-1"} 41 node_disk_io_time_ms{device="dm-2"} 319 node_disk_io_time_ms{device="dm-3"} 61024 node_disk_io_time_ms{device="dm-4"} 1159 node_disk_io_time_ms{device="dm-5"} 772 node_disk_io_time_ms{device="dm-6"} 1 node_disk_io_time_ms{device="sda"} 6965 node_disk_io_time_ms{device="sdb"} 2004 node_disk_io_time_ms{device="sdc"} 60718 node_disk_io_time_ms{device="sr0"} 5 # HELP node_disk_io_time_weighted The weighted # of milliseconds spent doing I/Os. See https://www.kernel.org/doc/Documentation/iostats.txt. # TYPE node_disk_io_time_weighted counter node_disk_io_time_weighted{device="dm-0"} 9972 node_disk_io_time_weighted{device="dm-1"} 46 node_disk_io_time_weighted{device="dm-2"} 369 node_disk_io_time_weighted{device="dm-3"} 147704 node_disk_io_time_weighted{device="dm-4"} 1618 node_disk_io_time_weighted{device="dm-5"} 968 node_disk_io_time_weighted{device="dm-6"} 1 node_disk_io_time_weighted{device="sda"} 10365 node_disk_io_time_weighted{device="sdb"} 3213 node_disk_io_time_weighted{device="sdc"} 143822 node_disk_io_time_weighted{device="sr0"} 5 # HELP node_disk_read_time_ms The total number of milliseconds spent by all reads. # TYPE node_disk_read_time_ms counter node_disk_read_time_ms{device="dm-0"} 7037 node_disk_read_time_ms{device="dm-1"} 46 node_disk_read_time_ms{device="dm-2"} 312 node_disk_read_time_ms{device="dm-3"} 56 node_disk_read_time_ms{device="dm-4"} 115 node_disk_read_time_ms{device="dm-5"} 906 node_disk_read_time_ms{device="dm-6"} 1 node_disk_read_time_ms{device="sda"} 7646 node_disk_read_time_ms{device="sdb"} 1192 node_disk_read_time_ms{device="sdc"} 412 node_disk_read_time_ms{device="sr0"} 5 # HELP node_disk_reads_completed The total number of reads completed successfully. # TYPE node_disk_reads_completed counter node_disk_reads_completed{device="dm-0"} 11521 node_disk_reads_completed{device="dm-1"} 133 node_disk_reads_completed{device="dm-2"} 1139 node_disk_reads_completed{device="dm-3"} 130 node_disk_reads_completed{device="dm-4"} 199 node_disk_reads_completed{device="dm-5"} 2077 node_disk_reads_completed{device="dm-6"} 42 node_disk_reads_completed{device="sda"} 13429 node_disk_reads_completed{device="sdb"} 2483 node_disk_reads_completed{device="sdc"} 1376 node_disk_reads_completed{device="sr0"} 12 # HELP node_disk_reads_merged The number of reads merged. See https://www.kernel.org/doc/Documentation/iostats.txt. # TYPE node_disk_reads_merged counter node_disk_reads_merged{device="dm-0"} 0 node_disk_reads_merged{device="dm-1"} 0 node_disk_reads_merged{device="dm-2"} 0 node_disk_reads_merged{device="dm-3"} 0 node_disk_reads_merged{device="dm-4"} 0 node_disk_reads_merged{device="dm-5"} 0 node_disk_reads_merged{device="dm-6"} 0 node_disk_reads_merged{device="sda"} 12 node_disk_reads_merged{device="sdb"} 0 node_disk_reads_merged{device="sdc"} 0 node_disk_reads_merged{device="sr0"} 0 # HELP node_disk_sectors_read The total number of sectors read successfully. # TYPE node_disk_sectors_read counter node_disk_sectors_read{device="dm-0"} 409338 node_disk_sectors_read{device="dm-1"} 2176 node_disk_sectors_read{device="dm-2"} 10743 node_disk_sectors_read{device="dm-3"} 5835 node_disk_sectors_read{device="dm-4"} 6798 node_disk_sectors_read{device="dm-5"} 80046 node_disk_sectors_read{device="dm-6"} 592 node_disk_sectors_read{device="sda"} 469291 node_disk_sectors_read{device="sdb"} 88964 node_disk_sectors_read{device="sdc"} 17514 node_disk_sectors_read{device="sr0"} 96 # HELP node_disk_sectors_written The total number of sectors written successfully. # TYPE node_disk_sectors_written counter node_disk_sectors_written{device="dm-0"} 80391 node_disk_sectors_written{device="dm-1"} 0 node_disk_sectors_written{device="dm-2"} 4714 node_disk_sectors_written{device="dm-3"} 176932 node_disk_sectors_written{device="dm-4"} 9046 node_disk_sectors_written{device="dm-5"} 4181 node_disk_sectors_written{device="dm-6"} 0 node_disk_sectors_written{device="sda"} 84487 node_disk_sectors_written{device="sdb"} 13227 node_disk_sectors_written{device="sdc"} 181646 node_disk_sectors_written{device="sr0"} 0 # HELP node_disk_write_time_ms This is the total number of milliseconds spent by all writes. # TYPE node_disk_write_time_ms counter node_disk_write_time_ms{device="dm-0"} 2935 node_disk_write_time_ms{device="dm-1"} 0 node_disk_write_time_ms{device="dm-2"} 57 node_disk_write_time_ms{device="dm-3"} 146868 node_disk_write_time_ms{device="dm-4"} 1503 node_disk_write_time_ms{device="dm-5"} 62 node_disk_write_time_ms{device="dm-6"} 0 node_disk_write_time_ms{device="sda"} 2797 node_disk_write_time_ms{device="sdb"} 2096 node_disk_write_time_ms{device="sdc"} 143569 node_disk_write_time_ms{device="sr0"} 0 # HELP node_disk_writes_completed The total number of writes completed successfully. # TYPE node_disk_writes_completed counter node_disk_writes_completed{device="dm-0"} 2701 node_disk_writes_completed{device="dm-1"} 0 node_disk_writes_completed{device="dm-2"} 70 node_disk_writes_completed{device="dm-3"} 415913 node_disk_writes_completed{device="dm-4"} 2737 node_disk_writes_completed{device="dm-5"} 17 node_disk_writes_completed{device="dm-6"} 0 node_disk_writes_completed{device="sda"} 3733 node_disk_writes_completed{device="sdb"} 3189 node_disk_writes_completed{device="sdc"} 421341 node_disk_writes_completed{device="sr0"} 0 # HELP node_disk_writes_merged The number of writes merged. See https://www.kernel.org/doc/Documentation/iostats.txt. # TYPE node_disk_writes_merged counter node_disk_writes_merged{device="dm-0"} 0 node_disk_writes_merged{device="dm-1"} 0 node_disk_writes_merged{device="dm-2"} 0 node_disk_writes_merged{device="dm-3"} 0 node_disk_writes_merged{device="dm-4"} 0 node_disk_writes_merged{device="dm-5"} 0 node_disk_writes_merged{device="dm-6"} 0 node_disk_writes_merged{device="sda"} 147 node_disk_writes_merged{device="sdb"} 19 node_disk_writes_merged{device="sdc"} 2182 node_disk_writes_merged{device="sr0"} 0 # HELP node_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which node_exporter was built. # TYPE node_exporter_build_info gauge node_exporter_build_info{branch="master",goversion="go1.7.4",revision="2d78e22000779d63c714011e4fb30c65623b9c77",version="1.1.1"} 1 # HELP node_exporter_scrape_duration_seconds node_exporter: Duration of a scrape job. # TYPE node_exporter_scrape_duration_seconds summary node_exporter_scrape_duration_seconds{collector="diskstats",result="success",quantile="0.5"} 0.00018768700000000002 node_exporter_scrape_duration_seconds{collector="diskstats",result="success",quantile="0.9"} 0.000199063 node_exporter_scrape_duration_seconds{collector="diskstats",result="success",quantile="0.99"} 0.000199063 node_exporter_scrape_duration_seconds_sum{collector="diskstats",result="success"} 0.0008093360000000001 node_exporter_scrape_duration_seconds_count{collector="diskstats",result="success"} 3 node_exporter_scrape_duration_seconds{collector="filefd",result="success",quantile="0.5"} 2.6501000000000003e-05 node_exporter_scrape_duration_seconds{collector="filefd",result="success",quantile="0.9"} 0.000149764 node_exporter_scrape_duration_seconds{collector="filefd",result="success",quantile="0.99"} 0.000149764 node_exporter_scrape_duration_seconds_sum{collector="filefd",result="success"} 0.0004463360000000001 node_exporter_scrape_duration_seconds_count{collector="filefd",result="success"} 3 node_exporter_scrape_duration_seconds{collector="filesystem",result="success",quantile="0.5"} 0.000350556 node_exporter_scrape_duration_seconds{collector="filesystem",result="success",quantile="0.9"} 0.000661542 node_exporter_scrape_duration_seconds{collector="filesystem",result="success",quantile="0.99"} 0.000661542 node_exporter_scrape_duration_seconds_sum{collector="filesystem",result="success"} 0.006669826000000001 node_exporter_scrape_duration_seconds_count{collector="filesystem",result="success"} 3 node_exporter_scrape_duration_seconds{collector="loadavg",result="success",quantile="0.5"} 4.1976e-05 node_exporter_scrape_duration_seconds{collector="loadavg",result="success",quantile="0.9"} 4.2433000000000005e-05 node_exporter_scrape_duration_seconds{collector="loadavg",result="success",quantile="0.99"} 4.2433000000000005e-05 node_exporter_scrape_duration_seconds_sum{collector="loadavg",result="success"} 0.0006603080000000001 node_exporter_scrape_duration_seconds_count{collector="loadavg",result="success"} 3 node_exporter_scrape_duration_seconds{collector="meminfo",result="success",quantile="0.5"} 0.000244919 node_exporter_scrape_duration_seconds{collector="meminfo",result="success",quantile="0.9"} 0.00038740000000000004 node_exporter_scrape_duration_seconds{collector="meminfo",result="success",quantile="0.99"} 0.00038740000000000004 node_exporter_scrape_duration_seconds_sum{collector="meminfo",result="success"} 0.003157154 node_exporter_scrape_duration_seconds_count{collector="meminfo",result="success"} 3 node_exporter_scrape_duration_seconds{collector="netdev",result="success",quantile="0.5"} 0.000152886 node_exporter_scrape_duration_seconds{collector="netdev",result="success",quantile="0.9"} 0.00048569400000000006 node_exporter_scrape_duration_seconds{collector="netdev",result="success",quantile="0.99"} 0.00048569400000000006 node_exporter_scrape_duration_seconds_sum{collector="netdev",result="success"} 0.0033770220000000004 node_exporter_scrape_duration_seconds_count{collector="netdev",result="success"} 3 node_exporter_scrape_duration_seconds{collector="netstat",result="success",quantile="0.5"} 0.0007874710000000001 node_exporter_scrape_duration_seconds{collector="netstat",result="success",quantile="0.9"} 0.001612906 node_exporter_scrape_duration_seconds{collector="netstat",result="success",quantile="0.99"} 0.001612906 node_exporter_scrape_duration_seconds_sum{collector="netstat",result="success"} 0.0055138800000000005 node_exporter_scrape_duration_seconds_count{collector="netstat",result="success"} 3 node_exporter_scrape_duration_seconds{collector="stat",result="success",quantile="0.5"} 9.3368e-05 node_exporter_scrape_duration_seconds{collector="stat",result="success",quantile="0.9"} 0.00014552100000000002 node_exporter_scrape_duration_seconds{collector="stat",result="success",quantile="0.99"} 0.00014552100000000002 node_exporter_scrape_duration_seconds_sum{collector="stat",result="success"} 0.00039008900000000004 node_exporter_scrape_duration_seconds_count{collector="stat",result="success"} 3 node_exporter_scrape_duration_seconds{collector="time",result="success",quantile="0.5"} 1.0485e-05 node_exporter_scrape_duration_seconds{collector="time",result="success",quantile="0.9"} 2.462e-05 node_exporter_scrape_duration_seconds{collector="time",result="success",quantile="0.99"} 2.462e-05 node_exporter_scrape_duration_seconds_sum{collector="time",result="success"} 6.4423e-05 node_exporter_scrape_duration_seconds_count{collector="time",result="success"} 3 node_exporter_scrape_duration_seconds{collector="uname",result="success",quantile="0.5"} 1.811e-05 node_exporter_scrape_duration_seconds{collector="uname",result="success",quantile="0.9"} 6.731300000000001e-05 node_exporter_scrape_duration_seconds{collector="uname",result="success",quantile="0.99"} 6.731300000000001e-05 node_exporter_scrape_duration_seconds_sum{collector="uname",result="success"} 0.00030852200000000004 node_exporter_scrape_duration_seconds_count{collector="uname",result="success"} 3 node_exporter_scrape_duration_seconds{collector="vmstat",result="success",quantile="0.5"} 0.00035561100000000003 node_exporter_scrape_duration_seconds{collector="vmstat",result="success",quantile="0.9"} 0.00046660900000000004 node_exporter_scrape_duration_seconds{collector="vmstat",result="success",quantile="0.99"} 0.00046660900000000004 node_exporter_scrape_duration_seconds_sum{collector="vmstat",result="success"} 0.002186003 node_exporter_scrape_duration_seconds_count{collector="vmstat",result="success"} 3 # HELP node_filefd_allocated File descriptor statistics: allocated. # TYPE node_filefd_allocated gauge node_filefd_allocated 1696 # HELP node_filefd_maximum File descriptor statistics: maximum. # TYPE node_filefd_maximum gauge node_filefd_maximum 382428 # HELP node_filesystem_avail Filesystem space available to non-root users in bytes. # TYPE node_filesystem_avail gauge node_filesystem_avail{device="/dev/mapper/centos-root",fstype="xfs",mountpoint="/"} 2.977058816e+09 node_filesystem_avail{device="/dev/mapper/data-home",fstype="xfs",mountpoint="/home"} 2.3885000704e+10 node_filesystem_avail{device="/dev/mapper/data-opt",fstype="xfs",mountpoint="/opt"} 6.520070144e+09 node_filesystem_avail{device="/dev/mapper/tmpdata-docker",fstype="xfs",mountpoint="/var/lib/docker"} 8.374427648e+09 node_filesystem_avail{device="/dev/mapper/tmpdata-mongo_data",fstype="xfs",mountpoint="/home/tim/mongo/data"} 6.386853888e+10 node_filesystem_avail{device="/dev/sda1",fstype="xfs",mountpoint="/boot"} 1.37248768e+08 node_filesystem_avail{device="rootfs",fstype="rootfs",mountpoint="/"} 2.977058816e+09 node_filesystem_avail{device="tmpfs",fstype="tmpfs",mountpoint="/run"} 1.978773504e+09 node_filesystem_avail{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/0"} 3.975168e+08 node_filesystem_avail{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/1000"} 3.975168e+08 # HELP node_filesystem_files Filesystem total file nodes. # TYPE node_filesystem_files gauge node_filesystem_files{device="/dev/mapper/centos-root",fstype="xfs",mountpoint="/"} 6.991872e+06 node_filesystem_files{device="/dev/mapper/data-home",fstype="xfs",mountpoint="/home"} 2.9360128e+07 node_filesystem_files{device="/dev/mapper/data-opt",fstype="xfs",mountpoint="/opt"} 8.388608e+06 node_filesystem_files{device="/dev/mapper/tmpdata-docker",fstype="xfs",mountpoint="/var/lib/docker"} 4.194304e+06 node_filesystem_files{device="/dev/mapper/tmpdata-mongo_data",fstype="xfs",mountpoint="/home/tim/mongo/data"} 3.145728e+07 node_filesystem_files{device="/dev/sda1",fstype="xfs",mountpoint="/boot"} 512000 node_filesystem_files{device="rootfs",fstype="rootfs",mountpoint="/"} 6.991872e+06 node_filesystem_files{device="tmpfs",fstype="tmpfs",mountpoint="/run"} 485249 node_filesystem_files{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/0"} 485249 node_filesystem_files{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/1000"} 485249 # HELP node_filesystem_files_free Filesystem total free file nodes. # TYPE node_filesystem_files_free gauge node_filesystem_files_free{device="/dev/mapper/centos-root",fstype="xfs",mountpoint="/"} 6.839777e+06 node_filesystem_files_free{device="/dev/mapper/data-home",fstype="xfs",mountpoint="/home"} 2.9273287e+07 node_filesystem_files_free{device="/dev/mapper/data-opt",fstype="xfs",mountpoint="/opt"} 8.376436e+06 node_filesystem_files_free{device="/dev/mapper/tmpdata-docker",fstype="xfs",mountpoint="/var/lib/docker"} 4.194188e+06 node_filesystem_files_free{device="/dev/mapper/tmpdata-mongo_data",fstype="xfs",mountpoint="/home/tim/mongo/data"} 3.1457181e+07 node_filesystem_files_free{device="/dev/sda1",fstype="xfs",mountpoint="/boot"} 511638 node_filesystem_files_free{device="rootfs",fstype="rootfs",mountpoint="/"} 6.839777e+06 node_filesystem_files_free{device="tmpfs",fstype="tmpfs",mountpoint="/run"} 484725 node_filesystem_files_free{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/0"} 485248 node_filesystem_files_free{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/1000"} 485248 # HELP node_filesystem_free Filesystem free space in bytes. # TYPE node_filesystem_free gauge node_filesystem_free{device="/dev/mapper/centos-root",fstype="xfs",mountpoint="/"} 2.977058816e+09 node_filesystem_free{device="/dev/mapper/data-home",fstype="xfs",mountpoint="/home"} 2.3885000704e+10 node_filesystem_free{device="/dev/mapper/data-opt",fstype="xfs",mountpoint="/opt"} 6.520070144e+09 node_filesystem_free{device="/dev/mapper/tmpdata-docker",fstype="xfs",mountpoint="/var/lib/docker"} 8.374427648e+09 node_filesystem_free{device="/dev/mapper/tmpdata-mongo_data",fstype="xfs",mountpoint="/home/tim/mongo/data"} 6.386853888e+10 node_filesystem_free{device="/dev/sda1",fstype="xfs",mountpoint="/boot"} 1.37248768e+08 node_filesystem_free{device="rootfs",fstype="rootfs",mountpoint="/"} 2.977058816e+09 node_filesystem_free{device="tmpfs",fstype="tmpfs",mountpoint="/run"} 1.978773504e+09 node_filesystem_free{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/0"} 3.975168e+08 node_filesystem_free{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/1000"} 3.975168e+08 # HELP node_filesystem_readonly Filesystem read-only status. # TYPE node_filesystem_readonly gauge node_filesystem_readonly{device="/dev/mapper/centos-root",fstype="xfs",mountpoint="/"} 0 node_filesystem_readonly{device="/dev/mapper/data-home",fstype="xfs",mountpoint="/home"} 0 node_filesystem_readonly{device="/dev/mapper/data-opt",fstype="xfs",mountpoint="/opt"} 0 node_filesystem_readonly{device="/dev/mapper/tmpdata-docker",fstype="xfs",mountpoint="/var/lib/docker"} 0 node_filesystem_readonly{device="/dev/mapper/tmpdata-mongo_data",fstype="xfs",mountpoint="/home/tim/mongo/data"} 0 node_filesystem_readonly{device="/dev/sda1",fstype="xfs",mountpoint="/boot"} 0 node_filesystem_readonly{device="rootfs",fstype="rootfs",mountpoint="/"} 0 node_filesystem_readonly{device="tmpfs",fstype="tmpfs",mountpoint="/run"} 0 node_filesystem_readonly{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/0"} 0 node_filesystem_readonly{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/1000"} 0 # HELP node_filesystem_size Filesystem size in bytes. # TYPE node_filesystem_size gauge node_filesystem_size{device="/dev/mapper/centos-root",fstype="xfs",mountpoint="/"} 7.149191168e+09 node_filesystem_size{device="/dev/mapper/data-home",fstype="xfs",mountpoint="/home"} 3.0054285312e+10 node_filesystem_size{device="/dev/mapper/data-opt",fstype="xfs",mountpoint="/opt"} 8.579448832e+09 node_filesystem_size{device="/dev/mapper/tmpdata-docker",fstype="xfs",mountpoint="/var/lib/docker"} 8.579448832e+09 node_filesystem_size{device="/dev/mapper/tmpdata-mongo_data",fstype="xfs",mountpoint="/home/tim/mongo/data"} 6.439305216e+10 node_filesystem_size{device="/dev/sda1",fstype="xfs",mountpoint="/boot"} 5.20794112e+08 node_filesystem_size{device="rootfs",fstype="rootfs",mountpoint="/"} 7.149191168e+09 node_filesystem_size{device="tmpfs",fstype="tmpfs",mountpoint="/run"} 1.987579904e+09 node_filesystem_size{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/0"} 3.975168e+08 node_filesystem_size{device="tmpfs",fstype="tmpfs",mountpoint="/run/user/1000"} 3.975168e+08 # HELP node_forks Total number of forks. # TYPE node_forks counter node_forks 3928 # HELP node_intr Total number of interrupts serviced. # TYPE node_intr counter node_intr 4.629898e+06 # HELP node_load1 1m load average. # TYPE node_load1 gauge node_load1 0.2 # HELP node_load15 15m load average. # TYPE node_load15 gauge node_load15 0.21 # HELP node_load5 5m load average. # TYPE node_load5 gauge node_load5 0.22 # HELP node_memory_Active Memory information field Active. # TYPE node_memory_Active gauge node_memory_Active 4.80772096e+08 # HELP node_memory_Active_anon Memory information field Active_anon. # TYPE node_memory_Active_anon gauge node_memory_Active_anon 3.62610688e+08 # HELP node_memory_Active_file Memory information field Active_file. # TYPE node_memory_Active_file gauge node_memory_Active_file 1.18161408e+08 # HELP node_memory_AnonHugePages Memory information field AnonHugePages. # TYPE node_memory_AnonHugePages gauge node_memory_AnonHugePages 1.2582912e+07 # HELP node_memory_AnonPages Memory information field AnonPages. # TYPE node_memory_AnonPages gauge node_memory_AnonPages 3.62438656e+08 # HELP node_memory_Bounce Memory information field Bounce. # TYPE node_memory_Bounce gauge node_memory_Bounce 0 # HELP node_memory_Buffers Memory information field Buffers. # TYPE node_memory_Buffers gauge node_memory_Buffers 1.90464e+06 # HELP node_memory_Cached Memory information field Cached. # TYPE node_memory_Cached gauge node_memory_Cached 3.12205312e+08 # HELP node_memory_CommitLimit Memory information field CommitLimit. # TYPE node_memory_CommitLimit gauge node_memory_CommitLimit 2.847408128e+09 # HELP node_memory_Committed_AS Memory information field Committed_AS. # TYPE node_memory_Committed_AS gauge node_memory_Committed_AS 5.08229632e+09 # HELP node_memory_DirectMap2M Memory information field DirectMap2M. # TYPE node_memory_DirectMap2M gauge node_memory_DirectMap2M 4.22576128e+09 # HELP node_memory_DirectMap4k Memory information field DirectMap4k. # TYPE node_memory_DirectMap4k gauge node_memory_DirectMap4k 6.914048e+07 # HELP node_memory_Dirty Memory information field Dirty. # TYPE node_memory_Dirty gauge node_memory_Dirty 69632 # HELP node_memory_HardwareCorrupted Memory information field HardwareCorrupted. # TYPE node_memory_HardwareCorrupted gauge node_memory_HardwareCorrupted 0 # HELP node_memory_HugePages_Free Memory information field HugePages_Free. # TYPE node_memory_HugePages_Free gauge node_memory_HugePages_Free 0 # HELP node_memory_HugePages_Rsvd Memory information field HugePages_Rsvd. # TYPE node_memory_HugePages_Rsvd gauge node_memory_HugePages_Rsvd 0 # HELP node_memory_HugePages_Surp Memory information field HugePages_Surp. # TYPE node_memory_HugePages_Surp gauge node_memory_HugePages_Surp 0 # HELP node_memory_HugePages_Total Memory information field HugePages_Total. # TYPE node_memory_HugePages_Total gauge node_memory_HugePages_Total 0 # HELP node_memory_Hugepagesize Memory information field Hugepagesize. # TYPE node_memory_Hugepagesize gauge node_memory_Hugepagesize 2.097152e+06 # HELP node_memory_Inactive Memory information field Inactive. # TYPE node_memory_Inactive gauge node_memory_Inactive 1.95702784e+08 # HELP node_memory_Inactive_anon Memory information field Inactive_anon. # TYPE node_memory_Inactive_anon gauge node_memory_Inactive_anon 8.564736e+06 # HELP node_memory_Inactive_file Memory information field Inactive_file. # TYPE node_memory_Inactive_file gauge node_memory_Inactive_file 1.87138048e+08 # HELP node_memory_KernelStack Memory information field KernelStack. # TYPE node_memory_KernelStack gauge node_memory_KernelStack 1.1583488e+07 # HELP node_memory_Mapped Memory information field Mapped. # TYPE node_memory_Mapped gauge node_memory_Mapped 6.7530752e+07 # HELP node_memory_MemAvailable Memory information field MemAvailable. # TYPE node_memory_MemAvailable gauge node_memory_MemAvailable 3.273515008e+09 # HELP node_memory_MemFree Memory information field MemFree. # TYPE node_memory_MemFree gauge node_memory_MemFree 3.15346944e+09 # HELP node_memory_MemTotal Memory information field MemTotal. # TYPE node_memory_MemTotal gauge node_memory_MemTotal 3.975163904e+09 # HELP node_memory_Mlocked Memory information field Mlocked. # TYPE node_memory_Mlocked gauge node_memory_Mlocked 0 # HELP node_memory_NFS_Unstable Memory information field NFS_Unstable. # TYPE node_memory_NFS_Unstable gauge node_memory_NFS_Unstable 0 # HELP node_memory_PageTables Memory information field PageTables. # TYPE node_memory_PageTables gauge node_memory_PageTables 9.105408e+06 # HELP node_memory_SReclaimable Memory information field SReclaimable. # TYPE node_memory_SReclaimable gauge node_memory_SReclaimable 4.4638208e+07 # HELP node_memory_SUnreclaim Memory information field SUnreclaim. # TYPE node_memory_SUnreclaim gauge node_memory_SUnreclaim 3.2059392e+07 # HELP node_memory_Shmem Memory information field Shmem. # TYPE node_memory_Shmem gauge node_memory_Shmem 8.810496e+06 # HELP node_memory_Slab Memory information field Slab. # TYPE node_memory_Slab gauge node_memory_Slab 7.66976e+07 # HELP node_memory_SwapCached Memory information field SwapCached. # TYPE node_memory_SwapCached gauge node_memory_SwapCached 0 # HELP node_memory_SwapFree Memory information field SwapFree. # TYPE node_memory_SwapFree gauge node_memory_SwapFree 8.59828224e+08 # HELP node_memory_SwapTotal Memory information field SwapTotal. # TYPE node_memory_SwapTotal gauge node_memory_SwapTotal 8.59828224e+08 # HELP node_memory_Unevictable Memory information field Unevictable. # TYPE node_memory_Unevictable gauge node_memory_Unevictable 0 # HELP node_memory_VmallocChunk Memory information field VmallocChunk. # TYPE node_memory_VmallocChunk gauge node_memory_VmallocChunk 3.5184346918912e+13 # HELP node_memory_VmallocTotal Memory information field VmallocTotal. # TYPE node_memory_VmallocTotal gauge node_memory_VmallocTotal 3.5184372087808e+13 # HELP node_memory_VmallocUsed Memory information field VmallocUsed. # TYPE node_memory_VmallocUsed gauge node_memory_VmallocUsed 1.6748544e+07 # HELP node_memory_Writeback Memory information field Writeback. # TYPE node_memory_Writeback gauge node_memory_Writeback 0 # HELP node_memory_WritebackTmp Memory information field WritebackTmp. # TYPE node_memory_WritebackTmp gauge node_memory_WritebackTmp 0 # HELP node_netstat_IcmpMsg_InType3 Protocol IcmpMsg statistic InType3. # TYPE node_netstat_IcmpMsg_InType3 untyped node_netstat_IcmpMsg_InType3 32 # HELP node_netstat_IcmpMsg_OutType3 Protocol IcmpMsg statistic OutType3. # TYPE node_netstat_IcmpMsg_OutType3 untyped node_netstat_IcmpMsg_OutType3 32 # HELP node_netstat_Icmp_InAddrMaskReps Protocol Icmp statistic InAddrMaskReps. # TYPE node_netstat_Icmp_InAddrMaskReps untyped node_netstat_Icmp_InAddrMaskReps 0 # HELP node_netstat_Icmp_InAddrMasks Protocol Icmp statistic InAddrMasks. # TYPE node_netstat_Icmp_InAddrMasks untyped node_netstat_Icmp_InAddrMasks 0 # HELP node_netstat_Icmp_InCsumErrors Protocol Icmp statistic InCsumErrors. # TYPE node_netstat_Icmp_InCsumErrors untyped node_netstat_Icmp_InCsumErrors 0 # HELP node_netstat_Icmp_InDestUnreachs Protocol Icmp statistic InDestUnreachs. # TYPE node_netstat_Icmp_InDestUnreachs untyped node_netstat_Icmp_InDestUnreachs 32 # HELP node_netstat_Icmp_InEchoReps Protocol Icmp statistic InEchoReps. # TYPE node_netstat_Icmp_InEchoReps untyped node_netstat_Icmp_InEchoReps 0 # HELP node_netstat_Icmp_InEchos Protocol Icmp statistic InEchos. # TYPE node_netstat_Icmp_InEchos untyped node_netstat_Icmp_InEchos 0 # HELP node_netstat_Icmp_InErrors Protocol Icmp statistic InErrors. # TYPE node_netstat_Icmp_InErrors untyped node_netstat_Icmp_InErrors 0 # HELP node_netstat_Icmp_InMsgs Protocol Icmp statistic InMsgs. # TYPE node_netstat_Icmp_InMsgs untyped node_netstat_Icmp_InMsgs 32 # HELP node_netstat_Icmp_InParmProbs Protocol Icmp statistic InParmProbs. # TYPE node_netstat_Icmp_InParmProbs untyped node_netstat_Icmp_InParmProbs 0 # HELP node_netstat_Icmp_InRedirects Protocol Icmp statistic InRedirects. # TYPE node_netstat_Icmp_InRedirects untyped node_netstat_Icmp_InRedirects 0 # HELP node_netstat_Icmp_InSrcQuenchs Protocol Icmp statistic InSrcQuenchs. # TYPE node_netstat_Icmp_InSrcQuenchs untyped node_netstat_Icmp_InSrcQuenchs 0 # HELP node_netstat_Icmp_InTimeExcds Protocol Icmp statistic InTimeExcds. # TYPE node_netstat_Icmp_InTimeExcds untyped node_netstat_Icmp_InTimeExcds 0 # HELP node_netstat_Icmp_InTimestampReps Protocol Icmp statistic InTimestampReps. # TYPE node_netstat_Icmp_InTimestampReps untyped node_netstat_Icmp_InTimestampReps 0 # HELP node_netstat_Icmp_InTimestamps Protocol Icmp statistic InTimestamps. # TYPE node_netstat_Icmp_InTimestamps untyped node_netstat_Icmp_InTimestamps 0 # HELP node_netstat_Icmp_OutAddrMaskReps Protocol Icmp statistic OutAddrMaskReps. # TYPE node_netstat_Icmp_OutAddrMaskReps untyped node_netstat_Icmp_OutAddrMaskReps 0 # HELP node_netstat_Icmp_OutAddrMasks Protocol Icmp statistic OutAddrMasks. # TYPE node_netstat_Icmp_OutAddrMasks untyped node_netstat_Icmp_OutAddrMasks 0 # HELP node_netstat_Icmp_OutDestUnreachs Protocol Icmp statistic OutDestUnreachs. # TYPE node_netstat_Icmp_OutDestUnreachs untyped node_netstat_Icmp_OutDestUnreachs 32 # HELP node_netstat_Icmp_OutEchoReps Protocol Icmp statistic OutEchoReps. # TYPE node_netstat_Icmp_OutEchoReps untyped node_netstat_Icmp_OutEchoReps 0 # HELP node_netstat_Icmp_OutEchos Protocol Icmp statistic OutEchos. # TYPE node_netstat_Icmp_OutEchos untyped node_netstat_Icmp_OutEchos 0 # HELP node_netstat_Icmp_OutErrors Protocol Icmp statistic OutErrors. # TYPE node_netstat_Icmp_OutErrors untyped node_netstat_Icmp_OutErrors 0 # HELP node_netstat_Icmp_OutMsgs Protocol Icmp statistic OutMsgs. # TYPE node_netstat_Icmp_OutMsgs untyped node_netstat_Icmp_OutMsgs 32 # HELP node_netstat_Icmp_OutParmProbs Protocol Icmp statistic OutParmProbs. # TYPE node_netstat_Icmp_OutParmProbs untyped node_netstat_Icmp_OutParmProbs 0 # HELP node_netstat_Icmp_OutRedirects Protocol Icmp statistic OutRedirects. # TYPE node_netstat_Icmp_OutRedirects untyped node_netstat_Icmp_OutRedirects 0 # HELP node_netstat_Icmp_OutSrcQuenchs Protocol Icmp statistic OutSrcQuenchs. # TYPE node_netstat_Icmp_OutSrcQuenchs untyped node_netstat_Icmp_OutSrcQuenchs 0 # HELP node_netstat_Icmp_OutTimeExcds Protocol Icmp statistic OutTimeExcds. # TYPE node_netstat_Icmp_OutTimeExcds untyped node_netstat_Icmp_OutTimeExcds 0 # HELP node_netstat_Icmp_OutTimestampReps Protocol Icmp statistic OutTimestampReps. # TYPE node_netstat_Icmp_OutTimestampReps untyped node_netstat_Icmp_OutTimestampReps 0 # HELP node_netstat_Icmp_OutTimestamps Protocol Icmp statistic OutTimestamps. # TYPE node_netstat_Icmp_OutTimestamps untyped node_netstat_Icmp_OutTimestamps 0 # HELP node_netstat_IpExt_InBcastOctets Protocol IpExt statistic InBcastOctets. # TYPE node_netstat_IpExt_InBcastOctets untyped node_netstat_IpExt_InBcastOctets 16200 # HELP node_netstat_IpExt_InBcastPkts Protocol IpExt statistic InBcastPkts. # TYPE node_netstat_IpExt_InBcastPkts untyped node_netstat_IpExt_InBcastPkts 225 # HELP node_netstat_IpExt_InCEPkts Protocol IpExt statistic InCEPkts. # TYPE node_netstat_IpExt_InCEPkts untyped node_netstat_IpExt_InCEPkts 0 # HELP node_netstat_IpExt_InCsumErrors Protocol IpExt statistic InCsumErrors. # TYPE node_netstat_IpExt_InCsumErrors untyped node_netstat_IpExt_InCsumErrors 0 # HELP node_netstat_IpExt_InECT0Pkts Protocol IpExt statistic InECT0Pkts. # TYPE node_netstat_IpExt_InECT0Pkts untyped node_netstat_IpExt_InECT0Pkts 0 # HELP node_netstat_IpExt_InECT1Pkts Protocol IpExt statistic InECT1Pkts. # TYPE node_netstat_IpExt_InECT1Pkts untyped node_netstat_IpExt_InECT1Pkts 0 # HELP node_netstat_IpExt_InMcastOctets Protocol IpExt statistic InMcastOctets. # TYPE node_netstat_IpExt_InMcastOctets untyped node_netstat_IpExt_InMcastOctets 0 # HELP node_netstat_IpExt_InMcastPkts Protocol IpExt statistic InMcastPkts. # TYPE node_netstat_IpExt_InMcastPkts untyped node_netstat_IpExt_InMcastPkts 0 # HELP node_netstat_IpExt_InNoECTPkts Protocol IpExt statistic InNoECTPkts. # TYPE node_netstat_IpExt_InNoECTPkts untyped node_netstat_IpExt_InNoECTPkts 139536 # HELP node_netstat_IpExt_InNoRoutes Protocol IpExt statistic InNoRoutes. # TYPE node_netstat_IpExt_InNoRoutes untyped node_netstat_IpExt_InNoRoutes 1 # HELP node_netstat_IpExt_InOctets Protocol IpExt statistic InOctets. # TYPE node_netstat_IpExt_InOctets untyped node_netstat_IpExt_InOctets 3.4550209e+07 # HELP node_netstat_IpExt_InTruncatedPkts Protocol IpExt statistic InTruncatedPkts. # TYPE node_netstat_IpExt_InTruncatedPkts untyped node_netstat_IpExt_InTruncatedPkts 0 # HELP node_netstat_IpExt_OutBcastOctets Protocol IpExt statistic OutBcastOctets. # TYPE node_netstat_IpExt_OutBcastOctets untyped node_netstat_IpExt_OutBcastOctets 0 # HELP node_netstat_IpExt_OutBcastPkts Protocol IpExt statistic OutBcastPkts. # TYPE node_netstat_IpExt_OutBcastPkts untyped node_netstat_IpExt_OutBcastPkts 0 # HELP node_netstat_IpExt_OutMcastOctets Protocol IpExt statistic OutMcastOctets. # TYPE node_netstat_IpExt_OutMcastOctets untyped node_netstat_IpExt_OutMcastOctets 0 # HELP node_netstat_IpExt_OutMcastPkts Protocol IpExt statistic OutMcastPkts. # TYPE node_netstat_IpExt_OutMcastPkts untyped node_netstat_IpExt_OutMcastPkts 0 # HELP node_netstat_IpExt_OutOctets Protocol IpExt statistic OutOctets. # TYPE node_netstat_IpExt_OutOctets untyped node_netstat_IpExt_OutOctets 3.469735e+07 # HELP node_netstat_Ip_DefaultTTL Protocol Ip statistic DefaultTTL. # TYPE node_netstat_Ip_DefaultTTL untyped node_netstat_Ip_DefaultTTL 64 # HELP node_netstat_Ip_ForwDatagrams Protocol Ip statistic ForwDatagrams. # TYPE node_netstat_Ip_ForwDatagrams untyped node_netstat_Ip_ForwDatagrams 0 # HELP node_netstat_Ip_Forwarding Protocol Ip statistic Forwarding. # TYPE node_netstat_Ip_Forwarding untyped node_netstat_Ip_Forwarding 1 # HELP node_netstat_Ip_FragCreates Protocol Ip statistic FragCreates. # TYPE node_netstat_Ip_FragCreates untyped node_netstat_Ip_FragCreates 0 # HELP node_netstat_Ip_FragFails Protocol Ip statistic FragFails. # TYPE node_netstat_Ip_FragFails untyped node_netstat_Ip_FragFails 0 # HELP node_netstat_Ip_FragOKs Protocol Ip statistic FragOKs. # TYPE node_netstat_Ip_FragOKs untyped node_netstat_Ip_FragOKs 0 # HELP node_netstat_Ip_InAddrErrors Protocol Ip statistic InAddrErrors. # TYPE node_netstat_Ip_InAddrErrors untyped node_netstat_Ip_InAddrErrors 0 # HELP node_netstat_Ip_InDelivers Protocol Ip statistic InDelivers. # TYPE node_netstat_Ip_InDelivers untyped node_netstat_Ip_InDelivers 139535 # HELP node_netstat_Ip_InDiscards Protocol Ip statistic InDiscards. # TYPE node_netstat_Ip_InDiscards untyped node_netstat_Ip_InDiscards 0 # HELP node_netstat_Ip_InHdrErrors Protocol Ip statistic InHdrErrors. # TYPE node_netstat_Ip_InHdrErrors untyped node_netstat_Ip_InHdrErrors 0 # HELP node_netstat_Ip_InReceives Protocol Ip statistic InReceives. # TYPE node_netstat_Ip_InReceives untyped node_netstat_Ip_InReceives 139536 # HELP node_netstat_Ip_InUnknownProtos Protocol Ip statistic InUnknownProtos. # TYPE node_netstat_Ip_InUnknownProtos untyped node_netstat_Ip_InUnknownProtos 0 # HELP node_netstat_Ip_OutDiscards Protocol Ip statistic OutDiscards. # TYPE node_netstat_Ip_OutDiscards untyped node_netstat_Ip_OutDiscards 16 # HELP node_netstat_Ip_OutNoRoutes Protocol Ip statistic OutNoRoutes. # TYPE node_netstat_Ip_OutNoRoutes untyped node_netstat_Ip_OutNoRoutes 0 # HELP node_netstat_Ip_OutRequests Protocol Ip statistic OutRequests. # TYPE node_netstat_Ip_OutRequests untyped node_netstat_Ip_OutRequests 139089 # HELP node_netstat_Ip_ReasmFails Protocol Ip statistic ReasmFails. # TYPE node_netstat_Ip_ReasmFails untyped node_netstat_Ip_ReasmFails 0 # HELP node_netstat_Ip_ReasmOKs Protocol Ip statistic ReasmOKs. # TYPE node_netstat_Ip_ReasmOKs untyped node_netstat_Ip_ReasmOKs 0 # HELP node_netstat_Ip_ReasmReqds Protocol Ip statistic ReasmReqds. # TYPE node_netstat_Ip_ReasmReqds untyped node_netstat_Ip_ReasmReqds 0 # HELP node_netstat_Ip_ReasmTimeout Protocol Ip statistic ReasmTimeout. # TYPE node_netstat_Ip_ReasmTimeout untyped node_netstat_Ip_ReasmTimeout 0 # HELP node_netstat_TcpExt_ArpFilter Protocol TcpExt statistic ArpFilter. # TYPE node_netstat_TcpExt_ArpFilter untyped node_netstat_TcpExt_ArpFilter 0 # HELP node_netstat_TcpExt_BusyPollRxPackets Protocol TcpExt statistic BusyPollRxPackets. # TYPE node_netstat_TcpExt_BusyPollRxPackets untyped node_netstat_TcpExt_BusyPollRxPackets 0 # HELP node_netstat_TcpExt_DelayedACKLocked Protocol TcpExt statistic DelayedACKLocked. # TYPE node_netstat_TcpExt_DelayedACKLocked untyped node_netstat_TcpExt_DelayedACKLocked 0 # HELP node_netstat_TcpExt_DelayedACKLost Protocol TcpExt statistic DelayedACKLost. # TYPE node_netstat_TcpExt_DelayedACKLost untyped node_netstat_TcpExt_DelayedACKLost 0 # HELP node_netstat_TcpExt_DelayedACKs Protocol TcpExt statistic DelayedACKs. # TYPE node_netstat_TcpExt_DelayedACKs untyped node_netstat_TcpExt_DelayedACKs 3374 # HELP node_netstat_TcpExt_EmbryonicRsts Protocol TcpExt statistic EmbryonicRsts. # TYPE node_netstat_TcpExt_EmbryonicRsts untyped node_netstat_TcpExt_EmbryonicRsts 0 # HELP node_netstat_TcpExt_IPReversePathFilter Protocol TcpExt statistic IPReversePathFilter. # TYPE node_netstat_TcpExt_IPReversePathFilter untyped node_netstat_TcpExt_IPReversePathFilter 0 # HELP node_netstat_TcpExt_ListenDrops Protocol TcpExt statistic ListenDrops. # TYPE node_netstat_TcpExt_ListenDrops untyped node_netstat_TcpExt_ListenDrops 0 # HELP node_netstat_TcpExt_ListenOverflows Protocol TcpExt statistic ListenOverflows. # TYPE node_netstat_TcpExt_ListenOverflows untyped node_netstat_TcpExt_ListenOverflows 0 # HELP node_netstat_TcpExt_LockDroppedIcmps Protocol TcpExt statistic LockDroppedIcmps. # TYPE node_netstat_TcpExt_LockDroppedIcmps untyped node_netstat_TcpExt_LockDroppedIcmps 0 # HELP node_netstat_TcpExt_OfoPruned Protocol TcpExt statistic OfoPruned. # TYPE node_netstat_TcpExt_OfoPruned untyped node_netstat_TcpExt_OfoPruned 0 # HELP node_netstat_TcpExt_OutOfWindowIcmps Protocol TcpExt statistic OutOfWindowIcmps. # TYPE node_netstat_TcpExt_OutOfWindowIcmps untyped node_netstat_TcpExt_OutOfWindowIcmps 0 # HELP node_netstat_TcpExt_PAWSActive Protocol TcpExt statistic PAWSActive. # TYPE node_netstat_TcpExt_PAWSActive untyped node_netstat_TcpExt_PAWSActive 0 # HELP node_netstat_TcpExt_PAWSEstab Protocol TcpExt statistic PAWSEstab. # TYPE node_netstat_TcpExt_PAWSEstab untyped node_netstat_TcpExt_PAWSEstab 0 # HELP node_netstat_TcpExt_PAWSPassive Protocol TcpExt statistic PAWSPassive. # TYPE node_netstat_TcpExt_PAWSPassive untyped node_netstat_TcpExt_PAWSPassive 0 # HELP node_netstat_TcpExt_PruneCalled Protocol TcpExt statistic PruneCalled. # TYPE node_netstat_TcpExt_PruneCalled untyped node_netstat_TcpExt_PruneCalled 0 # HELP node_netstat_TcpExt_RcvPruned Protocol TcpExt statistic RcvPruned. # TYPE node_netstat_TcpExt_RcvPruned untyped node_netstat_TcpExt_RcvPruned 0 # HELP node_netstat_TcpExt_SyncookiesFailed Protocol TcpExt statistic SyncookiesFailed. # TYPE node_netstat_TcpExt_SyncookiesFailed untyped node_netstat_TcpExt_SyncookiesFailed 0 # HELP node_netstat_TcpExt_SyncookiesRecv Protocol TcpExt statistic SyncookiesRecv. # TYPE node_netstat_TcpExt_SyncookiesRecv untyped node_netstat_TcpExt_SyncookiesRecv 0 # HELP node_netstat_TcpExt_SyncookiesSent Protocol TcpExt statistic SyncookiesSent. # TYPE node_netstat_TcpExt_SyncookiesSent untyped node_netstat_TcpExt_SyncookiesSent 0 # HELP node_netstat_TcpExt_TCPACKSkippedChallenge Protocol TcpExt statistic TCPACKSkippedChallenge. # TYPE node_netstat_TcpExt_TCPACKSkippedChallenge untyped node_netstat_TcpExt_TCPACKSkippedChallenge 0 # HELP node_netstat_TcpExt_TCPACKSkippedFinWait2 Protocol TcpExt statistic TCPACKSkippedFinWait2. # TYPE node_netstat_TcpExt_TCPACKSkippedFinWait2 untyped node_netstat_TcpExt_TCPACKSkippedFinWait2 0 # HELP node_netstat_TcpExt_TCPACKSkippedPAWS Protocol TcpExt statistic TCPACKSkippedPAWS. # TYPE node_netstat_TcpExt_TCPACKSkippedPAWS untyped node_netstat_TcpExt_TCPACKSkippedPAWS 0 # HELP node_netstat_TcpExt_TCPACKSkippedSeq Protocol TcpExt statistic TCPACKSkippedSeq. # TYPE node_netstat_TcpExt_TCPACKSkippedSeq untyped node_netstat_TcpExt_TCPACKSkippedSeq 0 # HELP node_netstat_TcpExt_TCPACKSkippedSynRecv Protocol TcpExt statistic TCPACKSkippedSynRecv. # TYPE node_netstat_TcpExt_TCPACKSkippedSynRecv untyped node_netstat_TcpExt_TCPACKSkippedSynRecv 0 # HELP node_netstat_TcpExt_TCPACKSkippedTimeWait Protocol TcpExt statistic TCPACKSkippedTimeWait. # TYPE node_netstat_TcpExt_TCPACKSkippedTimeWait untyped node_netstat_TcpExt_TCPACKSkippedTimeWait 0 # HELP node_netstat_TcpExt_TCPAbortFailed Protocol TcpExt statistic TCPAbortFailed. # TYPE node_netstat_TcpExt_TCPAbortFailed untyped node_netstat_TcpExt_TCPAbortFailed 0 # HELP node_netstat_TcpExt_TCPAbortOnClose Protocol TcpExt statistic TCPAbortOnClose. # TYPE node_netstat_TcpExt_TCPAbortOnClose untyped node_netstat_TcpExt_TCPAbortOnClose 1 # HELP node_netstat_TcpExt_TCPAbortOnData Protocol TcpExt statistic TCPAbortOnData. # TYPE node_netstat_TcpExt_TCPAbortOnData untyped node_netstat_TcpExt_TCPAbortOnData 2 # HELP node_netstat_TcpExt_TCPAbortOnLinger Protocol TcpExt statistic TCPAbortOnLinger. # TYPE node_netstat_TcpExt_TCPAbortOnLinger untyped node_netstat_TcpExt_TCPAbortOnLinger 0 # HELP node_netstat_TcpExt_TCPAbortOnMemory Protocol TcpExt statistic TCPAbortOnMemory. # TYPE node_netstat_TcpExt_TCPAbortOnMemory untyped node_netstat_TcpExt_TCPAbortOnMemory 0 # HELP node_netstat_TcpExt_TCPAbortOnTimeout Protocol TcpExt statistic TCPAbortOnTimeout. # TYPE node_netstat_TcpExt_TCPAbortOnTimeout untyped node_netstat_TcpExt_TCPAbortOnTimeout 0 # HELP node_netstat_TcpExt_TCPAutoCorking Protocol TcpExt statistic TCPAutoCorking. # TYPE node_netstat_TcpExt_TCPAutoCorking untyped node_netstat_TcpExt_TCPAutoCorking 4 # HELP node_netstat_TcpExt_TCPBacklogDrop Protocol TcpExt statistic TCPBacklogDrop. # TYPE node_netstat_TcpExt_TCPBacklogDrop untyped node_netstat_TcpExt_TCPBacklogDrop 0 # HELP node_netstat_TcpExt_TCPChallengeACK Protocol TcpExt statistic TCPChallengeACK. # TYPE node_netstat_TcpExt_TCPChallengeACK untyped node_netstat_TcpExt_TCPChallengeACK 0 # HELP node_netstat_TcpExt_TCPDSACKIgnoredNoUndo Protocol TcpExt statistic TCPDSACKIgnoredNoUndo. # TYPE node_netstat_TcpExt_TCPDSACKIgnoredNoUndo untyped node_netstat_TcpExt_TCPDSACKIgnoredNoUndo 0 # HELP node_netstat_TcpExt_TCPDSACKIgnoredOld Protocol TcpExt statistic TCPDSACKIgnoredOld. # TYPE node_netstat_TcpExt_TCPDSACKIgnoredOld untyped node_netstat_TcpExt_TCPDSACKIgnoredOld 0 # HELP node_netstat_TcpExt_TCPDSACKOfoRecv Protocol TcpExt statistic TCPDSACKOfoRecv. # TYPE node_netstat_TcpExt_TCPDSACKOfoRecv untyped node_netstat_TcpExt_TCPDSACKOfoRecv 0 # HELP node_netstat_TcpExt_TCPDSACKOfoSent Protocol TcpExt statistic TCPDSACKOfoSent. # TYPE node_netstat_TcpExt_TCPDSACKOfoSent untyped node_netstat_TcpExt_TCPDSACKOfoSent 0 # HELP node_netstat_TcpExt_TCPDSACKOldSent Protocol TcpExt statistic TCPDSACKOldSent. # TYPE node_netstat_TcpExt_TCPDSACKOldSent untyped node_netstat_TcpExt_TCPDSACKOldSent 0 # HELP node_netstat_TcpExt_TCPDSACKRecv Protocol TcpExt statistic TCPDSACKRecv. # TYPE node_netstat_TcpExt_TCPDSACKRecv untyped node_netstat_TcpExt_TCPDSACKRecv 0 # HELP node_netstat_TcpExt_TCPDSACKUndo Protocol TcpExt statistic TCPDSACKUndo. # TYPE node_netstat_TcpExt_TCPDSACKUndo untyped node_netstat_TcpExt_TCPDSACKUndo 0 # HELP node_netstat_TcpExt_TCPDeferAcceptDrop Protocol TcpExt statistic TCPDeferAcceptDrop. # TYPE node_netstat_TcpExt_TCPDeferAcceptDrop untyped node_netstat_TcpExt_TCPDeferAcceptDrop 0 # HELP node_netstat_TcpExt_TCPDirectCopyFromBacklog Protocol TcpExt statistic TCPDirectCopyFromBacklog. # TYPE node_netstat_TcpExt_TCPDirectCopyFromBacklog untyped node_netstat_TcpExt_TCPDirectCopyFromBacklog 0 # HELP node_netstat_TcpExt_TCPDirectCopyFromPrequeue Protocol TcpExt statistic TCPDirectCopyFromPrequeue. # TYPE node_netstat_TcpExt_TCPDirectCopyFromPrequeue untyped node_netstat_TcpExt_TCPDirectCopyFromPrequeue 1.062e+06 # HELP node_netstat_TcpExt_TCPFACKReorder Protocol TcpExt statistic TCPFACKReorder. # TYPE node_netstat_TcpExt_TCPFACKReorder untyped node_netstat_TcpExt_TCPFACKReorder 0 # HELP node_netstat_TcpExt_TCPFastOpenActive Protocol TcpExt statistic TCPFastOpenActive. # TYPE node_netstat_TcpExt_TCPFastOpenActive untyped node_netstat_TcpExt_TCPFastOpenActive 0 # HELP node_netstat_TcpExt_TCPFastOpenActiveFail Protocol TcpExt statistic TCPFastOpenActiveFail. # TYPE node_netstat_TcpExt_TCPFastOpenActiveFail untyped node_netstat_TcpExt_TCPFastOpenActiveFail 0 # HELP node_netstat_TcpExt_TCPFastOpenCookieReqd Protocol TcpExt statistic TCPFastOpenCookieReqd. # TYPE node_netstat_TcpExt_TCPFastOpenCookieReqd untyped node_netstat_TcpExt_TCPFastOpenCookieReqd 0 # HELP node_netstat_TcpExt_TCPFastOpenListenOverflow Protocol TcpExt statistic TCPFastOpenListenOverflow. # TYPE node_netstat_TcpExt_TCPFastOpenListenOverflow untyped node_netstat_TcpExt_TCPFastOpenListenOverflow 0 # HELP node_netstat_TcpExt_TCPFastOpenPassive Protocol TcpExt statistic TCPFastOpenPassive. # TYPE node_netstat_TcpExt_TCPFastOpenPassive untyped node_netstat_TcpExt_TCPFastOpenPassive 0 # HELP node_netstat_TcpExt_TCPFastOpenPassiveFail Protocol TcpExt statistic TCPFastOpenPassiveFail. # TYPE node_netstat_TcpExt_TCPFastOpenPassiveFail untyped node_netstat_TcpExt_TCPFastOpenPassiveFail 0 # HELP node_netstat_TcpExt_TCPFastRetrans Protocol TcpExt statistic TCPFastRetrans. # TYPE node_netstat_TcpExt_TCPFastRetrans untyped node_netstat_TcpExt_TCPFastRetrans 0 # HELP node_netstat_TcpExt_TCPForwardRetrans Protocol TcpExt statistic TCPForwardRetrans. # TYPE node_netstat_TcpExt_TCPForwardRetrans untyped node_netstat_TcpExt_TCPForwardRetrans 0 # HELP node_netstat_TcpExt_TCPFromZeroWindowAdv Protocol TcpExt statistic TCPFromZeroWindowAdv. # TYPE node_netstat_TcpExt_TCPFromZeroWindowAdv untyped node_netstat_TcpExt_TCPFromZeroWindowAdv 0 # HELP node_netstat_TcpExt_TCPFullUndo Protocol TcpExt statistic TCPFullUndo. # TYPE node_netstat_TcpExt_TCPFullUndo untyped node_netstat_TcpExt_TCPFullUndo 0 # HELP node_netstat_TcpExt_TCPHPAcks Protocol TcpExt statistic TCPHPAcks. # TYPE node_netstat_TcpExt_TCPHPAcks untyped node_netstat_TcpExt_TCPHPAcks 21314 # HELP node_netstat_TcpExt_TCPHPHits Protocol TcpExt statistic TCPHPHits. # TYPE node_netstat_TcpExt_TCPHPHits untyped node_netstat_TcpExt_TCPHPHits 3607 # HELP node_netstat_TcpExt_TCPHPHitsToUser Protocol TcpExt statistic TCPHPHitsToUser. # TYPE node_netstat_TcpExt_TCPHPHitsToUser untyped node_netstat_TcpExt_TCPHPHitsToUser 0 # HELP node_netstat_TcpExt_TCPHystartDelayCwnd Protocol TcpExt statistic TCPHystartDelayCwnd. # TYPE node_netstat_TcpExt_TCPHystartDelayCwnd untyped node_netstat_TcpExt_TCPHystartDelayCwnd 0 # HELP node_netstat_TcpExt_TCPHystartDelayDetect Protocol TcpExt statistic TCPHystartDelayDetect. # TYPE node_netstat_TcpExt_TCPHystartDelayDetect untyped node_netstat_TcpExt_TCPHystartDelayDetect 0 # HELP node_netstat_TcpExt_TCPHystartTrainCwnd Protocol TcpExt statistic TCPHystartTrainCwnd. # TYPE node_netstat_TcpExt_TCPHystartTrainCwnd untyped node_netstat_TcpExt_TCPHystartTrainCwnd 0 # HELP node_netstat_TcpExt_TCPHystartTrainDetect Protocol TcpExt statistic TCPHystartTrainDetect. # TYPE node_netstat_TcpExt_TCPHystartTrainDetect untyped node_netstat_TcpExt_TCPHystartTrainDetect 0 # HELP node_netstat_TcpExt_TCPLossFailures Protocol TcpExt statistic TCPLossFailures. # TYPE node_netstat_TcpExt_TCPLossFailures untyped node_netstat_TcpExt_TCPLossFailures 0 # HELP node_netstat_TcpExt_TCPLossProbeRecovery Protocol TcpExt statistic TCPLossProbeRecovery. # TYPE node_netstat_TcpExt_TCPLossProbeRecovery untyped node_netstat_TcpExt_TCPLossProbeRecovery 0 # HELP node_netstat_TcpExt_TCPLossProbes Protocol TcpExt statistic TCPLossProbes. # TYPE node_netstat_TcpExt_TCPLossProbes untyped node_netstat_TcpExt_TCPLossProbes 0 # HELP node_netstat_TcpExt_TCPLossUndo Protocol TcpExt statistic TCPLossUndo. # TYPE node_netstat_TcpExt_TCPLossUndo untyped node_netstat_TcpExt_TCPLossUndo 0 # HELP node_netstat_TcpExt_TCPLostRetransmit Protocol TcpExt statistic TCPLostRetransmit. # TYPE node_netstat_TcpExt_TCPLostRetransmit untyped node_netstat_TcpExt_TCPLostRetransmit 0 # HELP node_netstat_TcpExt_TCPMD5NotFound Protocol TcpExt statistic TCPMD5NotFound. # TYPE node_netstat_TcpExt_TCPMD5NotFound untyped node_netstat_TcpExt_TCPMD5NotFound 0 # HELP node_netstat_TcpExt_TCPMD5Unexpected Protocol TcpExt statistic TCPMD5Unexpected. # TYPE node_netstat_TcpExt_TCPMD5Unexpected untyped node_netstat_TcpExt_TCPMD5Unexpected 0 # HELP node_netstat_TcpExt_TCPMemoryPressures Protocol TcpExt statistic TCPMemoryPressures. # TYPE node_netstat_TcpExt_TCPMemoryPressures untyped node_netstat_TcpExt_TCPMemoryPressures 0 # HELP node_netstat_TcpExt_TCPMinTTLDrop Protocol TcpExt statistic TCPMinTTLDrop. # TYPE node_netstat_TcpExt_TCPMinTTLDrop untyped node_netstat_TcpExt_TCPMinTTLDrop 0 # HELP node_netstat_TcpExt_TCPOFODrop Protocol TcpExt statistic TCPOFODrop. # TYPE node_netstat_TcpExt_TCPOFODrop untyped node_netstat_TcpExt_TCPOFODrop 0 # HELP node_netstat_TcpExt_TCPOFOMerge Protocol TcpExt statistic TCPOFOMerge. # TYPE node_netstat_TcpExt_TCPOFOMerge untyped node_netstat_TcpExt_TCPOFOMerge 0 # HELP node_netstat_TcpExt_TCPOFOQueue Protocol TcpExt statistic TCPOFOQueue. # TYPE node_netstat_TcpExt_TCPOFOQueue untyped node_netstat_TcpExt_TCPOFOQueue 0 # HELP node_netstat_TcpExt_TCPOrigDataSent Protocol TcpExt statistic TCPOrigDataSent. # TYPE node_netstat_TcpExt_TCPOrigDataSent untyped node_netstat_TcpExt_TCPOrigDataSent 99807 # HELP node_netstat_TcpExt_TCPPartialUndo Protocol TcpExt statistic TCPPartialUndo. # TYPE node_netstat_TcpExt_TCPPartialUndo untyped node_netstat_TcpExt_TCPPartialUndo 0 # HELP node_netstat_TcpExt_TCPPrequeueDropped Protocol TcpExt statistic TCPPrequeueDropped. # TYPE node_netstat_TcpExt_TCPPrequeueDropped untyped node_netstat_TcpExt_TCPPrequeueDropped 0 # HELP node_netstat_TcpExt_TCPPrequeued Protocol TcpExt statistic TCPPrequeued. # TYPE node_netstat_TcpExt_TCPPrequeued untyped node_netstat_TcpExt_TCPPrequeued 68753 # HELP node_netstat_TcpExt_TCPPureAcks Protocol TcpExt statistic TCPPureAcks. # TYPE node_netstat_TcpExt_TCPPureAcks untyped node_netstat_TcpExt_TCPPureAcks 17834 # HELP node_netstat_TcpExt_TCPRcvCoalesce Protocol TcpExt statistic TCPRcvCoalesce. # TYPE node_netstat_TcpExt_TCPRcvCoalesce untyped node_netstat_TcpExt_TCPRcvCoalesce 2 # HELP node_netstat_TcpExt_TCPRcvCollapsed Protocol TcpExt statistic TCPRcvCollapsed. # TYPE node_netstat_TcpExt_TCPRcvCollapsed untyped node_netstat_TcpExt_TCPRcvCollapsed 0 # HELP node_netstat_TcpExt_TCPRenoFailures Protocol TcpExt statistic TCPRenoFailures. # TYPE node_netstat_TcpExt_TCPRenoFailures untyped node_netstat_TcpExt_TCPRenoFailures 0 # HELP node_netstat_TcpExt_TCPRenoRecovery Protocol TcpExt statistic TCPRenoRecovery. # TYPE node_netstat_TcpExt_TCPRenoRecovery untyped node_netstat_TcpExt_TCPRenoRecovery 0 # HELP node_netstat_TcpExt_TCPRenoRecoveryFail Protocol TcpExt statistic TCPRenoRecoveryFail. # TYPE node_netstat_TcpExt_TCPRenoRecoveryFail untyped node_netstat_TcpExt_TCPRenoRecoveryFail 0 # HELP node_netstat_TcpExt_TCPRenoReorder Protocol TcpExt statistic TCPRenoReorder. # TYPE node_netstat_TcpExt_TCPRenoReorder untyped node_netstat_TcpExt_TCPRenoReorder 0 # HELP node_netstat_TcpExt_TCPReqQFullDoCookies Protocol TcpExt statistic TCPReqQFullDoCookies. # TYPE node_netstat_TcpExt_TCPReqQFullDoCookies untyped node_netstat_TcpExt_TCPReqQFullDoCookies 0 # HELP node_netstat_TcpExt_TCPReqQFullDrop Protocol TcpExt statistic TCPReqQFullDrop. # TYPE node_netstat_TcpExt_TCPReqQFullDrop untyped node_netstat_TcpExt_TCPReqQFullDrop 0 # HELP node_netstat_TcpExt_TCPRetransFail Protocol TcpExt statistic TCPRetransFail. # TYPE node_netstat_TcpExt_TCPRetransFail untyped node_netstat_TcpExt_TCPRetransFail 0 # HELP node_netstat_TcpExt_TCPSACKDiscard Protocol TcpExt statistic TCPSACKDiscard. # TYPE node_netstat_TcpExt_TCPSACKDiscard untyped node_netstat_TcpExt_TCPSACKDiscard 0 # HELP node_netstat_TcpExt_TCPSACKReneging Protocol TcpExt statistic TCPSACKReneging. # TYPE node_netstat_TcpExt_TCPSACKReneging untyped node_netstat_TcpExt_TCPSACKReneging 0 # HELP node_netstat_TcpExt_TCPSACKReorder Protocol TcpExt statistic TCPSACKReorder. # TYPE node_netstat_TcpExt_TCPSACKReorder untyped node_netstat_TcpExt_TCPSACKReorder 0 # HELP node_netstat_TcpExt_TCPSYNChallenge Protocol TcpExt statistic TCPSYNChallenge. # TYPE node_netstat_TcpExt_TCPSYNChallenge untyped node_netstat_TcpExt_TCPSYNChallenge 0 # HELP node_netstat_TcpExt_TCPSackFailures Protocol TcpExt statistic TCPSackFailures. # TYPE node_netstat_TcpExt_TCPSackFailures untyped node_netstat_TcpExt_TCPSackFailures 0 # HELP node_netstat_TcpExt_TCPSackMerged Protocol TcpExt statistic TCPSackMerged. # TYPE node_netstat_TcpExt_TCPSackMerged untyped node_netstat_TcpExt_TCPSackMerged 0 # HELP node_netstat_TcpExt_TCPSackRecovery Protocol TcpExt statistic TCPSackRecovery. # TYPE node_netstat_TcpExt_TCPSackRecovery untyped node_netstat_TcpExt_TCPSackRecovery 0 # HELP node_netstat_TcpExt_TCPSackRecoveryFail Protocol TcpExt statistic TCPSackRecoveryFail. # TYPE node_netstat_TcpExt_TCPSackRecoveryFail untyped node_netstat_TcpExt_TCPSackRecoveryFail 0 # HELP node_netstat_TcpExt_TCPSackShiftFallback Protocol TcpExt statistic TCPSackShiftFallback. # TYPE node_netstat_TcpExt_TCPSackShiftFallback untyped node_netstat_TcpExt_TCPSackShiftFallback 0 # HELP node_netstat_TcpExt_TCPSackShifted Protocol TcpExt statistic TCPSackShifted. # TYPE node_netstat_TcpExt_TCPSackShifted untyped node_netstat_TcpExt_TCPSackShifted 0 # HELP node_netstat_TcpExt_TCPSchedulerFailed Protocol TcpExt statistic TCPSchedulerFailed. # TYPE node_netstat_TcpExt_TCPSchedulerFailed untyped node_netstat_TcpExt_TCPSchedulerFailed 0 # HELP node_netstat_TcpExt_TCPSlowStartRetrans Protocol TcpExt statistic TCPSlowStartRetrans. # TYPE node_netstat_TcpExt_TCPSlowStartRetrans untyped node_netstat_TcpExt_TCPSlowStartRetrans 0 # HELP node_netstat_TcpExt_TCPSpuriousRTOs Protocol TcpExt statistic TCPSpuriousRTOs. # TYPE node_netstat_TcpExt_TCPSpuriousRTOs untyped node_netstat_TcpExt_TCPSpuriousRTOs 0 # HELP node_netstat_TcpExt_TCPSpuriousRtxHostQueues Protocol TcpExt statistic TCPSpuriousRtxHostQueues. # TYPE node_netstat_TcpExt_TCPSpuriousRtxHostQueues untyped node_netstat_TcpExt_TCPSpuriousRtxHostQueues 0 # HELP node_netstat_TcpExt_TCPSynRetrans Protocol TcpExt statistic TCPSynRetrans. # TYPE node_netstat_TcpExt_TCPSynRetrans untyped node_netstat_TcpExt_TCPSynRetrans 0 # HELP node_netstat_TcpExt_TCPTSReorder Protocol TcpExt statistic TCPTSReorder. # TYPE node_netstat_TcpExt_TCPTSReorder untyped node_netstat_TcpExt_TCPTSReorder 0 # HELP node_netstat_TcpExt_TCPTimeWaitOverflow Protocol TcpExt statistic TCPTimeWaitOverflow. # TYPE node_netstat_TcpExt_TCPTimeWaitOverflow untyped node_netstat_TcpExt_TCPTimeWaitOverflow 0 # HELP node_netstat_TcpExt_TCPTimeouts Protocol TcpExt statistic TCPTimeouts. # TYPE node_netstat_TcpExt_TCPTimeouts untyped node_netstat_TcpExt_TCPTimeouts 0 # HELP node_netstat_TcpExt_TCPToZeroWindowAdv Protocol TcpExt statistic TCPToZeroWindowAdv. # TYPE node_netstat_TcpExt_TCPToZeroWindowAdv untyped node_netstat_TcpExt_TCPToZeroWindowAdv 0 # HELP node_netstat_TcpExt_TCPWantZeroWindowAdv Protocol TcpExt statistic TCPWantZeroWindowAdv. # TYPE node_netstat_TcpExt_TCPWantZeroWindowAdv untyped node_netstat_TcpExt_TCPWantZeroWindowAdv 0 # HELP node_netstat_TcpExt_TW Protocol TcpExt statistic TW. # TYPE node_netstat_TcpExt_TW untyped node_netstat_TcpExt_TW 35 # HELP node_netstat_TcpExt_TWKilled Protocol TcpExt statistic TWKilled. # TYPE node_netstat_TcpExt_TWKilled untyped node_netstat_TcpExt_TWKilled 0 # HELP node_netstat_TcpExt_TWRecycled Protocol TcpExt statistic TWRecycled. # TYPE node_netstat_TcpExt_TWRecycled untyped node_netstat_TcpExt_TWRecycled 0 # HELP node_netstat_Tcp_ActiveOpens Protocol Tcp statistic ActiveOpens. # TYPE node_netstat_Tcp_ActiveOpens untyped node_netstat_Tcp_ActiveOpens 122 # HELP node_netstat_Tcp_AttemptFails Protocol Tcp statistic AttemptFails. # TYPE node_netstat_Tcp_AttemptFails untyped node_netstat_Tcp_AttemptFails 48 # HELP node_netstat_Tcp_CurrEstab Protocol Tcp statistic CurrEstab. # TYPE node_netstat_Tcp_CurrEstab untyped node_netstat_Tcp_CurrEstab 67 # HELP node_netstat_Tcp_EstabResets Protocol Tcp statistic EstabResets. # TYPE node_netstat_Tcp_EstabResets untyped node_netstat_Tcp_EstabResets 3 # HELP node_netstat_Tcp_InCsumErrors Protocol Tcp statistic InCsumErrors. # TYPE node_netstat_Tcp_InCsumErrors untyped node_netstat_Tcp_InCsumErrors 0 # HELP node_netstat_Tcp_InErrs Protocol Tcp statistic InErrs. # TYPE node_netstat_Tcp_InErrs untyped node_netstat_Tcp_InErrs 0 # HELP node_netstat_Tcp_InSegs Protocol Tcp statistic InSegs. # TYPE node_netstat_Tcp_InSegs untyped node_netstat_Tcp_InSegs 138959 # HELP node_netstat_Tcp_MaxConn Protocol Tcp statistic MaxConn. # TYPE node_netstat_Tcp_MaxConn untyped node_netstat_Tcp_MaxConn -1 # HELP node_netstat_Tcp_OutRsts Protocol Tcp statistic OutRsts. # TYPE node_netstat_Tcp_OutRsts untyped node_netstat_Tcp_OutRsts 51 # HELP node_netstat_Tcp_OutSegs Protocol Tcp statistic OutSegs. # TYPE node_netstat_Tcp_OutSegs untyped node_netstat_Tcp_OutSegs 138800 # HELP node_netstat_Tcp_PassiveOpens Protocol Tcp statistic PassiveOpens. # TYPE node_netstat_Tcp_PassiveOpens untyped node_netstat_Tcp_PassiveOpens 75 # HELP node_netstat_Tcp_RetransSegs Protocol Tcp statistic RetransSegs. # TYPE node_netstat_Tcp_RetransSegs untyped node_netstat_Tcp_RetransSegs 0 # HELP node_netstat_Tcp_RtoAlgorithm Protocol Tcp statistic RtoAlgorithm. # TYPE node_netstat_Tcp_RtoAlgorithm untyped node_netstat_Tcp_RtoAlgorithm 1 # HELP node_netstat_Tcp_RtoMax Protocol Tcp statistic RtoMax. # TYPE node_netstat_Tcp_RtoMax untyped node_netstat_Tcp_RtoMax 120000 # HELP node_netstat_Tcp_RtoMin Protocol Tcp statistic RtoMin. # TYPE node_netstat_Tcp_RtoMin untyped node_netstat_Tcp_RtoMin 200 # HELP node_netstat_UdpLite_InCsumErrors Protocol UdpLite statistic InCsumErrors. # TYPE node_netstat_UdpLite_InCsumErrors untyped node_netstat_UdpLite_InCsumErrors 0 # HELP node_netstat_UdpLite_InDatagrams Protocol UdpLite statistic InDatagrams. # TYPE node_netstat_UdpLite_InDatagrams untyped node_netstat_UdpLite_InDatagrams 0 # HELP node_netstat_UdpLite_InErrors Protocol UdpLite statistic InErrors. # TYPE node_netstat_UdpLite_InErrors untyped node_netstat_UdpLite_InErrors 0 # HELP node_netstat_UdpLite_NoPorts Protocol UdpLite statistic NoPorts. # TYPE node_netstat_UdpLite_NoPorts untyped node_netstat_UdpLite_NoPorts 0 # HELP node_netstat_UdpLite_OutDatagrams Protocol UdpLite statistic OutDatagrams. # TYPE node_netstat_UdpLite_OutDatagrams untyped node_netstat_UdpLite_OutDatagrams 0 # HELP node_netstat_UdpLite_RcvbufErrors Protocol UdpLite statistic RcvbufErrors. # TYPE node_netstat_UdpLite_RcvbufErrors untyped node_netstat_UdpLite_RcvbufErrors 0 # HELP node_netstat_UdpLite_SndbufErrors Protocol UdpLite statistic SndbufErrors. # TYPE node_netstat_UdpLite_SndbufErrors untyped node_netstat_UdpLite_SndbufErrors 0 # HELP node_netstat_Udp_InCsumErrors Protocol Udp statistic InCsumErrors. # TYPE node_netstat_Udp_InCsumErrors untyped node_netstat_Udp_InCsumErrors 0 # HELP node_netstat_Udp_InDatagrams Protocol Udp statistic InDatagrams. # TYPE node_netstat_Udp_InDatagrams untyped node_netstat_Udp_InDatagrams 347 # HELP node_netstat_Udp_InErrors Protocol Udp statistic InErrors. # TYPE node_netstat_Udp_InErrors untyped node_netstat_Udp_InErrors 0 # HELP node_netstat_Udp_NoPorts Protocol Udp statistic NoPorts. # TYPE node_netstat_Udp_NoPorts untyped node_netstat_Udp_NoPorts 32 # HELP node_netstat_Udp_OutDatagrams Protocol Udp statistic OutDatagrams. # TYPE node_netstat_Udp_OutDatagrams untyped node_netstat_Udp_OutDatagrams 388 # HELP node_netstat_Udp_RcvbufErrors Protocol Udp statistic RcvbufErrors. # TYPE node_netstat_Udp_RcvbufErrors untyped node_netstat_Udp_RcvbufErrors 0 # HELP node_netstat_Udp_SndbufErrors Protocol Udp statistic SndbufErrors. # TYPE node_netstat_Udp_SndbufErrors untyped node_netstat_Udp_SndbufErrors 0 # HELP node_network_receive_bytes Network device statistic receive_bytes. # TYPE node_network_receive_bytes gauge node_network_receive_bytes{device="docker0"} 0 node_network_receive_bytes{device="enp0s3"} 36421 node_network_receive_bytes{device="enp0s8"} 80451 node_network_receive_bytes{device="lo"} 3.4463599e+07 # HELP node_network_receive_compressed Network device statistic receive_compressed. # TYPE node_network_receive_compressed gauge node_network_receive_compressed{device="docker0"} 0 node_network_receive_compressed{device="enp0s3"} 0 node_network_receive_compressed{device="enp0s8"} 0 node_network_receive_compressed{device="lo"} 0 # HELP node_network_receive_drop Network device statistic receive_drop. # TYPE node_network_receive_drop gauge node_network_receive_drop{device="docker0"} 0 node_network_receive_drop{device="enp0s3"} 0 node_network_receive_drop{device="enp0s8"} 0 node_network_receive_drop{device="lo"} 0 # HELP node_network_receive_errs Network device statistic receive_errs. # TYPE node_network_receive_errs gauge node_network_receive_errs{device="docker0"} 0 node_network_receive_errs{device="enp0s3"} 0 node_network_receive_errs{device="enp0s8"} 0 node_network_receive_errs{device="lo"} 0 # HELP node_network_receive_fifo Network device statistic receive_fifo. # TYPE node_network_receive_fifo gauge node_network_receive_fifo{device="docker0"} 0 node_network_receive_fifo{device="enp0s3"} 0 node_network_receive_fifo{device="enp0s8"} 0 node_network_receive_fifo{device="lo"} 0 # HELP node_network_receive_frame Network device statistic receive_frame. # TYPE node_network_receive_frame gauge node_network_receive_frame{device="docker0"} 0 node_network_receive_frame{device="enp0s3"} 0 node_network_receive_frame{device="enp0s8"} 0 node_network_receive_frame{device="lo"} 0 # HELP node_network_receive_multicast Network device statistic receive_multicast. # TYPE node_network_receive_multicast gauge node_network_receive_multicast{device="docker0"} 0 node_network_receive_multicast{device="enp0s3"} 0 node_network_receive_multicast{device="enp0s8"} 0 node_network_receive_multicast{device="lo"} 0 # HELP node_network_receive_packets Network device statistic receive_packets. # TYPE node_network_receive_packets gauge node_network_receive_packets{device="docker0"} 0 node_network_receive_packets{device="enp0s3"} 421 node_network_receive_packets{device="enp0s8"} 997 node_network_receive_packets{device="lo"} 138313 # HELP node_network_transmit_bytes Network device statistic transmit_bytes. # TYPE node_network_transmit_bytes gauge node_network_transmit_bytes{device="docker0"} 0 node_network_transmit_bytes{device="enp0s3"} 37074 node_network_transmit_bytes{device="enp0s8"} 330717 node_network_transmit_bytes{device="lo"} 3.4463599e+07 # HELP node_network_transmit_compressed Network device statistic transmit_compressed. # TYPE node_network_transmit_compressed gauge node_network_transmit_compressed{device="docker0"} 0 node_network_transmit_compressed{device="enp0s3"} 0 node_network_transmit_compressed{device="enp0s8"} 0 node_network_transmit_compressed{device="lo"} 0 # HELP node_network_transmit_drop Network device statistic transmit_drop. # TYPE node_network_transmit_drop gauge node_network_transmit_drop{device="docker0"} 0 node_network_transmit_drop{device="enp0s3"} 0 node_network_transmit_drop{device="enp0s8"} 0 node_network_transmit_drop{device="lo"} 0 # HELP node_network_transmit_errs Network device statistic transmit_errs. # TYPE node_network_transmit_errs gauge node_network_transmit_errs{device="docker0"} 0 node_network_transmit_errs{device="enp0s3"} 0 node_network_transmit_errs{device="enp0s8"} 0 node_network_transmit_errs{device="lo"} 0 # HELP node_network_transmit_fifo Network device statistic transmit_fifo. # TYPE node_network_transmit_fifo gauge node_network_transmit_fifo{device="docker0"} 0 node_network_transmit_fifo{device="enp0s3"} 0 node_network_transmit_fifo{device="enp0s8"} 0 node_network_transmit_fifo{device="lo"} 0 # HELP node_network_transmit_frame Network device statistic transmit_frame. # TYPE node_network_transmit_frame gauge node_network_transmit_frame{device="docker0"} 0 node_network_transmit_frame{device="enp0s3"} 0 node_network_transmit_frame{device="enp0s8"} 0 node_network_transmit_frame{device="lo"} 0 # HELP node_network_transmit_multicast Network device statistic transmit_multicast. # TYPE node_network_transmit_multicast gauge node_network_transmit_multicast{device="docker0"} 0 node_network_transmit_multicast{device="enp0s3"} 0 node_network_transmit_multicast{device="enp0s8"} 0 node_network_transmit_multicast{device="lo"} 0 # HELP node_network_transmit_packets Network device statistic transmit_packets. # TYPE node_network_transmit_packets gauge node_network_transmit_packets{device="docker0"} 0 node_network_transmit_packets{device="enp0s3"} 438 node_network_transmit_packets{device="enp0s8"} 627 node_network_transmit_packets{device="lo"} 138313 # HELP node_procs_blocked Number of processes blocked waiting for I/O to complete. # TYPE node_procs_blocked gauge node_procs_blocked 0 # HELP node_procs_running Number of processes in runnable state. # TYPE node_procs_running gauge node_procs_running 2 # HELP node_time System time in seconds since epoch (1970). # TYPE node_time counter node_time 1.48891092e+09 # HELP node_uname_info Labeled system information as provided by the uname system call. # TYPE node_uname_info gauge node_uname_info{domainname="(none)",machine="x86_64",nodename="centos7",release="3.10.0-514.10.2.el7.x86_64",sysname="Linux",version="#1 SMP Fri Mar 3 00:04:05 UTC 2017"} 1 # HELP node_vmstat_allocstall /proc/vmstat information field allocstall. # TYPE node_vmstat_allocstall untyped node_vmstat_allocstall 0 # HELP node_vmstat_balloon_deflate /proc/vmstat information field balloon_deflate. # TYPE node_vmstat_balloon_deflate untyped node_vmstat_balloon_deflate 0 # HELP node_vmstat_balloon_inflate /proc/vmstat information field balloon_inflate. # TYPE node_vmstat_balloon_inflate untyped node_vmstat_balloon_inflate 0 # HELP node_vmstat_balloon_migrate /proc/vmstat information field balloon_migrate. # TYPE node_vmstat_balloon_migrate untyped node_vmstat_balloon_migrate 0 # HELP node_vmstat_compact_fail /proc/vmstat information field compact_fail. # TYPE node_vmstat_compact_fail untyped node_vmstat_compact_fail 0 # HELP node_vmstat_compact_free_scanned /proc/vmstat information field compact_free_scanned. # TYPE node_vmstat_compact_free_scanned untyped node_vmstat_compact_free_scanned 0 # HELP node_vmstat_compact_isolated /proc/vmstat information field compact_isolated. # TYPE node_vmstat_compact_isolated untyped node_vmstat_compact_isolated 0 # HELP node_vmstat_compact_migrate_scanned /proc/vmstat information field compact_migrate_scanned. # TYPE node_vmstat_compact_migrate_scanned untyped node_vmstat_compact_migrate_scanned 0 # HELP node_vmstat_compact_stall /proc/vmstat information field compact_stall. # TYPE node_vmstat_compact_stall untyped node_vmstat_compact_stall 0 # HELP node_vmstat_compact_success /proc/vmstat information field compact_success. # TYPE node_vmstat_compact_success untyped node_vmstat_compact_success 0 # HELP node_vmstat_drop_pagecache /proc/vmstat information field drop_pagecache. # TYPE node_vmstat_drop_pagecache untyped node_vmstat_drop_pagecache 0 # HELP node_vmstat_drop_slab /proc/vmstat information field drop_slab. # TYPE node_vmstat_drop_slab untyped node_vmstat_drop_slab 0 # HELP node_vmstat_htlb_buddy_alloc_fail /proc/vmstat information field htlb_buddy_alloc_fail. # TYPE node_vmstat_htlb_buddy_alloc_fail untyped node_vmstat_htlb_buddy_alloc_fail 0 # HELP node_vmstat_htlb_buddy_alloc_success /proc/vmstat information field htlb_buddy_alloc_success. # TYPE node_vmstat_htlb_buddy_alloc_success untyped node_vmstat_htlb_buddy_alloc_success 0 # HELP node_vmstat_kswapd_high_wmark_hit_quickly /proc/vmstat information field kswapd_high_wmark_hit_quickly. # TYPE node_vmstat_kswapd_high_wmark_hit_quickly untyped node_vmstat_kswapd_high_wmark_hit_quickly 0 # HELP node_vmstat_kswapd_inodesteal /proc/vmstat information field kswapd_inodesteal. # TYPE node_vmstat_kswapd_inodesteal untyped node_vmstat_kswapd_inodesteal 0 # HELP node_vmstat_kswapd_low_wmark_hit_quickly /proc/vmstat information field kswapd_low_wmark_hit_quickly. # TYPE node_vmstat_kswapd_low_wmark_hit_quickly untyped node_vmstat_kswapd_low_wmark_hit_quickly 0 # HELP node_vmstat_nr_active_anon /proc/vmstat information field nr_active_anon. # TYPE node_vmstat_nr_active_anon untyped node_vmstat_nr_active_anon 88528 # HELP node_vmstat_nr_active_file /proc/vmstat information field nr_active_file. # TYPE node_vmstat_nr_active_file untyped node_vmstat_nr_active_file 28848 # HELP node_vmstat_nr_alloc_batch /proc/vmstat information field nr_alloc_batch. # TYPE node_vmstat_nr_alloc_batch untyped node_vmstat_nr_alloc_batch 2642 # HELP node_vmstat_nr_anon_pages /proc/vmstat information field nr_anon_pages. # TYPE node_vmstat_nr_anon_pages untyped node_vmstat_nr_anon_pages 85414 # HELP node_vmstat_nr_anon_transparent_hugepages /proc/vmstat information field nr_anon_transparent_hugepages. # TYPE node_vmstat_nr_anon_transparent_hugepages untyped node_vmstat_nr_anon_transparent_hugepages 6 # HELP node_vmstat_nr_bounce /proc/vmstat information field nr_bounce. # TYPE node_vmstat_nr_bounce untyped node_vmstat_nr_bounce 0 # HELP node_vmstat_nr_dirtied /proc/vmstat information field nr_dirtied. # TYPE node_vmstat_nr_dirtied untyped node_vmstat_nr_dirtied 29852 # HELP node_vmstat_nr_dirty /proc/vmstat information field nr_dirty. # TYPE node_vmstat_nr_dirty untyped node_vmstat_nr_dirty 17 # HELP node_vmstat_nr_dirty_background_threshold /proc/vmstat information field nr_dirty_background_threshold. # TYPE node_vmstat_nr_dirty_background_threshold untyped node_vmstat_nr_dirty_background_threshold 39900 # HELP node_vmstat_nr_dirty_threshold /proc/vmstat information field nr_dirty_threshold. # TYPE node_vmstat_nr_dirty_threshold untyped node_vmstat_nr_dirty_threshold 119701 # HELP node_vmstat_nr_file_pages /proc/vmstat information field nr_file_pages. # TYPE node_vmstat_nr_file_pages untyped node_vmstat_nr_file_pages 76687 # HELP node_vmstat_nr_free_cma /proc/vmstat information field nr_free_cma. # TYPE node_vmstat_nr_free_cma untyped node_vmstat_nr_free_cma 0 # HELP node_vmstat_nr_free_pages /proc/vmstat information field nr_free_pages. # TYPE node_vmstat_nr_free_pages untyped node_vmstat_nr_free_pages 769929 # HELP node_vmstat_nr_inactive_anon /proc/vmstat information field nr_inactive_anon. # TYPE node_vmstat_nr_inactive_anon untyped node_vmstat_nr_inactive_anon 2091 # HELP node_vmstat_nr_inactive_file /proc/vmstat information field nr_inactive_file. # TYPE node_vmstat_nr_inactive_file untyped node_vmstat_nr_inactive_file 45688 # HELP node_vmstat_nr_isolated_anon /proc/vmstat information field nr_isolated_anon. # TYPE node_vmstat_nr_isolated_anon untyped node_vmstat_nr_isolated_anon 0 # HELP node_vmstat_nr_isolated_file /proc/vmstat information field nr_isolated_file. # TYPE node_vmstat_nr_isolated_file untyped node_vmstat_nr_isolated_file 0 # HELP node_vmstat_nr_kernel_stack /proc/vmstat information field nr_kernel_stack. # TYPE node_vmstat_nr_kernel_stack untyped node_vmstat_nr_kernel_stack 707 # HELP node_vmstat_nr_mapped /proc/vmstat information field nr_mapped. # TYPE node_vmstat_nr_mapped untyped node_vmstat_nr_mapped 16487 # HELP node_vmstat_nr_mlock /proc/vmstat information field nr_mlock. # TYPE node_vmstat_nr_mlock untyped node_vmstat_nr_mlock 0 # HELP node_vmstat_nr_page_table_pages /proc/vmstat information field nr_page_table_pages. # TYPE node_vmstat_nr_page_table_pages untyped node_vmstat_nr_page_table_pages 2223 # HELP node_vmstat_nr_shmem /proc/vmstat information field nr_shmem. # TYPE node_vmstat_nr_shmem untyped node_vmstat_nr_shmem 2151 # HELP node_vmstat_nr_slab_reclaimable /proc/vmstat information field nr_slab_reclaimable. # TYPE node_vmstat_nr_slab_reclaimable untyped node_vmstat_nr_slab_reclaimable 10898 # HELP node_vmstat_nr_slab_unreclaimable /proc/vmstat information field nr_slab_unreclaimable. # TYPE node_vmstat_nr_slab_unreclaimable untyped node_vmstat_nr_slab_unreclaimable 7827 # HELP node_vmstat_nr_unevictable /proc/vmstat information field nr_unevictable. # TYPE node_vmstat_nr_unevictable untyped node_vmstat_nr_unevictable 0 # HELP node_vmstat_nr_unstable /proc/vmstat information field nr_unstable. # TYPE node_vmstat_nr_unstable untyped node_vmstat_nr_unstable 0 # HELP node_vmstat_nr_vmscan_immediate_reclaim /proc/vmstat information field nr_vmscan_immediate_reclaim. # TYPE node_vmstat_nr_vmscan_immediate_reclaim untyped node_vmstat_nr_vmscan_immediate_reclaim 0 # HELP node_vmstat_nr_vmscan_write /proc/vmstat information field nr_vmscan_write. # TYPE node_vmstat_nr_vmscan_write untyped node_vmstat_nr_vmscan_write 0 # HELP node_vmstat_nr_writeback /proc/vmstat information field nr_writeback. # TYPE node_vmstat_nr_writeback untyped node_vmstat_nr_writeback 0 # HELP node_vmstat_nr_writeback_temp /proc/vmstat information field nr_writeback_temp. # TYPE node_vmstat_nr_writeback_temp untyped node_vmstat_nr_writeback_temp 0 # HELP node_vmstat_nr_written /proc/vmstat information field nr_written. # TYPE node_vmstat_nr_written untyped node_vmstat_nr_written 20344 # HELP node_vmstat_numa_foreign /proc/vmstat information field numa_foreign. # TYPE node_vmstat_numa_foreign untyped node_vmstat_numa_foreign 0 # HELP node_vmstat_numa_hint_faults /proc/vmstat information field numa_hint_faults. # TYPE node_vmstat_numa_hint_faults untyped node_vmstat_numa_hint_faults 0 # HELP node_vmstat_numa_hint_faults_local /proc/vmstat information field numa_hint_faults_local. # TYPE node_vmstat_numa_hint_faults_local untyped node_vmstat_numa_hint_faults_local 0 # HELP node_vmstat_numa_hit /proc/vmstat information field numa_hit. # TYPE node_vmstat_numa_hit untyped node_vmstat_numa_hit 824961 # HELP node_vmstat_numa_huge_pte_updates /proc/vmstat information field numa_huge_pte_updates. # TYPE node_vmstat_numa_huge_pte_updates untyped node_vmstat_numa_huge_pte_updates 0 # HELP node_vmstat_numa_interleave /proc/vmstat information field numa_interleave. # TYPE node_vmstat_numa_interleave untyped node_vmstat_numa_interleave 14563 # HELP node_vmstat_numa_local /proc/vmstat information field numa_local. # TYPE node_vmstat_numa_local untyped node_vmstat_numa_local 824961 # HELP node_vmstat_numa_miss /proc/vmstat information field numa_miss. # TYPE node_vmstat_numa_miss untyped node_vmstat_numa_miss 0 # HELP node_vmstat_numa_other /proc/vmstat information field numa_other. # TYPE node_vmstat_numa_other untyped node_vmstat_numa_other 0 # HELP node_vmstat_numa_pages_migrated /proc/vmstat information field numa_pages_migrated. # TYPE node_vmstat_numa_pages_migrated untyped node_vmstat_numa_pages_migrated 0 # HELP node_vmstat_numa_pte_updates /proc/vmstat information field numa_pte_updates. # TYPE node_vmstat_numa_pte_updates untyped node_vmstat_numa_pte_updates 0 # HELP node_vmstat_pageoutrun /proc/vmstat information field pageoutrun. # TYPE node_vmstat_pageoutrun untyped node_vmstat_pageoutrun 1 # HELP node_vmstat_pgactivate /proc/vmstat information field pgactivate. # TYPE node_vmstat_pgactivate untyped node_vmstat_pgactivate 30452 # HELP node_vmstat_pgalloc_dma /proc/vmstat information field pgalloc_dma. # TYPE node_vmstat_pgalloc_dma untyped node_vmstat_pgalloc_dma 487 # HELP node_vmstat_pgalloc_dma32 /proc/vmstat information field pgalloc_dma32. # TYPE node_vmstat_pgalloc_dma32 untyped node_vmstat_pgalloc_dma32 750526 # HELP node_vmstat_pgalloc_movable /proc/vmstat information field pgalloc_movable. # TYPE node_vmstat_pgalloc_movable untyped node_vmstat_pgalloc_movable 0 # HELP node_vmstat_pgalloc_normal /proc/vmstat information field pgalloc_normal. # TYPE node_vmstat_pgalloc_normal untyped node_vmstat_pgalloc_normal 124283 # HELP node_vmstat_pgdeactivate /proc/vmstat information field pgdeactivate. # TYPE node_vmstat_pgdeactivate untyped node_vmstat_pgdeactivate 0 # HELP node_vmstat_pgfault /proc/vmstat information field pgfault. # TYPE node_vmstat_pgfault untyped node_vmstat_pgfault 1.37842e+06 # HELP node_vmstat_pgfree /proc/vmstat information field pgfree. # TYPE node_vmstat_pgfree untyped node_vmstat_pgfree 1.645789e+06 # HELP node_vmstat_pginodesteal /proc/vmstat information field pginodesteal. # TYPE node_vmstat_pginodesteal untyped node_vmstat_pginodesteal 0 # HELP node_vmstat_pgmajfault /proc/vmstat information field pgmajfault. # TYPE node_vmstat_pgmajfault untyped node_vmstat_pgmajfault 4286 # HELP node_vmstat_pgmigrate_fail /proc/vmstat information field pgmigrate_fail. # TYPE node_vmstat_pgmigrate_fail untyped node_vmstat_pgmigrate_fail 0 # HELP node_vmstat_pgmigrate_success /proc/vmstat information field pgmigrate_success. # TYPE node_vmstat_pgmigrate_success untyped node_vmstat_pgmigrate_success 0 # HELP node_vmstat_pgpgin /proc/vmstat information field pgpgin. # TYPE node_vmstat_pgpgin untyped node_vmstat_pgpgin 289776 # HELP node_vmstat_pgpgout /proc/vmstat information field pgpgout. # TYPE node_vmstat_pgpgout untyped node_vmstat_pgpgout 139684 # HELP node_vmstat_pgrefill_dma /proc/vmstat information field pgrefill_dma. # TYPE node_vmstat_pgrefill_dma untyped node_vmstat_pgrefill_dma 0 # HELP node_vmstat_pgrefill_dma32 /proc/vmstat information field pgrefill_dma32. # TYPE node_vmstat_pgrefill_dma32 untyped node_vmstat_pgrefill_dma32 0 # HELP node_vmstat_pgrefill_movable /proc/vmstat information field pgrefill_movable. # TYPE node_vmstat_pgrefill_movable untyped node_vmstat_pgrefill_movable 0 # HELP node_vmstat_pgrefill_normal /proc/vmstat information field pgrefill_normal. # TYPE node_vmstat_pgrefill_normal untyped node_vmstat_pgrefill_normal 0 # HELP node_vmstat_pgrotated /proc/vmstat information field pgrotated. # TYPE node_vmstat_pgrotated untyped node_vmstat_pgrotated 0 # HELP node_vmstat_pgscan_direct_dma /proc/vmstat information field pgscan_direct_dma. # TYPE node_vmstat_pgscan_direct_dma untyped node_vmstat_pgscan_direct_dma 0 # HELP node_vmstat_pgscan_direct_dma32 /proc/vmstat information field pgscan_direct_dma32. # TYPE node_vmstat_pgscan_direct_dma32 untyped node_vmstat_pgscan_direct_dma32 0 # HELP node_vmstat_pgscan_direct_movable /proc/vmstat information field pgscan_direct_movable. # TYPE node_vmstat_pgscan_direct_movable untyped node_vmstat_pgscan_direct_movable 0 # HELP node_vmstat_pgscan_direct_normal /proc/vmstat information field pgscan_direct_normal. # TYPE node_vmstat_pgscan_direct_normal untyped node_vmstat_pgscan_direct_normal 0 # HELP node_vmstat_pgscan_direct_throttle /proc/vmstat information field pgscan_direct_throttle. # TYPE node_vmstat_pgscan_direct_throttle untyped node_vmstat_pgscan_direct_throttle 0 # HELP node_vmstat_pgscan_kswapd_dma /proc/vmstat information field pgscan_kswapd_dma. # TYPE node_vmstat_pgscan_kswapd_dma untyped node_vmstat_pgscan_kswapd_dma 0 # HELP node_vmstat_pgscan_kswapd_dma32 /proc/vmstat information field pgscan_kswapd_dma32. # TYPE node_vmstat_pgscan_kswapd_dma32 untyped node_vmstat_pgscan_kswapd_dma32 0 # HELP node_vmstat_pgscan_kswapd_movable /proc/vmstat information field pgscan_kswapd_movable. # TYPE node_vmstat_pgscan_kswapd_movable untyped node_vmstat_pgscan_kswapd_movable 0 # HELP node_vmstat_pgscan_kswapd_normal /proc/vmstat information field pgscan_kswapd_normal. # TYPE node_vmstat_pgscan_kswapd_normal untyped node_vmstat_pgscan_kswapd_normal 0 # HELP node_vmstat_pgsteal_direct_dma /proc/vmstat information field pgsteal_direct_dma. # TYPE node_vmstat_pgsteal_direct_dma untyped node_vmstat_pgsteal_direct_dma 0 # HELP node_vmstat_pgsteal_direct_dma32 /proc/vmstat information field pgsteal_direct_dma32. # TYPE node_vmstat_pgsteal_direct_dma32 untyped node_vmstat_pgsteal_direct_dma32 0 # HELP node_vmstat_pgsteal_direct_movable /proc/vmstat information field pgsteal_direct_movable. # TYPE node_vmstat_pgsteal_direct_movable untyped node_vmstat_pgsteal_direct_movable 0 # HELP node_vmstat_pgsteal_direct_normal /proc/vmstat information field pgsteal_direct_normal. # TYPE node_vmstat_pgsteal_direct_normal untyped node_vmstat_pgsteal_direct_normal 0 # HELP node_vmstat_pgsteal_kswapd_dma /proc/vmstat information field pgsteal_kswapd_dma. # TYPE node_vmstat_pgsteal_kswapd_dma untyped node_vmstat_pgsteal_kswapd_dma 0 # HELP node_vmstat_pgsteal_kswapd_dma32 /proc/vmstat information field pgsteal_kswapd_dma32. # TYPE node_vmstat_pgsteal_kswapd_dma32 untyped node_vmstat_pgsteal_kswapd_dma32 0 # HELP node_vmstat_pgsteal_kswapd_movable /proc/vmstat information field pgsteal_kswapd_movable. # TYPE node_vmstat_pgsteal_kswapd_movable untyped node_vmstat_pgsteal_kswapd_movable 0 # HELP node_vmstat_pgsteal_kswapd_normal /proc/vmstat information field pgsteal_kswapd_normal. # TYPE node_vmstat_pgsteal_kswapd_normal untyped node_vmstat_pgsteal_kswapd_normal 0 # HELP node_vmstat_pswpin /proc/vmstat information field pswpin. # TYPE node_vmstat_pswpin untyped node_vmstat_pswpin 0 # HELP node_vmstat_pswpout /proc/vmstat information field pswpout. # TYPE node_vmstat_pswpout untyped node_vmstat_pswpout 0 # HELP node_vmstat_slabs_scanned /proc/vmstat information field slabs_scanned. # TYPE node_vmstat_slabs_scanned untyped node_vmstat_slabs_scanned 0 # HELP node_vmstat_thp_collapse_alloc /proc/vmstat information field thp_collapse_alloc. # TYPE node_vmstat_thp_collapse_alloc untyped node_vmstat_thp_collapse_alloc 0 # HELP node_vmstat_thp_collapse_alloc_failed /proc/vmstat information field thp_collapse_alloc_failed. # TYPE node_vmstat_thp_collapse_alloc_failed untyped node_vmstat_thp_collapse_alloc_failed 0 # HELP node_vmstat_thp_fault_alloc /proc/vmstat information field thp_fault_alloc. # TYPE node_vmstat_thp_fault_alloc untyped node_vmstat_thp_fault_alloc 53 # HELP node_vmstat_thp_fault_fallback /proc/vmstat information field thp_fault_fallback. # TYPE node_vmstat_thp_fault_fallback untyped node_vmstat_thp_fault_fallback 0 # HELP node_vmstat_thp_split /proc/vmstat information field thp_split. # TYPE node_vmstat_thp_split untyped node_vmstat_thp_split 1 # HELP node_vmstat_thp_zero_page_alloc /proc/vmstat information field thp_zero_page_alloc. # TYPE node_vmstat_thp_zero_page_alloc untyped node_vmstat_thp_zero_page_alloc 0 # HELP node_vmstat_thp_zero_page_alloc_failed /proc/vmstat information field thp_zero_page_alloc_failed. # TYPE node_vmstat_thp_zero_page_alloc_failed untyped node_vmstat_thp_zero_page_alloc_failed 0 # HELP node_vmstat_unevictable_pgs_cleared /proc/vmstat information field unevictable_pgs_cleared. # TYPE node_vmstat_unevictable_pgs_cleared untyped node_vmstat_unevictable_pgs_cleared 0 # HELP node_vmstat_unevictable_pgs_culled /proc/vmstat information field unevictable_pgs_culled. # TYPE node_vmstat_unevictable_pgs_culled untyped node_vmstat_unevictable_pgs_culled 8348 # HELP node_vmstat_unevictable_pgs_mlocked /proc/vmstat information field unevictable_pgs_mlocked. # TYPE node_vmstat_unevictable_pgs_mlocked untyped node_vmstat_unevictable_pgs_mlocked 11226 # HELP node_vmstat_unevictable_pgs_munlocked /proc/vmstat information field unevictable_pgs_munlocked. # TYPE node_vmstat_unevictable_pgs_munlocked untyped node_vmstat_unevictable_pgs_munlocked 9966 # HELP node_vmstat_unevictable_pgs_rescued /proc/vmstat information field unevictable_pgs_rescued. # TYPE node_vmstat_unevictable_pgs_rescued untyped node_vmstat_unevictable_pgs_rescued 7446 # HELP node_vmstat_unevictable_pgs_scanned /proc/vmstat information field unevictable_pgs_scanned. # TYPE node_vmstat_unevictable_pgs_scanned untyped node_vmstat_unevictable_pgs_scanned 0 # HELP node_vmstat_unevictable_pgs_stranded /proc/vmstat information field unevictable_pgs_stranded. # TYPE node_vmstat_unevictable_pgs_stranded untyped node_vmstat_unevictable_pgs_stranded 0 # HELP node_vmstat_workingset_activate /proc/vmstat information field workingset_activate. # TYPE node_vmstat_workingset_activate untyped node_vmstat_workingset_activate 0 # HELP node_vmstat_workingset_nodereclaim /proc/vmstat information field workingset_nodereclaim. # TYPE node_vmstat_workingset_nodereclaim untyped node_vmstat_workingset_nodereclaim 0 # HELP node_vmstat_workingset_refault /proc/vmstat information field workingset_refault. # TYPE node_vmstat_workingset_refault untyped node_vmstat_workingset_refault 0 # HELP node_vmstat_zone_reclaim_failed /proc/vmstat information field zone_reclaim_failed. # TYPE node_vmstat_zone_reclaim_failed untyped node_vmstat_zone_reclaim_failed 0

percona/mongodb_exporter (mongodb:metrics)

Behind the scenes of the “mongodb:metric” Percona Monitoring and Management service, percona/mongodb_exporter is the Prometheus exporter that provides detailed database metrics for graphing on the Percona Monitoring and Management server. percona/mongodb_exporter is a Percona fork of the project dcu/mongodb_exporter, with some valuable additional metrics for MongoDB sharding, storage engines, etc.

The mongodb_exporter process is designed to automatically detect the MongoDB node type, storage engine, etc., without any special configuration aside from the connection string.

As of Percona Monitoring and Management 1.1.1, here are the number of metrics collected from a single replication-enabled mongodb instance on a single ‘pull’ from Prometheus:

  • Percona Server for MongoDB 3.2 w/WiredTiger: 173 metrics per “pull”
  • Percona Server for MongoDB 3.2 w/RocksDB (with 3 x levels): 239 metrics per “pul”
  • Percona Server for MongoDB 3.2 w/MMAPv1: 172 metrics per “pul”‘

Note: each additional replica set member adds an additional seven metrics to the list of metrics.

On the sharding side of things, a “mongos” process within a cluster with one shard reports 58 metrics, with one extra metric added for each additional cluster shard and 3-4 extra metrics added for each additional “mongos” instance.

Below is a full example of a single “pull” of metrics from one RocksDB instance. Prometheus exporters provide metrics at the HTTP URL: “/metrics”. This is the exact same payload Prometheus would poll from the exporter:

$ curl -sk https://192.168.99.10:42003/metrics | grep mongodb # HELP mongodb_mongod_asserts_total The asserts document reports the number of asserts on the database. While assert errors are typically uncommon, if there are non-zero values for the asserts, you should check the log file for the mongod process for more information. In many cases these errors are trivial, but are worth investigating. # TYPE mongodb_mongod_asserts_total counter mongodb_mongod_asserts_total{type="msg"} 0 mongodb_mongod_asserts_total{type="regular"} 0 mongodb_mongod_asserts_total{type="rollovers"} 0 mongodb_mongod_asserts_total{type="user"} 1 mongodb_mongod_asserts_total{type="warning"} 0 # HELP mongodb_mongod_connections The connections sub document data regarding the current status of incoming connections and availability of the database server. Use these values to assess the current load and capacity requirements of the server # TYPE mongodb_mongod_connections gauge mongodb_mongod_connections{state="available"} 811 mongodb_mongod_connections{state="current"} 8 # HELP mongodb_mongod_connections_metrics_created_total totalCreated provides a count of all incoming connections created to the server. This number includes connections that have since closed # TYPE mongodb_mongod_connections_metrics_created_total counter mongodb_mongod_connections_metrics_created_total 1977 # HELP mongodb_mongod_extra_info_heap_usage_bytes The heap_usage_bytes field is only available on Unix/Linux systems, and reports the total size in bytes of heap space used by the database process # TYPE mongodb_mongod_extra_info_heap_usage_bytes gauge mongodb_mongod_extra_info_heap_usage_bytes 7.77654e+07 # HELP mongodb_mongod_extra_info_page_faults_total The page_faults Reports the total number of page faults that require disk operations. Page faults refer to operations that require the database server to access data which isn’t available in active memory. The page_faults counter may increase dramatically during moments of poor performance and may correlate with limited memory environments and larger data sets. Limited and sporadic page faults do not necessarily indicate an issue # TYPE mongodb_mongod_extra_info_page_faults_total gauge mongodb_mongod_extra_info_page_faults_total 47 # HELP mongodb_mongod_global_lock_client The activeClients data structure provides more granular information about the number of connected clients and the operation types (e.g. read or write) performed by these clients # TYPE mongodb_mongod_global_lock_client gauge mongodb_mongod_global_lock_client{type="reader"} 0 mongodb_mongod_global_lock_client{type="writer"} 0 # HELP mongodb_mongod_global_lock_current_queue The currentQueue data structure value provides more granular information concerning the number of operations queued because of a lock # TYPE mongodb_mongod_global_lock_current_queue gauge mongodb_mongod_global_lock_current_queue{type="reader"} 0 mongodb_mongod_global_lock_current_queue{type="writer"} 0 # HELP mongodb_mongod_global_lock_ratio The value of ratio displays the relationship between lockTime and totalTime. Low values indicate that operations have held the globalLock frequently for shorter periods of time. High values indicate that operations have held globalLock infrequently for longer periods of time # TYPE mongodb_mongod_global_lock_ratio gauge mongodb_mongod_global_lock_ratio 0 # HELP mongodb_mongod_global_lock_total The value of totalTime represents the time, in microseconds, since the database last started and creation of the globalLock. This is roughly equivalent to total server uptime # TYPE mongodb_mongod_global_lock_total counter mongodb_mongod_global_lock_total 0 # HELP mongodb_mongod_instance_local_time The localTime value is the current time, according to the server, in UTC specified in an ISODate format. # TYPE mongodb_mongod_instance_local_time counter mongodb_mongod_instance_local_time 1.488834066e+09 # HELP mongodb_mongod_instance_uptime_estimate_seconds uptimeEstimate provides the uptime as calculated from MongoDB's internal course-grained time keeping system. # TYPE mongodb_mongod_instance_uptime_estimate_seconds counter mongodb_mongod_instance_uptime_estimate_seconds 15881 # HELP mongodb_mongod_instance_uptime_seconds The value of the uptime field corresponds to the number of seconds that the mongos or mongod process has been active. # TYPE mongodb_mongod_instance_uptime_seconds counter mongodb_mongod_instance_uptime_seconds 15881 # HELP mongodb_mongod_locks_time_acquiring_global_microseconds_total amount of time in microseconds that any database has spent waiting for the global lock # TYPE mongodb_mongod_locks_time_acquiring_global_microseconds_total counter mongodb_mongod_locks_time_acquiring_global_microseconds_total{database="Collection",type="read"} 0 mongodb_mongod_locks_time_acquiring_global_microseconds_total{database="Collection",type="write"} 0 mongodb_mongod_locks_time_acquiring_global_microseconds_total{database="Database",type="read"} 0 mongodb_mongod_locks_time_acquiring_global_microseconds_total{database="Database",type="write"} 0 mongodb_mongod_locks_time_acquiring_global_microseconds_total{database="Global",type="read"} 0 mongodb_mongod_locks_time_acquiring_global_microseconds_total{database="Global",type="write"} 5005 mongodb_mongod_locks_time_acquiring_global_microseconds_total{database="Metadata",type="read"} 0 mongodb_mongod_locks_time_acquiring_global_microseconds_total{database="Metadata",type="write"} 0 mongodb_mongod_locks_time_acquiring_global_microseconds_total{database="oplog",type="read"} 0 mongodb_mongod_locks_time_acquiring_global_microseconds_total{database="oplog",type="write"} 0 # HELP mongodb_mongod_locks_time_locked_global_microseconds_total amount of time in microseconds that any database has held the global lock # TYPE mongodb_mongod_locks_time_locked_global_microseconds_total counter mongodb_mongod_locks_time_locked_global_microseconds_total{database="Collection",type="read"} 0 mongodb_mongod_locks_time_locked_global_microseconds_total{database="Collection",type="write"} 0 mongodb_mongod_locks_time_locked_global_microseconds_total{database="Database",type="read"} 0 mongodb_mongod_locks_time_locked_global_microseconds_total{database="Database",type="write"} 0 mongodb_mongod_locks_time_locked_global_microseconds_total{database="Global",type="read"} 0 mongodb_mongod_locks_time_locked_global_microseconds_total{database="Global",type="write"} 0 mongodb_mongod_locks_time_locked_global_microseconds_total{database="Metadata",type="read"} 0 mongodb_mongod_locks_time_locked_global_microseconds_total{database="Metadata",type="write"} 0 mongodb_mongod_locks_time_locked_global_microseconds_total{database="oplog",type="read"} 0 mongodb_mongod_locks_time_locked_global_microseconds_total{database="oplog",type="write"} 0 # HELP mongodb_mongod_locks_time_locked_local_microseconds_total amount of time in microseconds that any database has held the local lock # TYPE mongodb_mongod_locks_time_locked_local_microseconds_total counter mongodb_mongod_locks_time_locked_local_microseconds_total{database="Collection",type="read"} 0 mongodb_mongod_locks_time_locked_local_microseconds_total{database="Collection",type="write"} 0 mongodb_mongod_locks_time_locked_local_microseconds_total{database="Database",type="read"} 0 mongodb_mongod_locks_time_locked_local_microseconds_total{database="Database",type="write"} 0 mongodb_mongod_locks_time_locked_local_microseconds_total{database="Global",type="read"} 0 mongodb_mongod_locks_time_locked_local_microseconds_total{database="Global",type="write"} 0 mongodb_mongod_locks_time_locked_local_microseconds_total{database="Metadata",type="read"} 0 mongodb_mongod_locks_time_locked_local_microseconds_total{database="Metadata",type="write"} 0 mongodb_mongod_locks_time_locked_local_microseconds_total{database="oplog",type="read"} 0 mongodb_mongod_locks_time_locked_local_microseconds_total{database="oplog",type="write"} 0 # HELP mongodb_mongod_memory The mem data structure holds information regarding the target system architecture of mongod and current memory use # TYPE mongodb_mongod_memory gauge mongodb_mongod_memory{type="mapped"} 0 mongodb_mongod_memory{type="mapped_with_journal"} 0 mongodb_mongod_memory{type="resident"} 205 mongodb_mongod_memory{type="virtual"} 1064 # HELP mongodb_mongod_metrics_cursor_open The open is an embedded document that contains data regarding open cursors # TYPE mongodb_mongod_metrics_cursor_open gauge mongodb_mongod_metrics_cursor_open{state="noTimeout"} 0 mongodb_mongod_metrics_cursor_open{state="pinned"} 1 mongodb_mongod_metrics_cursor_open{state="total"} 1 # HELP mongodb_mongod_metrics_cursor_timed_out_total timedOut provides the total number of cursors that have timed out since the server process started. If this number is large or growing at a regular rate, this may indicate an application error # TYPE mongodb_mongod_metrics_cursor_timed_out_total counter mongodb_mongod_metrics_cursor_timed_out_total 0 # HELP mongodb_mongod_metrics_document_total The document holds a document of that reflect document access and modification patterns and data use. Compare these values to the data in the opcounters document, which track total number of operations # TYPE mongodb_mongod_metrics_document_total counter mongodb_mongod_metrics_document_total{state="deleted"} 0 mongodb_mongod_metrics_document_total{state="inserted"} 140396 mongodb_mongod_metrics_document_total{state="returned"} 74105 mongodb_mongod_metrics_document_total{state="updated"} 0 # HELP mongodb_mongod_metrics_get_last_error_wtime_num_total num reports the total number of getLastError operations with a specified write concern (i.e. w) that wait for one or more members of a replica set to acknowledge the write operation (i.e. a w value greater than 1.) # TYPE mongodb_mongod_metrics_get_last_error_wtime_num_total gauge mongodb_mongod_metrics_get_last_error_wtime_num_total 0 # HELP mongodb_mongod_metrics_get_last_error_wtime_total_milliseconds total_millis reports the total amount of time in milliseconds that the mongod has spent performing getLastError operations with write concern (i.e. w) that wait for one or more members of a replica set to acknowledge the write operation (i.e. a w value greater than 1.) # TYPE mongodb_mongod_metrics_get_last_error_wtime_total_milliseconds counter mongodb_mongod_metrics_get_last_error_wtime_total_milliseconds 0 # HELP mongodb_mongod_metrics_get_last_error_wtimeouts_total wtimeouts reports the number of times that write concern operations have timed out as a result of the wtimeout threshold to getLastError. # TYPE mongodb_mongod_metrics_get_last_error_wtimeouts_total counter mongodb_mongod_metrics_get_last_error_wtimeouts_total 0 # HELP mongodb_mongod_metrics_operation_total operation is a sub-document that holds counters for several types of update and query operations that MongoDB handles using special operation types # TYPE mongodb_mongod_metrics_operation_total counter mongodb_mongod_metrics_operation_total{type="fastmod"} 0 mongodb_mongod_metrics_operation_total{type="idhack"} 0 mongodb_mongod_metrics_operation_total{type="scan_and_order"} 0 # HELP mongodb_mongod_metrics_query_executor_total queryExecutor is a document that reports data from the query execution system # TYPE mongodb_mongod_metrics_query_executor_total counter mongodb_mongod_metrics_query_executor_total{state="scanned"} 0 mongodb_mongod_metrics_query_executor_total{state="scanned_objects"} 74105 # HELP mongodb_mongod_metrics_record_moves_total moves reports the total number of times documents move within the on-disk representation of the MongoDB data set. Documents move as a result of operations that increase the size of the document beyond their allocated record size # TYPE mongodb_mongod_metrics_record_moves_total counter mongodb_mongod_metrics_record_moves_total 0 # HELP mongodb_mongod_metrics_repl_apply_batches_num_total num reports the total number of batches applied across all databases # TYPE mongodb_mongod_metrics_repl_apply_batches_num_total counter mongodb_mongod_metrics_repl_apply_batches_num_total 0 # HELP mongodb_mongod_metrics_repl_apply_batches_total_milliseconds total_millis reports the total amount of time the mongod has spent applying operations from the oplog # TYPE mongodb_mongod_metrics_repl_apply_batches_total_milliseconds counter mongodb_mongod_metrics_repl_apply_batches_total_milliseconds 0 # HELP mongodb_mongod_metrics_repl_apply_ops_total ops reports the total number of oplog operations applied # TYPE mongodb_mongod_metrics_repl_apply_ops_total counter mongodb_mongod_metrics_repl_apply_ops_total 0 # HELP mongodb_mongod_metrics_repl_buffer_count count reports the current number of operations in the oplog buffer # TYPE mongodb_mongod_metrics_repl_buffer_count gauge mongodb_mongod_metrics_repl_buffer_count 0 # HELP mongodb_mongod_metrics_repl_buffer_max_size_bytes maxSizeBytes reports the maximum size of the buffer. This value is a constant setting in the mongod, and is not configurable # TYPE mongodb_mongod_metrics_repl_buffer_max_size_bytes counter mongodb_mongod_metrics_repl_buffer_max_size_bytes 2.68435456e+08 # HELP mongodb_mongod_metrics_repl_buffer_size_bytes sizeBytes reports the current size of the contents of the oplog buffer # TYPE mongodb_mongod_metrics_repl_buffer_size_bytes gauge mongodb_mongod_metrics_repl_buffer_size_bytes 0 # HELP mongodb_mongod_metrics_repl_network_bytes_total bytes reports the total amount of data read from the replication sync source # TYPE mongodb_mongod_metrics_repl_network_bytes_total counter mongodb_mongod_metrics_repl_network_bytes_total 0 # HELP mongodb_mongod_metrics_repl_network_getmores_num_total num reports the total number of getmore operations, which are operations that request an additional set of operations from the replication sync source. # TYPE mongodb_mongod_metrics_repl_network_getmores_num_total counter mongodb_mongod_metrics_repl_network_getmores_num_total 0 # HELP mongodb_mongod_metrics_repl_network_getmores_total_milliseconds total_millis reports the total amount of time required to collect data from getmore operations # TYPE mongodb_mongod_metrics_repl_network_getmores_total_milliseconds counter mongodb_mongod_metrics_repl_network_getmores_total_milliseconds 0 # HELP mongodb_mongod_metrics_repl_network_ops_total ops reports the total number of operations read from the replication source. # TYPE mongodb_mongod_metrics_repl_network_ops_total counter mongodb_mongod_metrics_repl_network_ops_total 0 # HELP mongodb_mongod_metrics_repl_network_readers_created_total readersCreated reports the total number of oplog query processes created. MongoDB will create a new oplog query any time an error occurs in the connection, including a timeout, or a network operation. Furthermore, readersCreated will increment every time MongoDB selects a new source fore replication. # TYPE mongodb_mongod_metrics_repl_network_readers_created_total counter mongodb_mongod_metrics_repl_network_readers_created_total 1 # HELP mongodb_mongod_metrics_repl_oplog_insert_bytes_total insertBytes the total size of documents inserted into the oplog. # TYPE mongodb_mongod_metrics_repl_oplog_insert_bytes_total counter mongodb_mongod_metrics_repl_oplog_insert_bytes_total 0 # HELP mongodb_mongod_metrics_repl_oplog_insert_num_total num reports the total number of items inserted into the oplog. # TYPE mongodb_mongod_metrics_repl_oplog_insert_num_total counter mongodb_mongod_metrics_repl_oplog_insert_num_total 0 # HELP mongodb_mongod_metrics_repl_oplog_insert_total_milliseconds total_millis reports the total amount of time spent for the mongod to insert data into the oplog. # TYPE mongodb_mongod_metrics_repl_oplog_insert_total_milliseconds counter mongodb_mongod_metrics_repl_oplog_insert_total_milliseconds 0 # HELP mongodb_mongod_metrics_repl_preload_docs_num_total num reports the total number of documents loaded during the pre-fetch stage of replication # TYPE mongodb_mongod_metrics_repl_preload_docs_num_total counter mongodb_mongod_metrics_repl_preload_docs_num_total 0 # HELP mongodb_mongod_metrics_repl_preload_docs_total_milliseconds total_millis reports the total amount of time spent loading documents as part of the pre-fetch stage of replication # TYPE mongodb_mongod_metrics_repl_preload_docs_total_milliseconds counter mongodb_mongod_metrics_repl_preload_docs_total_milliseconds 0 # HELP mongodb_mongod_metrics_repl_preload_indexes_num_total num reports the total number of index entries loaded by members before updating documents as part of the pre-fetch stage of replication # TYPE mongodb_mongod_metrics_repl_preload_indexes_num_total counter mongodb_mongod_metrics_repl_preload_indexes_num_total 0 # HELP mongodb_mongod_metrics_repl_preload_indexes_total_milliseconds total_millis reports the total amount of time spent loading index entries as part of the pre-fetch stage of replication # TYPE mongodb_mongod_metrics_repl_preload_indexes_total_milliseconds counter mongodb_mongod_metrics_repl_preload_indexes_total_milliseconds 0 # HELP mongodb_mongod_metrics_storage_freelist_search_total metrics about searching records in the database. # TYPE mongodb_mongod_metrics_storage_freelist_search_total counter mongodb_mongod_metrics_storage_freelist_search_total{type="bucket_exhausted"} 0 mongodb_mongod_metrics_storage_freelist_search_total{type="requests"} 0 mongodb_mongod_metrics_storage_freelist_search_total{type="scanned"} 0 # HELP mongodb_mongod_metrics_ttl_deleted_documents_total deletedDocuments reports the total number of documents deleted from collections with a ttl index. # TYPE mongodb_mongod_metrics_ttl_deleted_documents_total counter mongodb_mongod_metrics_ttl_deleted_documents_total 0 # HELP mongodb_mongod_metrics_ttl_passes_total passes reports the number of times the background process removes documents from collections with a ttl index # TYPE mongodb_mongod_metrics_ttl_passes_total counter mongodb_mongod_metrics_ttl_passes_total 0 # HELP mongodb_mongod_network_bytes_total The network data structure contains data regarding MongoDB’s network use # TYPE mongodb_mongod_network_bytes_total counter mongodb_mongod_network_bytes_total{state="in_bytes"} 1.323188842e+09 mongodb_mongod_network_bytes_total{state="out_bytes"} 1.401963024e+09 # HELP mongodb_mongod_network_metrics_num_requests_total The numRequests field is a counter of the total number of distinct requests that the server has received. Use this value to provide context for the bytesIn and bytesOut values to ensure that MongoDB’s network utilization is consistent with expectations and application use # TYPE mongodb_mongod_network_metrics_num_requests_total counter mongodb_mongod_network_metrics_num_requests_total 313023 # HELP mongodb_mongod_op_counters_repl_total The opcountersRepl data structure, similar to the opcounters data structure, provides an overview of database replication operations by type and makes it possible to analyze the load on the replica in more granular manner. These values only appear when the current host has replication enabled # TYPE mongodb_mongod_op_counters_repl_total counter mongodb_mongod_op_counters_repl_total{type="command"} 0 mongodb_mongod_op_counters_repl_total{type="delete"} 0 mongodb_mongod_op_counters_repl_total{type="getmore"} 0 mongodb_mongod_op_counters_repl_total{type="insert"} 0 mongodb_mongod_op_counters_repl_total{type="query"} 0 mongodb_mongod_op_counters_repl_total{type="update"} 0 # HELP mongodb_mongod_op_counters_total The opcounters data structure provides an overview of database operations by type and makes it possible to analyze the load on the database in more granular manner. These numbers will grow over time and in response to database use. Analyze these values over time to track database utilization # TYPE mongodb_mongod_op_counters_total counter mongodb_mongod_op_counters_total{type="command"} 164008 mongodb_mongod_op_counters_total{type="delete"} 0 mongodb_mongod_op_counters_total{type="getmore"} 74912 mongodb_mongod_op_counters_total{type="insert"} 70198 mongodb_mongod_op_counters_total{type="query"} 3907 mongodb_mongod_op_counters_total{type="update"} 0 # HELP mongodb_mongod_replset_heatbeat_interval_millis The frequency in milliseconds of the heartbeats # TYPE mongodb_mongod_replset_heatbeat_interval_millis gauge mongodb_mongod_replset_heatbeat_interval_millis{set="test1"} 2000 # HELP mongodb_mongod_replset_member_config_version The configVersion value is the replica set configuration version. # TYPE mongodb_mongod_replset_member_config_version gauge mongodb_mongod_replset_member_config_version{name="localhost:27017",set="test1",state="PRIMARY"} 2 mongodb_mongod_replset_member_config_version{name="localhost:27027",set="test1",state="SECONDARY"} 2 # HELP mongodb_mongod_replset_member_election_date The timestamp the node was elected as replica leader # TYPE mongodb_mongod_replset_member_election_date gauge mongodb_mongod_replset_member_election_date{name="localhost:27017",set="test1",state="PRIMARY"} 1.488818198e+09 # HELP mongodb_mongod_replset_member_health This field conveys if the member is up (1) or down (0). # TYPE mongodb_mongod_replset_member_health gauge mongodb_mongod_replset_member_health{name="localhost:27017",set="test1",state="PRIMARY"} 1 mongodb_mongod_replset_member_health{name="localhost:27027",set="test1",state="SECONDARY"} 1 # HELP mongodb_mongod_replset_member_last_heartbeat The lastHeartbeat value provides an ISODate formatted date and time of the transmission time of last heartbeat received from this member # TYPE mongodb_mongod_replset_member_last_heartbeat gauge mongodb_mongod_replset_member_last_heartbeat{name="localhost:27027",set="test1",state="SECONDARY"} 1.488834065e+09 # HELP mongodb_mongod_replset_member_last_heartbeat_recv The lastHeartbeatRecv value provides an ISODate formatted date and time that the last heartbeat was received from this member # TYPE mongodb_mongod_replset_member_last_heartbeat_recv gauge mongodb_mongod_replset_member_last_heartbeat_recv{name="localhost:27027",set="test1",state="SECONDARY"} 1.488834066e+09 # HELP mongodb_mongod_replset_member_optime_date The timestamp of the last oplog entry that this member applied. # TYPE mongodb_mongod_replset_member_optime_date gauge mongodb_mongod_replset_member_optime_date{name="localhost:27017",set="test1",state="PRIMARY"} 1.488833953e+09 mongodb_mongod_replset_member_optime_date{name="localhost:27027",set="test1",state="SECONDARY"} 1.488833953e+09 # HELP mongodb_mongod_replset_member_ping_ms The pingMs represents the number of milliseconds (ms) that a round-trip packet takes to travel between the remote member and the local instance. # TYPE mongodb_mongod_replset_member_ping_ms gauge mongodb_mongod_replset_member_ping_ms{name="localhost:27027",set="test1",state="SECONDARY"} 0 # HELP mongodb_mongod_replset_member_state The value of state is an integer between 0 and 10 that represents the replica state of the member. # TYPE mongodb_mongod_replset_member_state gauge mongodb_mongod_replset_member_state{name="localhost:27017",set="test1",state="PRIMARY"} 1 mongodb_mongod_replset_member_state{name="localhost:27027",set="test1",state="SECONDARY"} 2 # HELP mongodb_mongod_replset_member_uptime The uptime field holds a value that reflects the number of seconds that this member has been online. # TYPE mongodb_mongod_replset_member_uptime counter mongodb_mongod_replset_member_uptime{name="localhost:27017",set="test1",state="PRIMARY"} 15881 mongodb_mongod_replset_member_uptime{name="localhost:27027",set="test1",state="SECONDARY"} 15855 # HELP mongodb_mongod_replset_my_name The replica state name of the current member # TYPE mongodb_mongod_replset_my_name gauge mongodb_mongod_replset_my_name{name="localhost:27017",set="test1"} 1 # HELP mongodb_mongod_replset_my_state An integer between 0 and 10 that represents the replica state of the current member # TYPE mongodb_mongod_replset_my_state gauge mongodb_mongod_replset_my_state{set="test1"} 1 # HELP mongodb_mongod_replset_number_of_members The number of replica set mebers # TYPE mongodb_mongod_replset_number_of_members gauge mongodb_mongod_replset_number_of_members{set="test1"} 2 # HELP mongodb_mongod_replset_oplog_head_timestamp The timestamp of the newest change in the oplog # TYPE mongodb_mongod_replset_oplog_head_timestamp gauge mongodb_mongod_replset_oplog_head_timestamp 1.488833953e+09 # HELP mongodb_mongod_replset_oplog_items_total The total number of changes in the oplog # TYPE mongodb_mongod_replset_oplog_items_total gauge mongodb_mongod_replset_oplog_items_total 70202 # HELP mongodb_mongod_replset_oplog_size_bytes Size of oplog in bytes # TYPE mongodb_mongod_replset_oplog_size_bytes gauge mongodb_mongod_replset_oplog_size_bytes{type="current"} 1.273140034e+09 mongodb_mongod_replset_oplog_size_bytes{type="storage"} 1.273139968e+09 # HELP mongodb_mongod_replset_oplog_tail_timestamp The timestamp of the oldest change in the oplog # TYPE mongodb_mongod_replset_oplog_tail_timestamp gauge mongodb_mongod_replset_oplog_tail_timestamp 1.488818198e+09 # HELP mongodb_mongod_replset_term The election count for the replica set, as known to this replica set member # TYPE mongodb_mongod_replset_term gauge mongodb_mongod_replset_term{set="test1"} 1 # HELP mongodb_mongod_rocksdb_background_errors The total number of background errors in RocksDB # TYPE mongodb_mongod_rocksdb_background_errors gauge mongodb_mongod_rocksdb_background_errors 0 # HELP mongodb_mongod_rocksdb_block_cache_bytes The current bytes used in the RocksDB Block Cache # TYPE mongodb_mongod_rocksdb_block_cache_bytes gauge mongodb_mongod_rocksdb_block_cache_bytes 2.5165824e+07 # HELP mongodb_mongod_rocksdb_block_cache_hits_total The total number of hits to the RocksDB Block Cache # TYPE mongodb_mongod_rocksdb_block_cache_hits_total counter mongodb_mongod_rocksdb_block_cache_hits_total 519483 # HELP mongodb_mongod_rocksdb_block_cache_misses_total The total number of misses to the RocksDB Block Cache # TYPE mongodb_mongod_rocksdb_block_cache_misses_total counter mongodb_mongod_rocksdb_block_cache_misses_total 219214 # HELP mongodb_mongod_rocksdb_bloom_filter_useful_total The total number of times the RocksDB Bloom Filter was useful # TYPE mongodb_mongod_rocksdb_bloom_filter_useful_total counter mongodb_mongod_rocksdb_bloom_filter_useful_total 130251 # HELP mongodb_mongod_rocksdb_bytes_read_total The total number of bytes read by RocksDB # TYPE mongodb_mongod_rocksdb_bytes_read_total counter mongodb_mongod_rocksdb_bytes_read_total{type="compation"} 7.917332498e+09 mongodb_mongod_rocksdb_bytes_read_total{type="iteration"} 3.94838428e+09 mongodb_mongod_rocksdb_bytes_read_total{type="point_lookup"} 2.655612381e+09 # HELP mongodb_mongod_rocksdb_bytes_written_total The total number of bytes written by RocksDB # TYPE mongodb_mongod_rocksdb_bytes_written_total counter mongodb_mongod_rocksdb_bytes_written_total{type="compaction"} 9.531638929e+09 mongodb_mongod_rocksdb_bytes_written_total{type="flush"} 2.525874187e+09 mongodb_mongod_rocksdb_bytes_written_total{type="total"} 2.548843861e+09 # HELP mongodb_mongod_rocksdb_compaction_average_seconds The average time per compaction between levels N and N+1 in RocksDB # TYPE mongodb_mongod_rocksdb_compaction_average_seconds gauge mongodb_mongod_rocksdb_compaction_average_seconds{level="L0"} 0.505 mongodb_mongod_rocksdb_compaction_average_seconds{level="L5"} 3.606 mongodb_mongod_rocksdb_compaction_average_seconds{level="L6"} 14.579 mongodb_mongod_rocksdb_compaction_average_seconds{level="total"} 4.583 # HELP mongodb_mongod_rocksdb_compaction_bytes_per_second The rate at which data is processed during compaction between levels N and N+1 in RocksDB # TYPE mongodb_mongod_rocksdb_compaction_bytes_per_second gauge mongodb_mongod_rocksdb_compaction_bytes_per_second{level="L0",type="write"} 1.248854016e+08 mongodb_mongod_rocksdb_compaction_bytes_per_second{level="L5",type="read"} 7.45537536e+07 mongodb_mongod_rocksdb_compaction_bytes_per_second{level="L5",type="write"} 7.45537536e+07 mongodb_mongod_rocksdb_compaction_bytes_per_second{level="L6",type="read"} 2.44318208e+07 mongodb_mongod_rocksdb_compaction_bytes_per_second{level="L6",type="write"} 2.06569472e+07 mongodb_mongod_rocksdb_compaction_bytes_per_second{level="total",type="read"} 2.70532608e+07 mongodb_mongod_rocksdb_compaction_bytes_per_second{level="total",type="write"} 3.2505856e+07 # HELP mongodb_mongod_rocksdb_compaction_file_threads The number of threads currently doing compaction for levels in RocksDB # TYPE mongodb_mongod_rocksdb_compaction_file_threads gauge mongodb_mongod_rocksdb_compaction_file_threads{level="L0"} 0 mongodb_mongod_rocksdb_compaction_file_threads{level="L5"} 0 mongodb_mongod_rocksdb_compaction_file_threads{level="L6"} 0 mongodb_mongod_rocksdb_compaction_file_threads{level="total"} 0 # HELP mongodb_mongod_rocksdb_compaction_score The compaction score of RocksDB levels # TYPE mongodb_mongod_rocksdb_compaction_score gauge mongodb_mongod_rocksdb_compaction_score{level="L0"} 0 mongodb_mongod_rocksdb_compaction_score{level="L5"} 0.9 mongodb_mongod_rocksdb_compaction_score{level="L6"} 0 mongodb_mongod_rocksdb_compaction_score{level="total"} 0 # HELP mongodb_mongod_rocksdb_compaction_seconds_total The time spent doing compactions between levels N and N+1 in RocksDB # TYPE mongodb_mongod_rocksdb_compaction_seconds_total counter mongodb_mongod_rocksdb_compaction_seconds_total{level="L0"} 20 mongodb_mongod_rocksdb_compaction_seconds_total{level="L5"} 25 mongodb_mongod_rocksdb_compaction_seconds_total{level="L6"} 248 mongodb_mongod_rocksdb_compaction_seconds_total{level="total"} 293 # HELP mongodb_mongod_rocksdb_compaction_write_amplification The write amplification factor from compaction between levels N and N+1 in RocksDB # TYPE mongodb_mongod_rocksdb_compaction_write_amplification gauge mongodb_mongod_rocksdb_compaction_write_amplification{level="L5"} 1 mongodb_mongod_rocksdb_compaction_write_amplification{level="L6"} 1.7 mongodb_mongod_rocksdb_compaction_write_amplification{level="total"} 3.8 # HELP mongodb_mongod_rocksdb_compactions_total The total number of compactions between levels N and N+1 in RocksDB # TYPE mongodb_mongod_rocksdb_compactions_total counter mongodb_mongod_rocksdb_compactions_total{level="L0"} 40 mongodb_mongod_rocksdb_compactions_total{level="L5"} 7 mongodb_mongod_rocksdb_compactions_total{level="L6"} 17 mongodb_mongod_rocksdb_compactions_total{level="total"} 64 # HELP mongodb_mongod_rocksdb_estimate_table_readers_memory_bytes The estimate RocksDB table-reader memory bytes # TYPE mongodb_mongod_rocksdb_estimate_table_readers_memory_bytes gauge mongodb_mongod_rocksdb_estimate_table_readers_memory_bytes 2.10944e+06 # HELP mongodb_mongod_rocksdb_files The number of files in a RocksDB level # TYPE mongodb_mongod_rocksdb_files gauge mongodb_mongod_rocksdb_files{level="L0"} 0 mongodb_mongod_rocksdb_files{level="L5"} 2 mongodb_mongod_rocksdb_files{level="L6"} 23 mongodb_mongod_rocksdb_files{level="total"} 25 # HELP mongodb_mongod_rocksdb_immutable_memtables The total number of immutable MemTables in RocksDB # TYPE mongodb_mongod_rocksdb_immutable_memtables gauge mongodb_mongod_rocksdb_immutable_memtables 0 # HELP mongodb_mongod_rocksdb_iterations_total The total number of iterations performed by RocksDB # TYPE mongodb_mongod_rocksdb_iterations_total counter mongodb_mongod_rocksdb_iterations_total{type="backward"} 1955 mongodb_mongod_rocksdb_iterations_total{type="forward"} 206563 # HELP mongodb_mongod_rocksdb_keys_total The total number of RocksDB key operations # TYPE mongodb_mongod_rocksdb_keys_total counter mongodb_mongod_rocksdb_keys_total{type="read"} 209546 mongodb_mongod_rocksdb_keys_total{type="written"} 281332 # HELP mongodb_mongod_rocksdb_live_versions The current number of live versions in RocksDB # TYPE mongodb_mongod_rocksdb_live_versions gauge mongodb_mongod_rocksdb_live_versions 1 # HELP mongodb_mongod_rocksdb_memtable_bytes The current number of MemTable bytes in RocksDB # TYPE mongodb_mongod_rocksdb_memtable_bytes gauge mongodb_mongod_rocksdb_memtable_bytes{type="active"} 2.5165824e+07 mongodb_mongod_rocksdb_memtable_bytes{type="total"} 2.5165824e+07 # HELP mongodb_mongod_rocksdb_memtable_entries The current number of Memtable entries in RocksDB # TYPE mongodb_mongod_rocksdb_memtable_entries gauge mongodb_mongod_rocksdb_memtable_entries{type="active"} 4553 mongodb_mongod_rocksdb_memtable_entries{type="immutable"} 0 # HELP mongodb_mongod_rocksdb_oldest_snapshot_timestamp The timestamp of the oldest snapshot in RocksDB # TYPE mongodb_mongod_rocksdb_oldest_snapshot_timestamp gauge mongodb_mongod_rocksdb_oldest_snapshot_timestamp 0 # HELP mongodb_mongod_rocksdb_pending_compactions The total number of compactions pending in RocksDB # TYPE mongodb_mongod_rocksdb_pending_compactions gauge mongodb_mongod_rocksdb_pending_compactions 0 # HELP mongodb_mongod_rocksdb_pending_memtable_flushes The total number of MemTable flushes pending in RocksDB # TYPE mongodb_mongod_rocksdb_pending_memtable_flushes gauge mongodb_mongod_rocksdb_pending_memtable_flushes 0 # HELP mongodb_mongod_rocksdb_read_latency_microseconds The read latency in RocksDB in microseconds by level # TYPE mongodb_mongod_rocksdb_read_latency_microseconds gauge mongodb_mongod_rocksdb_read_latency_microseconds{level="L0",type="P50"} 9.33 mongodb_mongod_rocksdb_read_latency_microseconds{level="L0",type="P75"} 17.03 mongodb_mongod_rocksdb_read_latency_microseconds{level="L0",type="P99"} 1972.31 mongodb_mongod_rocksdb_read_latency_microseconds{level="L0",type="P99.9"} 9061.44 mongodb_mongod_rocksdb_read_latency_microseconds{level="L0",type="P99.99"} 18098.3 mongodb_mongod_rocksdb_read_latency_microseconds{level="L0",type="avg"} 87.3861 mongodb_mongod_rocksdb_read_latency_microseconds{level="L0",type="max"} 26434 mongodb_mongod_rocksdb_read_latency_microseconds{level="L0",type="median"} 9.3328 mongodb_mongod_rocksdb_read_latency_microseconds{level="L0",type="min"} 0 mongodb_mongod_rocksdb_read_latency_microseconds{level="L0",type="stddev"} 607.16 mongodb_mongod_rocksdb_read_latency_microseconds{level="L5",type="P50"} 10.85 mongodb_mongod_rocksdb_read_latency_microseconds{level="L5",type="P75"} 28.46 mongodb_mongod_rocksdb_read_latency_microseconds{level="L5",type="P99"} 4208.33 mongodb_mongod_rocksdb_read_latency_microseconds{level="L5",type="P99.9"} 13079.55 mongodb_mongod_rocksdb_read_latency_microseconds{level="L5",type="P99.99"} 23156.25 mongodb_mongod_rocksdb_read_latency_microseconds{level="L5",type="avg"} 216.3736 mongodb_mongod_rocksdb_read_latency_microseconds{level="L5",type="max"} 34585 mongodb_mongod_rocksdb_read_latency_microseconds{level="L5",type="median"} 10.8461 mongodb_mongod_rocksdb_read_latency_microseconds{level="L5",type="min"} 0 mongodb_mongod_rocksdb_read_latency_microseconds{level="L5",type="stddev"} 978.48 mongodb_mongod_rocksdb_read_latency_microseconds{level="L6",type="P50"} 9.42 mongodb_mongod_rocksdb_read_latency_microseconds{level="L6",type="P75"} 30.69 mongodb_mongod_rocksdb_read_latency_microseconds{level="L6",type="P99"} 6037.87 mongodb_mongod_rocksdb_read_latency_microseconds{level="L6",type="P99.9"} 15482.63 mongodb_mongod_rocksdb_read_latency_microseconds{level="L6",type="P99.99"} 30125.5 mongodb_mongod_rocksdb_read_latency_microseconds{level="L6",type="avg"} 294.9574 mongodb_mongod_rocksdb_read_latency_microseconds{level="L6",type="max"} 61051 mongodb_mongod_rocksdb_read_latency_microseconds{level="L6",type="median"} 9.4245 mongodb_mongod_rocksdb_read_latency_microseconds{level="L6",type="min"} 0 mongodb_mongod_rocksdb_read_latency_microseconds{level="L6",type="stddev"} 1264.12 # HELP mongodb_mongod_rocksdb_reads_total The total number of read operations in RocksDB # TYPE mongodb_mongod_rocksdb_reads_total counter mongodb_mongod_rocksdb_reads_total{level="L0"} 59017 mongodb_mongod_rocksdb_reads_total{level="L5"} 42125 mongodb_mongod_rocksdb_reads_total{level="L6"} 118745 # HELP mongodb_mongod_rocksdb_seeks_total The total number of seeks performed by RocksDB # TYPE mongodb_mongod_rocksdb_seeks_total counter mongodb_mongod_rocksdb_seeks_total 156184 # HELP mongodb_mongod_rocksdb_size_bytes The total byte size of levels in RocksDB # TYPE mongodb_mongod_rocksdb_size_bytes gauge mongodb_mongod_rocksdb_size_bytes{level="L0"} 0 mongodb_mongod_rocksdb_size_bytes{level="L5"} 1.2969836544e+08 mongodb_mongod_rocksdb_size_bytes{level="L6"} 1.46175688704e+09 mongodb_mongod_rocksdb_size_bytes{level="total"} 1.59145525248e+09 # HELP mongodb_mongod_rocksdb_snapshots The current number of snapshots in RocksDB # TYPE mongodb_mongod_rocksdb_snapshots gauge mongodb_mongod_rocksdb_snapshots 0 # HELP mongodb_mongod_rocksdb_stall_percent The percentage of time RocksDB has been stalled # TYPE mongodb_mongod_rocksdb_stall_percent gauge mongodb_mongod_rocksdb_stall_percent 0 # HELP mongodb_mongod_rocksdb_stalled_seconds_total The total number of seconds RocksDB has spent stalled # TYPE mongodb_mongod_rocksdb_stalled_seconds_total counter mongodb_mongod_rocksdb_stalled_seconds_total 0 # HELP mongodb_mongod_rocksdb_stalls_total The total number of stalls in RocksDB # TYPE mongodb_mongod_rocksdb_stalls_total counter mongodb_mongod_rocksdb_stalls_total{type="level0_numfiles"} 0 mongodb_mongod_rocksdb_stalls_total{type="level0_numfiles_with_compaction"} 0 mongodb_mongod_rocksdb_stalls_total{type="level0_slowdown"} 0 mongodb_mongod_rocksdb_stalls_total{type="level0_slowdown_with_compaction"} 0 mongodb_mongod_rocksdb_stalls_total{type="memtable_compaction"} 0 mongodb_mongod_rocksdb_stalls_total{type="memtable_slowdown"} 0 # HELP mongodb_mongod_rocksdb_total_live_recovery_units The total number of live recovery units in RocksDB # TYPE mongodb_mongod_rocksdb_total_live_recovery_units gauge mongodb_mongod_rocksdb_total_live_recovery_units 5 # HELP mongodb_mongod_rocksdb_transaction_engine_keys The current number of transaction engine keys in RocksDB # TYPE mongodb_mongod_rocksdb_transaction_engine_keys gauge mongodb_mongod_rocksdb_transaction_engine_keys 0 # HELP mongodb_mongod_rocksdb_transaction_engine_snapshots The current number of transaction engine snapshots in RocksDB # TYPE mongodb_mongod_rocksdb_transaction_engine_snapshots gauge mongodb_mongod_rocksdb_transaction_engine_snapshots 0 # HELP mongodb_mongod_rocksdb_write_ahead_log_bytes_per_second The number of bytes written per second by the Write-Ahead-Log in RocksDB # TYPE mongodb_mongod_rocksdb_write_ahead_log_bytes_per_second gauge mongodb_mongod_rocksdb_write_ahead_log_bytes_per_second 157286.4 # HELP mongodb_mongod_rocksdb_write_ahead_log_writes_per_sync The number of writes per Write-Ahead-Log sync in RocksDB # TYPE mongodb_mongod_rocksdb_write_ahead_log_writes_per_sync gauge mongodb_mongod_rocksdb_write_ahead_log_writes_per_sync 70538 # HELP mongodb_mongod_rocksdb_writes_per_batch The number of writes per batch in RocksDB # TYPE mongodb_mongod_rocksdb_writes_per_batch gauge mongodb_mongod_rocksdb_writes_per_batch 2.54476812288e+09 # HELP mongodb_mongod_rocksdb_writes_per_second The number of writes per second in RocksDB # TYPE mongodb_mongod_rocksdb_writes_per_second gauge mongodb_mongod_rocksdb_writes_per_second 157286.4 # HELP mongodb_mongod_storage_engine The storage engine used by the MongoDB instance # TYPE mongodb_mongod_storage_engine counter mongodb_mongod_storage_engine{engine="rocksdb"} 1

Grafana

PMM uses Grafana to visualize metrics stored in Prometheus. Grafana uses a concept of “dashboards” to store the definitions of what it should visualize. PMM’s dashboards are hosted under the GitHub project: percona/grafana-dashboards.

Grafana can support multiple “data sources” for metrics. As Percona Monitoring and Management uses Prometheus for metric storage, this is the “data source” used in Grafana.

Adding Custom Graphs to Grafana

In this section I will create an example graph to help indicate the “efficiency” of queries on a Mongod instance over the last 5 minutes. This is a good example to use because we plan to add this exact metric in an upcoming version of PMM.

This graph will rely on two metrics that are already provided to PMM via percona/mongodb_exporter since at least version 1.0.0 (“$host” = the hostname of a given node – explained later):

  1. mongodb_mongod_metrics_query_executor_total{instance=”$host”, state=”scanned_objects”} – A metric representing the total number of documents scanned by the server.
  2. mongodb_mongod_metrics_document_total{instance=”$host”, state=”returned”} – A metric representing the total number of documents returned by the server.

The graph will compute the change in these two metrics over five minutes and create a percentage/ratio of scanned vs. returned documents. A host with ten scanned documents and one document returned would have 10% efficiency and a host that scanned 100 documents to return 100 documents would have 100% efficiency (a 1:1 ratio).

Often you will encounter Prometheus metrics that are “total” metric counters that increment from the time the server is (re)started. Both of the metrics our new graph requires are incremented counters and thus need to be “scaled” or “trimmed” to only show the last five minutes of metrics (in this example), not the total since the server was (re)started.

Prometheus offers a very useful query function for incremented counters called “increase()”: https://prometheus.io/docs/querying/functions/#increase(). The Prometheus increase() function allows queries to return the amount a metric counter has increased over a given time period, making this trivial to do! It is also unaffected by counter “resets” due to server restarts as increase() only returns increases in counters.

The increase() syntax requires a time range to be specified before the closing round-bracket. In our case we will as increase() to consider the last five minutes, which is expressed with “[5m]” at the end of the increase() function, seen in the following example.

The full Prometheus query I will use to create query efficiency graph is:

sum( increase(mongodb_mongod_metrics_query_executor_total{instance="$host", state="scanned_objects"}[5m]) ) / sum( increase(mongodb_mongod_metrics_document_total{instance="$host", state="returned"}[5m]) )

Note: sum() is required around the increase() functions when dividing two numbers in Prometheus queries.

Now, let’s make this a new graph! To do this you can create a new dashboard in PMM’s Grafana or add to an existing dashboard. In this example I’ll create a new dashboard with a single graph.

To add a new dashboard, press the Dashboard selector in PMM’s Grafana and select “Create New”:

This will create a new dashboard named “New Dashboard”.

Most of PMM’s graphing uses a variable named “$host” in place of a hostname/IP. You’ll notice “$host” was used in the “query efficiency” Prometheus query earlier. The variable is set using a Grafana feature called Templating.

Let’s add a “$host” variable to our new dashboard so we can change what host we graph without modifying our queries. First, press the gear icon at the top of the dashboard and select “Templating”:

Then press “New” to create a new Templating variable.

Set “Name” to be host, set “Data Source” to Prometheus and set “Query” to label_values(instance). Leave all other settings default:

Press “Add” to add the template variable, then save and reload the dashboard.

This will add a drop-down of unique hosts in Prometheus like this:

On the first dashboard row let’s add a new graph by opening the row menu on the far left of the first row and then select “Add Panel”:

Select “Graph” as the type. Click the title of the blank graph to open a menu, press “Edit”:

This opens Grafana’s graph editor with an empty graph and some input boxes seen below.

Next, let’s add our “query efficiency” Prometheus query (earlier in this article) to the “Query” input field and add a legend name to “Legend format”:

Now we have some graph data, but the Y-axis and title don’t explain very much about the metric. What does “1.0K” on the Y-Axis mean?

As our metric is a ratio, let’s display the Y-axis as a percentage by switching to the “Axes” tab, then selecting “percent (0.0-1.0)” as the “Unit” selection for the “Left Y” axis, like so:

Next let’s set a graph title. To do this, go to the “General” tab of the graph editor and set the “Title” field:

And now we have a “Query Efficiency” graph with an accurate Y-axis and Title(!):

“Back to Dashboard” on the top-right will take you back to the dashboard view. Always remember to save your dashboards after making changes!

Adding Custom Metrics to percona/mongodb_exporter

For those familiar with the Go programming language, adding custom metrics to percona/mongodb_exporter is fairly straightforward. The percona/mongodb_exporter uses the Prometheus Go client to export metrics gathered from queries to MongoDB.

Adding a completely new metric to the exporter is unfortunately too open-ended to explain in a blog. Instead, I will cover how an existing metric is exported by percona/mongodb_exporter. The process for a new metric will be similar.

To follow our previous example, here is an simplified example of how the metric: “mongodb_mongod_metrics_query_executor_total” is exported via percona/mongodb_exporter. This source of this metric is “db.serverStatus().metrics.queryExecutor” from MongoDB shell perspective.

First a new Prometheus metric is defined as a go ‘var’:

  1. var ( metricsQueryExecutorTotal = prometheus.NewCounterVec(prometheus.CounterOpts{ Namespace: Namespace, Name: "metrics_query_executor_total", Help: "queryExecutor is a document that reports data from the query execution system", }, []string{"state"}) )
    https://github.com/percona/mongodb_exporter/blob/master/collector/mongod/metrics.go#L58-L64
  2. A struct for marshaling the BSON response from MongoDB and an “.Export()” function is defined for the struct (.Export() is called on the metric structs):
    // QueryExecutorStats are the stats associated with a query execution. type QueryExecutorStats struct { Scanned float64 `bson:"scanned"` ScannedObjects float64 `bson:"scannedObjects"` } // Export exports the query executor stats. func (queryExecutorStats *QueryExecutorStats) Export(ch chan<- prometheus.Metric) { metricsQueryExecutorTotal.WithLabelValues("scanned").Set(queryExecutorStats.Scanned) metricsQueryExecutorTotal.WithLabelValues("scanned_objects").Set(queryExecutorStats.ScannedObjects) }
    Notice that the float64 values unmarshaled from the BSON are used in the .Set() for the Prometheus metric. All Prometheus values must be float64.
    https://github.com/percona/mongodb_exporter/blob/master/collector/mongod/metrics.go#L271-L281
  3. In this case the “QueryExecutorStats” is a sub-struct of a larger “MetricStats” struct above it, also with its own “.Export()” function:
    // MetricsStats are all stats associated with metrics of the system type MetricsStats struct { Document *DocumentStats `bson:"document"` GetLastError *GetLastErrorStats `bson:"getLastError"` Operation *OperationStats `bson:"operation"` QueryExecutor *QueryExecutorStats `bson:"queryExecutor"` Record *RecordStats `bson:"record"` Repl *ReplStats `bson:"repl"` Storage *StorageStats `bson:"storage"` Cursor *CursorStats `bson:"cursor"` } // Export exports the metrics stats. func (metricsStats *MetricsStats) Export(ch chan<- prometheus.Metric) { if metricsStats.Document != nil { metricsStats.Document.Export(ch) } if metricsStats.GetLastError != nil { metricsStats.GetLastError.Export(ch) } if metricsStats.Operation != nil { metricsStats.Operation.Export(ch) } if metricsStats.QueryExecutor != nil { metricsStats.QueryExecutor.Export(ch) } ...
    https://github.com/percona/mongodb_exporter/blob/master/collector/mongod/metrics.go#L405-L430
  4. Finally a .Collect() and  .Describe()” (also required functions) is called on the metric to collect and describe it:
    https://github.com/percona/mongodb_exporter/blob/master/collector/mongod/metrics.go#L451
    https://github.com/percona/mongodb_exporter/blob/master/collector/mongod/metrics.go#L485
  5. Later on in this code, “MetricStats” is passed the result of the query “db.serverStatus().metrics”. This can be seen at:
    https://github.com/percona/mongodb_exporter/blob/master/collector/mongod/server_status.go#L60
    https://github.com/percona/mongodb_exporter/blob/master/collector/mongod/server_status.go#L177-L179
    and
    https://github.com/percona/mongodb_exporter/blob/master/collector/mongod/server_status.go#L197-L207

For those unfamiliar with Go and/or unable to contribute new metrics to the project, we suggest you open a JIRA ticket for any feature requests for new metrics here: https://jira.percona.com/projects/PMM.

Conclusion

With the flexibility of the monitoring components of Percona Monitoring and Management, the sky is the limit on what can be done with database monitoring! Hopefully this blog gives you a taste of what is possible if you need to add a new graph, a new metric or both to Percona Monitoring and Management. Also, it is worth repeating that a large number of metrics gathered in Percona Monitoring and Management are not graphed. Perhaps what you’re looking for is already collected. See “http://<pmm-server>/prometheus” for more details on what metrics are stored in Prometheus.

We are always open to improving our dashboards, and we would love to hear about any custom graphs you create and how they help solve problems!

Percona Server for MongoDB 3.2.12-3.2 is now available

March 9, 2017 - 10:52am

Percona announces the release of Percona Server for MongoDB 3.2.12-3.2 on March 9, 2017. Download the latest version from the Percona web site or the Percona Software Repositories.

Percona Server for MongoDB 3.2.11-3.1 is an enhanced, open-source, fully compatible, highly scalable, zero-maintenance downtime database supporting the MongoDB v3.2 protocol and drivers. It extends MongoDB with MongoRocks, Percona Memory Engine, and PerconaFT storage engine, as well as enterprise-grade features like External Authentication, Audit Logging, Profiling Rate Limiting, and Hot Backup at no extra cost. Percona Server for MongoDB requires no changes to MongoDB applications or code.

NOTE: We deprecated the PerconaFT storage engine. It will not be available in future releases.

This release is based on MongoDB 3.2.12 and includes the following additional changes:

  • PSMDB-17: Changed welcome message in the shell to mention Percona Server for MongoDB instead of MongoDB
  • PSMDB-90: Added error message for storage engines that do not support Hot Backup
  • PSMDB-91: Deprecated audit configuration section and added auditLog instead
  • PSMDB-95: Fixed version dependencies for sub packages so that all corresponding packages get updated accordingly
  • PSMDB-96: Excluded diagnostic.data directory when using TokuBackup with PerconaFT
  • PSMDB-98: Improved Hot Backup to create destination folder if it does not exist
  • PSMDB-101: Implemented the auditAuthorizationSuccess parameter to enable auditing of authorization success
  • PSMDB-104: Updated links in client shell output to point to Percona’s documentation and forum
  • PSMDB-107: Fixed behavior when creating the audit log file
  • PSMDB-111: Refactored external_auth tests
  • PSMDB-123: Fixed the creation of proper subdirectories inside the backup destination directory
  • PSMDB-126: Added index and collection name to duplicate key error message
  • Fixed startup scripts for Ubuntu 14.04.5 LTS (Trusty Tahr)
  • Fixed a number of other small issues and bugs

Percona Server for MongoDB 3.2.12-3.2 release notes are available in the official documentation.

Services Monitoring with Probabilistic Fault Detection

March 9, 2017 - 10:19am

In this blog post, we’ll discuss services monitoring using probabilistic fault detection.

Let’s admit it, the task of monitoring services is one of the most difficult. It is time-consuming, error-prone and difficult to automate. The usual monitoring approach has been pretty straightforward in the last few years: setup a service like Nagios, or pay money to get a cloud-based monitoring tool. Then choose the metrics you are interested in and set the thresholds. This is a manual process that works when you have a small number of services and servers, and you know exactly how they behave and what you should monitor. These days, we have hundred of servers with thousands of services sending us millions of metrics. That is the first problem: the manual approach to configuration doesn’t work.

That is not the only problem. We know that no two servers perform the same because no two servers have exactly the same workload. The thresholds that you setup for one server might not be the correct one for all of the other thousand. There are some approaches to the problem that will make it even worse (like taking averages and setting the thresholds based on those, for example, hoping it will work). Let me tell you a secret: it won’t work. Here we have a second problem: instances of the same type can demonstrate very different behaviors.

The last problem is that new shiny services you company may want to use are announced every week. It is impossible, because of time constraints, to know all of those services well enough to create a perfect monitoring template. In other words: sometimes we are asked to monitor software we don’t completely understand.

In summary, you have thousands of services, some of them you don’t even know how they work, that are sending you million of metrics that mean nothing to you. Now, set the thresholds and enable the pager alert. The nightmare has started. Is there a different approach?

Machine Learning

We have to stop thinking that monitoring is a bunch of config files with thresholds that we copy from one server to another. There are no magic templates that will work. We need to use a different technique that removes us from the process. That template is “machine learning.” As stated in Wikipedia, it is a subfield of computer science that gives computers the ability to learn without being explicitly programmed. In it’s most basic form, it can be used to solve classification problems. For example, open pet photos and identify if it is a cat or a dog. This is a classification problem that both humans and computers can solve, but we are much much slower. The computer has to take the time to learn the patterns, but at some point it will do the classification in no time.

I hope you are starting to see a pattern here. Why do we need to care about monitoring and its configuration if we have computers that can learn patterns and classify things for us?

There are two main ways of doing probabilistic fault detection: Novelty Detection and Outlier Detection.

Novelty Detection

Novelty Detection is easy to visualize and understand. It takes a series of inputs and tries to find anomalies, something that hasn’t been seen before. For example, our credit card company has a function that takes “category, expense, date, hour, country” as arguments and returns an integer so that they can classify and identify all the purchases. Your monthly use of the credit card looks like this:

[0,4,4,5,5,5,4,3]

That is the normal model that defines your use of the credit card. Now, it can be used to detect anomalies.

  • [0] – OK
  • [4] – OK
  • [4] – OK
  • [1] – Anomaly! Operation canceled.

Easy and straightforward. It is simple and very useful in a lot of areas to generate alerts when something anomalous happens. One of the machine learning models that can do this is One-Class Support Vector Machines, but since this is not the kind of fault detection we are looking for I won’t go into details. If you are interested, follow these links:

Outlier Detection

Let’s say we have this data:

[0, 3, 5, 1, -2, 19, 2, 10, -9, 2, 1, 8, 3, 21, -1, 3]

Now we know how to find anomalies, but how do we find outliers? Looking at the numbers above, it seems 21, 19 and -9 could be outliers. But a more exact definition is needed (not just intuition). The most simple and usual way of doing it is the following:

We divide our data into three pieces. One cut will be done at 25%, the second cut at 75%. The number that it is at 25% is called the First Quartile, and the value of the second cut is called the Third Quartile. The IQR or Interquartile Range is the subtraction of the Third Quartile with the First Quartile. Now, an outlier is any number that falls in one of these two categories:

  • If the value is below: (First Quartile) – (1.5 × IQR)
  • If the value is above: (Third Quartile + (1.5 × IQR)

Using Python:

inputs = [0, 3, 5, 1, -2, 19, 2, 10, -9, 2, 1, 8, 3, 21, -1, 3] Q1 = np.percentile(inputs,25) Q3 = np.percentile(inputs,75) step = (Q3-Q1)*1.5 Q1 0.75 Q3 5.75 step 7.5 outliers = [x for x in inputs if x < Q1-step or x > Q3+step] outliers [19, -9, 21]

This looks more like what we need. If we are monitoring a metric, and outliers are detected, then something is happening there that requires investigation. Some of the most used outlier detection models in scikit-learn are:

  • Elliptic Envelope: a robust co-variance estimate that assumes that our data is Gaussian distributed. It will define the shape of the data we have, creating a frontier that delimits the contour. As you probably guessed, it will be elliptical in shape. Don’t worry about the assumption of Gaussian distribution, data can be standardized. More about this later on.

 

  • Isolation Forest: this is the well-known “forest of random trees,” but applied to outlier detection. This is more suitable when we have many different input metrics. In the example I use below, I just use a single metric, so this model would not work that well.

Therefore, Elliptic Envelope looks like the best option for our proof-of-concept.

For visual reference, this is how the three models look like when they try to shape two data inputs:

Source: scikit-learn.org

 

Proof-of-Concept

I haven’t explained the model in detail, but a high level explanation should be enough to understand the problem and the possible solution. Let’s start building a proof-of-concept.

For this test, I got data from our Prometheus setup, where all the time-series monitoring data from our customers is stored. In this particular example, I got numbers from the “Threads Running” metric. Those will be used to train our Elliptical Envelope. It is important to take the following into account:

    • We need to collect enough data so that it captures the correct shape of our baseline performance. For example, usually nighttime hours have less of a workload than during the day (same with weekend days, in some cases).
    • As explained before, it assumes a Gaussian distribution, which means that the data needs to be scaled. I am going to standardize the data so that it has 0 mean and 1 variance. The same standardization needs to be applied to the data we test after the training process, when the monitoring is already in place. That standardization also needs to be applied to each metric individually. This is the formula:
Source: dataminingblog.com

With μ as the mean and σ as the standard deviation.

This is the summary of what our proof-of-concept will do:

  • Read Prometheus JSON dump.
  • Separate some data for training, standardizing it first.
  • Separate some data for testing, standardizing it first as well.
  • Make predictions on test data.
  • For those rows identified as outliers, get the original non-standardize data to see the number of threads running.

So, let’s start:

Import the Libraries

import pandas as pd import numpy as np import json from datetime import datetime from sklearn.preprocessing import StandardScaler from sklearn.covariance import EllipticEnvelope

Load the Data

All the information is in a JSON output from Prometheus that has the “threads_running” of a particular server. It has one second granularity for the last four weeks. I also converted “timestamps” to a normal “datetime” object so that it is easier to read:

with open('query.json') as data_file: data = json.load(data_file) data = pd.DataFrame(data["data"]["result"][0]["values"]) data[0] = data[0].astype(int) data[0] = data[0].apply(lambda x: datetime.fromtimestamp(x))

The data looks like this:

DateTime Threads Running 2017-01-19 20:32:44 1 2017-01-19 20:32:45 1 2017-01-19 20:32:46 2 … …

 

Create the Training and Testing Dataset

First, separate some of the data for use as training:

train_data = data[(data[0] >= "2017-01-22") & (data[0] <= "2017-01-28" )]

Ignore the date column, and just store the metrics:

train_data = train_data.iloc[:,[1]]

Standardize it:

escaler = StandardScaler() train_data = escaler.fit_transform(train_data)

Now the data looks like this:

Standardized Threads Running -0.4072634 -0.4072634 0.47153585 …

To create the test dataset we need to follow the exact same procedure, only select a different timeframe:

test_original_data = data[(data[0] >= "2017-02-2") & (data[0] <= "2017-02-17" )] test_data = test_original_data.iloc[:,[1]] test_data = escaler.transform(test_data)

Train the Model

Let’s create our model with the training data! I am using two parameters here:

  • assume_centered: to specify that our data is already Gaussian distributed.
  • contamination: to specify the ratio of outliers our training data has.

clf = EllipticEnvelope(assume_centered=True,contamination=0) clf.fit(train_data)

Search for Outliers

Now that we’ve trained the model and we have our test data, we can ask the model if it finds any outliers. It will return 1 or -1 for each row. “1” means that the value of threads running is normal and within the boundaries, while “-1” means that the value is an outlier:

predictions = clf.predict(test_data) outliers = np.where(predictions==-1)

The array “outliers” stores the row numbers where -1 was predicted.

At this point we have three important variables:

  • test_data: standardized testing data.
  • test_original_data: the original test data without modification.
  • outliers: the row numbers where an outlier was detected (-1).
Investigate the Outliers

Since we have the row number where an outlier was detected, now we can just query test_original_data and search for those rows. In this example, I show some random ones:

for indice in outliers[0]: if np.random.randn() > 2.5: print("{} - {} threads running".format(test_original_data.iloc[indice][0], test_original_data.iloc[indice][1])) 2017-02-03 11:26:03 - 41 threads running 2017-02-03 11:26:40 - 43 threads running 2017-02-03 11:27:50 - 48 threads running 2017-02-03 11:32:07 - 78 threads running 2017-02-03 11:33:25 - 90 threads running 2017-02-12 10:06:58 - 36 threads running 2017-02-12 10:12:11 - 60 threads running 2017-02-12 10:12:30 - 64 threads running

And there we have it! Dates and hours when something really out of the ordinary happened. No need to create a config file for each service, guess thresholds, adjust them … nothing. Just let the model learn, and you get alerts when something unexpected happens. Push all the metrics from your services to these models, and let them do the hard work.

Summary

Most companies have similar situations. Companies add new services on hundred of servers, and monitoring is an essential part of the infrastructure. The old method of monolithic config files with some thresholds doesn’t scale, because it needs a lot of manual work with a trial/error approach. The types of techniques explained in this blog post can help us deploy monitoring on hundred of servers, not really caring about the different nuances of each service or workload. It is even possible to start monitoring a service without even knowing anything about it — just let the probabilistic model take care of it.

It is important to clarify that, in my opinion, these fault detection models are not going to be a substitute for software like Nagios. In those areas where a binary test is needed (service is up/down for example), Nagios and other similar services do a good job. Actually, a Nagios check can use the procedure explained here. When there are many metrics to analyze, probabilistic methods can save us from a nightmare.

Migrating MongoDB Away from MMAPv1

March 8, 2017 - 3:16pm

This is another post in the series of blogs on the Percona Server for MongoDB 3.4 bundle release. In this blog post, we’ll discuss moving away from the MMAPv1 storage engine.

Introduction

WIth the MongoDB v3.0 release in February of 2015, the long-awaited ability to choose storage engines became a reality. As of version 3.0, you could choose two engines in MongoDB Community Server and, if you use Percona Server for MongoDB, you could choose from four. Here’s a table for ease of consumption:

Here’s a table for easy consumption:

Storage Engine Percona Server for MongoDB MongoDB Community Server MongoDB Enterprise Server (licensed) MMAPv1

WiredTiger

MongoRocks

In-memory

Encrypted

 

Why change engines?

With increased possibilities comes an increase in the decision-making process difficult (a concept that gets reinforced every time I take my mother out a restaurant with a large menu – ordering is never quick). In all seriousness, the introduction of the storage engine API to MongoDB is possibly the single greatest feature MongoDB, Inc has released to-date.

One of the biggest gripes from the pre-v3.0 days was MongoDB’s lack of scale. This was mostly due to the MMAPv1 storage engine, which suffered from a very primitive locking scheme. If you would like a illustration of the problem, think of the world’s biggest supermarket with one checkout line – you might be able to fit in lots of shoppers, but they’re not going to accomplish their goal quickly. So, the ability to increase performance and concurrency with a simple switch is huge! Additionally, modern storage engines support compression. This should reduce your space utilization when switching by at least 50%.

All the way up to MongoDB v3.2, the default storage engine was MMAPv1. If you didn’t make a conscious decision about what storage engine to choose when you started using MongoDB, there is a good chance that MMAPv1 is what you’re on. If you’d like to find out for sure what engine you’re using, simply run the command below. The output will be the name of the storage engine. As you can see, I was running the MMAPv1 storage engine on this machine. Now that we understand where we’re at, let’s get into where we can be in the future.

db.serverStatus().storageEngine.name mmapv1

Public Service Announcement

Before we get into what storage engine(s) to evaluate, we need to talk about testing. In my experience, a majority of the MySQL and MongoDB community members are rolling out changes to production without planning or testing. If you’re in the same boat, you’re in very good company (or at least in a great deal of company). However, you should stop this practice. It’s basic “sample size” in statistics – when engaged in risk-laden behavior, the optimal time to stop increasing the sample size is prior to the probability of failure reaching “1”. In other words, start your testing and planning process today!

At Percona, we recommend that you thoroughly test any database changes in a testing or development environment before you decide to roll them into production. Additionally, prior to rolling the changes into production (with a well thought out plan, of course), you’ll need to have a roll-back plan in case of unintended consequences. Luckily, with MongoDB’s built-in replication and election protocols, both are fairly easy. The key here is to plan. This is doubly true if you are undertaking a major version upgrade, or are jumping over major versions. With major version upgrades (or version jumps) comes the increased likelihood of a change in database behavior as it relates to your application’s response time (or even stability).

What should I think about?

In the table above, we listed the pre-packaged storage engine options that are available to us and other distributions. We also took a look at why you should consider moving off of MMAPv1 in the preceding section. To be clear, in my opinion a vast majority of MongoDB users that are on MMAPv1 can benefit from a switch. Which engine to switch to is the pressing question. Your first decision should be to evaluate whether or not your workload fits into the sweet spot for MMAPv1 by reading the section below. If that section doesn’t describe your application, then the additional sections should help you narrow down your choices.

Now, let’s take a look at what workloads match up with what storage engines.

MMAPv1

Believe it or not, there are some use cases where MMAPv1 is likely to give you as good (or better) performance as any other engine. If you’re not worried about the size of your database on disk, then you may not want to bother changing engines. Users that are likely to see no benefit from changing have read-heavy (or 100%) read applications. Also, certain update-heavy use cases, where you’re updating small amounts of data or performing $set operations, are likely to be faster on MMAPv1.

WiredTiger

WiredTiger is a the new default storage engine for MongoDB. It is a good option for general workloads that are currently running on MMAPv1. WiredTiger will give you good performance for most workloads and will reduce your storage utilization with compression that’s enabled by default. If you have a write-heavy workload, or are approaching high I/O utilization (>55%) with plans for it to rise, then you might benefit from a migration to WiredTiger.

MongoRocks (RocksDB from Facebook)

This is Facebook’s baby, which was forged in the fires of the former Parse business unit. MongoRocks, which uses LSM indexing, is advertised as “designed to work with fast storage.” Don’t let this claim fool you. For workloads that are heavy on writes, highly concurrent or approaching disk bound, MongoRocks could give you great benefits. In terms of compression, MongoRocks has the ability to efficiently handle deeper compression algorithms, which should further decrease your storage requirements.

In-Memory

The in-memory engine, whether we’re speaking about the MongoDB or Percona implementation, should be used for workloads where extreme low latency is the most important requirement. The types of applications that I’m talking about are usually low-latency, “real-time” apps — like decision making or user session tracking. The in-memory engine is not persistent, so it operates strictly out of the cache allocated to MongoDB. Consequently, the data may (and likely will) be lost if the server crashes.

Encrypted

This is for applications in highly secure environments where on-disk encryption is necessary for compliance. This engine will protect the MongoDB data in the case that a disk or server is stolen. On the flip side, this engine will not protect you from a hacker that has access to the server (MongoDB shell), or can intercept your application traffic. Another way to achieve the same level of encryption for compliance is using volume level encryption like LUKS. An additional benefit of volume level encryption, since it works outside the database, is re-use on all compliant servers (not just MongoDB).

Getting to your new engine

Switching to the new engine is actually pretty easy, especially if you’re running a replica set. One important caveat is that unlike MySQL, the storage engine can only be defined per mongod process (not per database or collection). This means that it’s an all or nothing operation on a single MongoDB process. You’ll need to reload your data on that server. This is necessary because the data files from one engine are not compatible with another engine. Thus reloading the data to transform from one engine format to another is necessary. Here are the high-level steps (assuming you’re running a replica set):

  1. Make sure you’re not in your production environment
  2. Backup your data (it can’t hurt)
  3. Remove a replica set member
  4. Rename (or delete) the old data directory. The member will re-sync with the replica set
    • Make sure you have enough disk space if you’re going to keep a copy of the old data directory
  5. Update the mongo.conf file to use a new storage engine. Here’s an example for RocksDB from our documentation:
    storage:  engine: rocksdb  rocksdb:    cacheSizeGB: 4    compression: snappy
  6. Start the MongoDB process again
  7. Join the member to the replica set (initial sync will happen)
  8. When the updated member is all caught up, pick another member and repeat the process.
  9. Continue until the primary is the only server left. At this point, you should step down the primary, but hold off switching storage engines until you are certain that the new storage engine meets your needs.
The Wrap Up

At this point I’ve explained how you can understand your options, where you can gain additional performance and what engines to evaluate. Please don’t forget to test your application with the new setup before launching into production. Please drop a comment below if you found this helpful or, on the other hand, if there’s something that would make it more useful to you. Chances are, if you’d find something helpful, the rest of the community will as well.

Webinar Thursday, March 9, 2017: Troubleshooting Issues with MySQL Character Sets

March 6, 2017 - 3:52pm

Please join Percona’s Principal Technical Services Engineer, Sveta Smirnova as she presents “Troubleshooting Issues with MySQL Character Sets ” on March 9, 2017, at 11:00 am PST / 2:00 pm EST (UTC-8).

Register Now

Many MySQL novices find MySQL character sets support puzzling. But after you understand how it is designed, you will find it much more powerful than many other competing database solutions.

MySQL allows to specify a character set for every object, and change it online. For years this has helped to create fast applications that can work with audiences all around the world. However, it also requires any DBA troubleshooting character set issues to have a deep understanding of how they work. Different sort rules and collations can complicate the process.

In the webinar we will discuss:

  • Which character sets and collations MySQL supports
  • How they can be set
  • How to understand error messages
  • How to solve character sets/collations compatibility issues
  • What server, application, command line and graphical tool options are available
  • What to check first and how to continue troubleshooting
  • What the various compatibility issues are
  • How to convert data, created in earlier versions
  • What the best practices are

Register for the webinar here.

Sveta Smirnova, Principal Technical Services Engineer

Sveta joined Percona in 2015. Her main professional interests are problem-solving, working with tricky issues, bugs, finding patterns that can solve typical issues quicker and teaching others how to deal with MySQL issues, bugs and gotchas effectively. Before joining Percona, Sveta worked as Support Engineer in MySQL Bugs Analysis Support Group in MySQL AB-Sun-Oracle. She is the author of the book “MySQL Troubleshooting” and JSON UDF functions for MySQL.

 

MySQL, –i-am-a-dummy!

March 6, 2017 - 11:54am

In this blog post, we’ll look at how “operator error” can cause serious problems (like the one we saw last week with AWS), and how to avoid them in MySQL using --i-am-a-dummy.

Recently, AWS had some serious downtime in their East region, which they explained as the consequence of a bad deployment. It seems like most of the Internet was affected in one way or another. Some on Twitter dubbed it “S3 Dependency Awareness Day.”

Since the outage, many companies (especially Amazon!) are reviewing their production access and deployment procedures. It would be a lie if I claimed I’ve never made a mistake in production. In fact, I would be afraid of working with someone who claims to have never made a mistake in a production environment.

Making a mistake or two is how you learn to have a full sense of fear when you start typing:

UPDATE t1 SET c1='x' ...

I think many of us have experienced forehead sweats and hand shaking in these cases – they save us from major mistakes!

The good news is that MySQL can help you with this. All you have to do is admit that you are human, and use the following command (you can also set this in your user directory .my.cnf):

mysql --i-am-a-dummy

Using this command (also known as safe-updates) sets the following SQL mode when logging into the server:

SET sql_safe_updates=1, sql_select_limit=1000, max_join_size=1000000;

The safe-updates and iam-a-dummy flags were introduced together in MySQL 3.23.11, and according to some sites from around the time of release, it’s “for users that once may have done a DELETE FROM table_name but forgot the WHERE clause.”

What this does is ensure you can’t perform an UPDATE or DELETE without a WHERE clause. This is great because it forces you to think through what you are doing. If you still want to update the whole table, you need to do something like WHERE ID > 0. Interestingly, safe-updates also blocks the use of WHERE 1, which means “where true” (or basically everything).

The other safety you get with this option is that SELECT is automatically limited to 1000 rows, and JOIN is limited to examining 1 million rows. You can override these latter limits with extra flags, such as:

--select_limit=500 --max_join_size=10000

I have added this to the .my.cnf on my own servers, and definitely use this with my clients.

MongoDB Audit Log: Why and How

March 3, 2017 - 3:24pm

This blog post is another in the series on the MongoDB 3.4 bundle release. In this blog post, we’ll talk about the MongoDB audit log.

Percona’s development team has always invested in the open-source community a priority – especially for MongoDB. As part of this commitment, Percona continues to build MongoDB Enterprise Server features into our free, alternative, open-source Percona Server for MongoDB. One of the key features that we have added to Percona Server for MongoDB is audit logging. Auditing your MongoDB environment strengthens your security and helps you keep track of who did what in your database.

In this blog post, we will show how to enable this functionality, what general actions can be logged, and how you can filter only the information that is important for your use-case.

Enable Audit Log

Audit messages can be logged into syslog, console or file (JSON or BSON format). In most cases, it’s preferable to log to the file in BSON format (the performance impact is smaller than JSON). In the last section, you can find some simple examples of how to further query this type of file.

Enable the audit log in the command line or the config file with:

mongod --dbpath /var/lib/mongodb --auditDestination file --auditFormat BSON --auditPath /var/lib/mongodb/auditLog.bson

auditLog: destination: file format: BSON path: /var/lib/mongodb/auditLog.bson

Just note that until this bug is fixed and released, if you’re using Percona Server for MongoDB and the --fork option while starting the mongod instance you’ll have to provide an absolute path for audit log file instead of relative path.

Actions logged

Generally speaking, the following actions can be logged:

  • Authentication and authorization
  • Cluster operations
  • Read and write operations (logged under authCheck event and require auditAuthorizationSuccess parameter to be enabled)
  • Schema operations
  • Custom application messages (logged under applicationMessage event if the client/app issues a logApplicationMessage command,  the user needs to have clusterAdmin role or the one that inherits from it to issue this command)

You can see the whole list of actions logged here.

By default, MongoDB doesn’t log all the read and write operations. So if you want to track those, you’ll have to enable the auditAuthorizationSuccess parameter. They then will be logged under the authCheck event. Note that this can have a serious performance impact.

Also, this parameter can be enabled dynamically on an already running instance with the audit log setup, while some other things can’t be changed once setup.

Enable logging of CRUD operations in the command line or config file:

mongod --dbpath /var/lib/mongodb --setParameter auditAuthorizationSuccess=true --auditDestination file --auditFormat BSON --auditPath /var/lib/mongodb/auditLog.bson

auditLog: destination: file format: BSON path: /var/lib/mongodb/auditLog.bson setParameter: { auditAuthorizationSuccess: true }

Or to enable it on the running instance, issue this command in the client:

db.adminCommand( { setParameter: 1, auditAuthorizationSuccess: true } )

Filtering

If you don’t want to track all the events MongoDB is logging by default, you can specify filters in the command line or the config file. Filters need to be valid JSON queries on the audit log message (format available here). In the filters, you can use standard query selectors ($eq, $in, $gt, $lt, $ne, …) as well as regex. Note that you can’t change the filters dynamically after the start.

Also, Percona Server for MongoDB 3.2 and 3.4 have slightly different message formats. 3.2 uses a “params” field, and 3.4 uses “param” just like MongoDB. When filtering on those fields, you might want to check for the difference.

Filter only events from one user:

mongod --dbpath /var/lib/mongodb --auditDestination file --auditFormat BSON --auditPath /var/lib/mongodb/auditLog.bson --auditFilter '{ "users.user": "prod_app" }'

auditLog: destination: file format: BSON path: /var/lib/mongodb/auditLog.bson filter: '{ "users.user": "prod_app" }'

Filter events from several users based on username prefix (using regex):

mongod --dbpath /var/lib/mongodb --auditDestination file --auditFormat BSON --auditPath /var/lib/mongodb/auditLog.bson --auditFilter '{ "users.user": /^prod_app/ }'

auditLog: destination: file format: BSON path: /var/lib/mongodb/auditLog.bson filter: '{ "users.user": /^prod_app/ }'

Filtering multiple event types by using standard query selectors:

mongod --dbpath /var/lib/mongodb --auditDestination file --auditFormat BSON --auditPath /var/lib/mongodb/auditLog.bson --auditFilter '{ atype: { $in: [ "dropCollection", "dropDatabase" ] } }'

auditLog: destination: file format: BSON path: /var/lib/mongodb/auditLog.bson filter: '{ atype: { $in: [ "dropCollection", "dropDatabase" ] } }'

Filter read and write operations on all the collections in the test database (notice the double escape of dot in regex):

mongod --dbpath /var/lib/mongodb --auditDestination file --auditFormat BSON --auditPath /var/lib/mongodb/auditLog.bson --setParameter auditAuthorizationSuccess=true --auditFilter '{ atype: "authCheck", "param.command": { $in: [ "find", "insert", "delete", "update", "findandmodify" ] }, "param.ns": /^test\\./ } }'

auditLog: destination: file format: BSON path: /var/lib/mongodb/auditLog.bson filter: '{ atype: "authCheck", "param.command": { $in: [ "find", "insert", "delete", "update", "findandmodify" ] }, "param.ns": /^test\\./ } }' setParameter: { auditAuthorizationSuccess: true }

Example messages

Here are two example messages from an audit log file. The first one is from a failed client authentication, and the second one is where the user tried to insert a document into a collection for which he has no write authorization.

> bsondump auditLog.bson {"atype":"authenticate","ts":{"$date":"2017-02-14T14:11:29.975+0100"},"local":{"ip":"127.0.1.1","port":27017},"remote":{"ip":"127.0.0.1","port":42634},"users":[],"roles":[],"param":{"user":"root","db":"admin","mechanism":"SCRAM-SHA-1"},"result":18}

> bsondump auditLog.bson {"atype":"authCheck","ts":{"$date":"2017-02-14T14:15:49.161+0100"},"local":{"ip":"127.0.1.1","port":27017},"remote":{"ip":"127.0.0.1","port":42636},"users":[{"user":"antun","db":"admin"}],"roles":[{"role":"read","db":"admin"}],"param":{"command":"insert","ns":"test.orders","args":{"insert":"orders","documents":[{"_id":{"$oid":"58a3030507bd5e3486b1220d"},"id":1.0,"item":"paper clips"}],"ordered":true}},"result":13}

Querying audit log for specific event

The audit log feature is now working, and we have some data in the BSON binary file. How do I query it to find some specific event that interests me? Obviously there are many simple or more complex ways to do that using different tools (Apache Drill or Elasticsearch come to mind), but for the purpose of this blog post, we’ll show two simple ways to do that.

The first way without exporting data anywhere is using the bsondump tool to convert BSON to JSON and pipe it into the jq tool (command-line JSON processor) to query JSON data. Install the jq tool in Ubuntu/Debian with:

sudo apt-get install jq

Or in Centos with:

sudo yum install epel-release sudo yum install jq

Then, if we want to know who created a database with the name “prod” for example, we can use something like this (I’m sure you’ll find better ways to use the jq tool for querying this kind of data):

> bsondump auditLog.bson | jq -c 'select(.atype == "createDatabase") | select(.param.ns == "prod")' {"atype":"createDatabase","ts":{"$date":"2017-02-17T12:13:48.142+0100"},"local":{"ip":"127.0.1.1","port":27017},"remote":{"ip":"127.0.0.1","port":47896},"users":[{"user":"prod_app","db":"admin"}],"roles":[{"role":"root","db":"admin"}],"param":{"ns":"prod"},"result":0}

In the second example, we’ll use the mongorestore tool to import data into another instance of mongod, and then just query it like a normal collection:

> mongorestore -d auditdb -c auditcol auditLog.bson 2017-02-17T12:28:56.756+0100 checking for collection data in auditLog.bson 2017-02-17T12:28:56.797+0100 restoring auditdb.auditcol from auditLog.bson 2017-02-17T12:28:56.858+0100 no indexes to restore 2017-02-17T12:28:56.858+0100 finished restoring auditdb.auditcol (142 documents) 2017-02-17T12:28:56.858+0100 done

The import is done, and now we can query the collection for the same data from the MongoDB client:

> use auditdb switched to db auditdb > db.auditcol.find({atype: "createDatabase", param: {ns: "prod"}}) { "_id" : ObjectId("58a6de78bdf080b8e8982a4f"), "atype" : "createDatabase", "ts" : { "$date" : "2017-02-17T12:13:48.142+0100" }, "local" : { "ip" : "127.0.1.1", "port" : 27017 }, "remote" : { "ip" : "127.0.0.1", "port" : 47896 }, "users" : [ { "user" : "prod_app", "db" : "admin" } ], "roles" : [ { "role" : "root", "db" : "admin" } ], "param" : { "ns" : "prod" }, "result" : 0 }

It looks like the audit log in MongoDB/Percona Server for MongoDB is a solid feature. Setting up tracking for information that is valuable to you only depends on your use case.

A Look at MariaDB Subquery Cache

March 2, 2017 - 6:11pm

The MariaDB subquery cache feature added in MariaDB 5.3 is not widely known. Let’s see what it is and how it works.

What is a subquery cache?

The MariaDB subquery cache optimizes the execution of correlated subqueries. Correlated subqueries refer to a value from the parent query. For example:

SELECT id FROM product WHERE price NOT IN (SELECT MAX(price) FROM product GROUP BY category);

MariaDB only uses this optimization if the parent query is a SELECT, not an UPDATE or a DELETE. The subquery results get cached only for the duration of the parent query.

MariaDB added the subquery cache in v5.3. It is controlled by optimizer_switch, a dynamic variable that contains many flags that enable or disable several optimizations. To disable the subquery cache, run:

SET GLOBAL optimizer_switch='subquery_cache=OFF';

You can also do this at the session level.

How does subquery cache work?

Let’s see how it works. To make things clearer, we will use an example. Consider these tables:

CREATE TABLE t1 (a INT, b INT); INSERT INTO t1 VALUES (1,2), (3,4), (1,2), (3,4), (3,4), (3,5), (3,5), (5,1), (5,2), (3,6), (1,5); CREATE TABLE t2 (c INT, d INT); INSERT INTO t2 VALUES (1,10), (2,20), (3,30), (4,40);

Now, we issue this query:

SELECT b, (SELECT d FROM t2 WHERE a = c) FROM t1;

The server decides to read t1 first (the bigger table, as expected), and then access t2 using the subquery cache. It creates a MEMORY temporary table to store the results of the subquery, with an index on c (it is used to match the rows). Then it reads the first row from t1, and checks if the search is cached. It is not, so it reads t1 looking for rows with c=1 and copies the results into the cache. The next time it will find the value 1, and it will not need to access t2 because the matches are already cached. If you look at the data, you may notice that the value “5” appears twice in t1 (and is absent in t2). But the search is cached anyway, so the server searches for 5 in t2 only once.

I hope that you aren’t blindly accepting what I wrote until now: good DBAs need facts and metrics. Let’s be scientific: we’ll make a prediction, conduct an experiment and check the status variables to verify the prediction. The prediction is the list of rows that will be read from t1, with the running total of hits and misses:

(1,2) -- Miss: 1 (3,4) -- Miss: 2 (1,2) -- Hit: 1 (3,4) -- Hit: 2 (3,4) -- Hit: 3 (3,5) -- Hit: 4 (3,5) -- Hit: 5 (5,1) -- Miss: 3 (5,2) -- Hit: 6 (3,6) -- Hit: 7 (1,5) -- Hit: 8

MariaDB [test]> SHOW STATUS LIKE "subquery_cache%"; +---------------------+-------+ | Variable_name | Value | +---------------------+-------+ | Subquery_cache_hit | 8 | | Subquery_cache_miss | 3 | +---------------------+-------+ 2 rows in set (0.00 sec) MariaDB [test]> SHOW STATUS LIKE "handler_tmp_write"; +-------------------+-------+ | Variable_name | Value | +-------------------+-------+ | Handler_tmp_write | 3 | +-------------------+-------+ 1 row in set (0.00 sec)

The totals match, and the number of writes to the cache is equal to the misses (after a miss, a table access is done and cached).

The maximum size of an individual table is the minimum of tmp_table_size and max_heap_table_size. If the table size grows over this limit, the table is written to disk. If the MEMORY table creation fails (perhaps because MEMORY does not support BLOB), the subquery is not cached.

The total of hits and misses can be seen by reading two status variables: subquery_cache_hit and subquery_cache_miss. After 200 misses, the server checks the hit ratio for that particular subquery. If it is < 20%, it disables the cache for that subquery. If the hit rate is < 70% the table cannot be written to disk in case it exceeds the size limit. These numbers (200, 0.2, 0.7) are hardcoded and cannot be changed. But if you really want to test how MariaDB behaves with different parameters, you can change these constants in sql/sql_expression_cache.cc and recompile the server.

Isn’t this subquery materialization?

Subquery materialization is another strategy that the optimizer can choose to execute a query. It might look similar, because some data from a subquery are written to a MEMORY table – but this is the only similarity. The purpose and implementation of subquery materialization is different.

Let’s try to explain this with pseudocode.

Subquery materialization is for uncorrelated IN subqueries. Therefore the subquery is executed and materialized altogether:

# Query to optimize: SELECT ... WHERE col1 IN (subquery) materialize subquery into a MEMORY table with UNIQUE keys; foreach (row in outer query) { check if col1 current value exists in materialized table }

The subquery cache is for correlated subqueries. Thus the subquery gets executed only for non-cached values:

# Query to optimize: SELECT col1, (SELECT ... WHERE ... = col1) ... FROM ... foreach (outer query row) { if (col1 current value is cached) { read from cache } else { read from subquery cache col1 current value } }

Some considerations

Despite the similarity in names, the MariaDB subquery cache is not a query cache for subqueries. These features are different, implemented for different purposes. Obviously, the subquery cache doesn’t have the scalability and performance problems of the query cache (global mutex, table invalidation). As mentioned, a subquery cache table only survives for the duration of a statement, so it should be considered an optimizer strategy. For example, in some cases you might use the subquery cache for a WHERE … NOT IN subquery, but not for the WHERE … IN version, because the optimizer prefers to rewrite it as a JOIN.

Of course, not all correlated subqueries automatically benefit from this feature. Consider the example above: it is built to show that the subquery cache is useful. But we can easily build an example to show that can have a negative impact on performance: add rows to t1, and delete all duplicate values of a. There will be no hits, a temporary table is created, 200 reads and writes are performed, but it won’t help. After 200 misses, the cache will be disabled, yes, but what if this happens for each subquery? The damage may not be huge in a realistic case, but it is still damage. That’s why you can disable the MariaDB subquery cache.

Using Percona Toolkit pt-mongodb-summary

March 2, 2017 - 3:08pm

In this blog post, we’ll look at the pt-mongodb-summary tool in Percona Toolkit.

The pt-mongodb-summary tool from Percona Toolkit provides a quick at-a-glance overview of MongoDB and Percona Server for MongoDB instances. It is equivalent to pt-mysql-summary for MySQL. 

pt-mongodb-summary also collects information about a MongoDB cluster. It collects information from several sources to provide an overview of the cluster.

How It Works

The usage for the command is as follows:

pt-mongodb-summary [OPTIONS] [HOST[:PORT]]

Options:

  • -a, –auth-db: Specifies the database used to establish credentials and privileges with a MongoDB server. By default, the admin database is used.
  • -p, –password: Specifies the password to use when connecting to a server with authentication enabled. Do not add a space between the option and its value: -p<password>.
    If you specify the option without any value, pt-mongodb-summary will ask for the password interactively.
  • -u, –user: Specifies the user name for connecting to a server with authentication enabled.

By default, if you run pt-mongodb-summary without parameters, it tries to connect to the localhost on port 27017. It collects information about the MongoDB instances by running administration commands, and formatting the output.

Sections Instances

The first thing the tool does is get the list of hosts connected to the specified MongoDB instance by running the listShards command. It also runs replSetGetStatus on every instance to collect the ID, type, and replica set for each instance.

This host

Next, it gathers information about the host it is connected to by grouping information collected from hostInfo, getCmdLineOpts, serverStatus and the OS process (by process ID). The result provides an overview of the running instance and the underlying OS.

Running ops

This section collects statistics by running the serverStatus command five times at regular intervals (every one second), and provides the minimum, maximum and average operation counters for insert, query, update, delete, getMore and command operations.

Security

This collects information about security configurations by parsing the getCmdLineOpts command and asking the admin.system.users, and admin.system.roles collections.

Oplog

From the MongoDB website:

The oplog (operations log) is a special capped collection that keeps a rolling record of all operations that modify the data stored in your databases. MongoDB applies database operations on the primary and then records the operations on the primary’s oplog. The secondary members then copy and apply these operations in an asynchronous process. All replica set members contain a copy of the oplog, in the local.oplog.rs collection, which allows them to maintain the current state of the database.

How do we get the oplog info? The program collects statistics from the oplog for every host in the cluster, and returns the information on the statistics having the smaller TimeDiffHours  value.

Cluster-wide

This section provides information about the number of sharded/unsharded databases, collections and their size.The information is collected by running the listDatabases command, and then running collStats for every collection in every database.

Conditional Sections

You may notice not all sections appear all the time. This is because there are three main patterns:

Sharded Connection to Mongos
  • Instances
  • This host
  • Running ops
  • Security
  • Cluster-wide
ReplicaSet Connection
  • Instances (limited to the current Replica Set)
  • This host
  • Running ops
  • Security
  • Oplog
Standalone Connection
  • Instances (limited to this host)
  • This host
  • Running ops
  • Security
Output Example

The following is an example of the output for pt-mongodb-summary:

./pt-mongodb-summary # Instances ############################################################################################## PID Host Type ReplSet Engine 11037 localhost:17001 SHARDSVR/PRIMARY r1 wiredTiger 11065 localhost:17002 SHARDSVR/SECONDARY r1 wiredTiger 11136 localhost:17003 SHARDSVR/SECONDARY r1 wiredTiger 11256 localhost:17004 SHARDSVR/ARBITER r1 wiredTiger 11291 localhost:18001 SHARDSVR/PRIMARY r2 wiredTiger 11362 localhost:18002 SHARDSVR/SECONDARY r2 wiredTiger 11435 localhost:18003 SHARDSVR/SECONDARY r2 wiredTiger 11513 localhost:18004 SHARDSVR/ARBITER r2 wiredTiger 11548 localhost:19001 CONFIGSVR - wiredTiger 11571 localhost:19002 CONFIGSVR - wiredTiger 11592 localhost:19003 CONFIGSVR - wiredTiger # This host # Mongo Executable ####################################################################################### Path to executable | /home/karl/tmp/MongoDB32Labs/3.2/bin/mongos # Report On karl-HP-ENVY ######################################## User | karl PID Owner | mongos Hostname | karl-HP-ENVY Version | 3.2.4 Built On | Linux x86_64 Started | 2017-02-22 11:39:20 -0300 ART Processes | 12 Process Type | mongos # Running Ops ############################################################################################ Type Min Max Avg Insert 0 0 0/5s Query 0 0 0/5s Update 0 0 0/5s Delete 0 0 0/5s GetMore 0 0 0/5s Command 1 1 5/5s # Security ############################################################################################### Users : 0 Roles : 0 Auth : disabled SSL : disabled Port : 0 Bind IP: # Cluster wide ########################################################################################### Databases: 4 Collections: 21 Sharded Collections: 5 Unsharded Collections: 16 Sharded Data Size: 134.87 MB Unsharded Data Size: 1.44 GB ### Chunks: 5 : samples.col2 132 : carlos.sample4 400 : carlos.sample3 50 : carlos.sample2 100 : carlos.sample1 # Balancer (per day) Success: 18 Failed: 0 Splits: 682 Drops: 0

 The following is an output example when connected to a secondary in the replica set.

./pt-mongodb-summary localhost:17002 # Instances ############################################################################################## PID Host Type ReplSet Engine 9247 localhost:17001 SHARDSVR/PRIMARY r1 wiredTiger 9318 localhost:17002 SHARDSVR/SECONDARY r1 wiredTiger 9391 localhost:17003 SHARDSVR/SECONDARY r1 wiredTiger 9466 localhost:17004 SHARDSVR/ARBITER r1 wiredTiger # This host # Mongo Executable ####################################################################################### Path to executable | /home/karl/tmp/MongoDB32Labs/3.2/bin/mongod # Report On karl-HP-ENVY:17002 ######################################## User | karl PID Owner | mongod Hostname | karl-HP-ENVY:17002 Version | 3.2.4 Built On | Linux x86_64 Started | 2017-02-23 10:26:27 -0300 ART Datadir | labs/r1-2 Processes | 12 Process Type | replset # Running Ops ############################################################################################ Type Min Max Avg Insert 0 0 0/5s Query 0 0 0/5s Update 0 0 0/5s Delete 0 0 0/5s GetMore 0 1 1/5s Command 1 3 13/5s # Security ############################################################################################### Users : 0 Roles : 0 Auth : disabled SSL : disabled Port : 17002 Bind IP:
This next example shows when it is connected to a stand alone instance:

/pt-mongodb-summary localhost:27018 # Instances ############################################################################################## PID Host Type ReplSet Engine 1 localhost:27018 - wiredTiger # This host # Report On 2f8862dce6c4 ######################################## PID Owner | mongod Hostname | 2f8862dce6c4 Version | 3.2.10 Built On | Linux x86_64 Started | 2017-02-23 08:57:36 -0300 ART Processes | 1 Process Type | mongod # Running Ops ############################################################################################ Type Min Max Avg Insert 0 0 0/5s Query 0 0 0/5s Update 0 0 0/5s Delete 0 0 0/5s GetMore 0 0 0/5s Command 1 1 5/5s # Security ############################################################################################### Users : 0 Roles : 0 Auth : disabled SSL : disabled Port : 0 Bind IP: Conclusion

The tool pt-mongodb-summaryis new in Percona Toolkit. In the future, we hope we can make this grow to the size of its MySQL big brother!

Using Percona Toolkit pt-mongodb-query-digest

March 1, 2017 - 3:21pm

In this blog post, we’ll look at how to use the pt-mongodb-query-digest tool in Percona Toolkit 3.0.

Percona’s pt-query-digest is one of our most popular Percona Toolkit MySQL tools. It is used on a daily basis by DBAs and developers to help identify the queries consuming the most resources. It helps in finding bottlenecks and optimizing database usage. The pt-mongodb-query-digest is a similar tool for MongoDB.

About the Profiler

Before we start, remember that the MongoDB database profiler is disabled by default, and should be enabled. It can be enabled server-wide, but the full mode that logs all queries is not recommended in production unless you are using Percona Server for MongoDB 3.2 or higher. We added a feature to allow the sample rate of non-slow queries (like in MySQL) to limit the overhead this causes. 

Additionally, by default, the profiler is only 1MB per database. You may want to remove/create the profiler to sufficient size to find the results useful. To do this, use:

org_prof_level = db.getProfilingLevel(); //Disable Profiler db.setProfilingLevel(0); db.system.profile.drop(); //Setup a 100M profile 1*Math.pow(1024,2) == 1M profiler_size = 100 * Math.pow(1024,2); db.runCommand( { create: "system.profile", capped: true, size: profiler_size } ); db.setProfilingLevel(org_prof_level);

According to the documentation, to check if the profiler is enabled for the samples database, run:

`echo "db.getProfilingStatus();" | mongo localhost:17001/samples`

Remember, you need to connect to a MongoDB instance, not a mongos. The output will be something like this:

MongoDB shell version: 3.2.12 connecting to: localhost:17001/samples { "was" : 0, "slowms" : 100 } bye

The value for the field “was” is 0, which means profiling is disabled. Let’s enable the profiler for the samples database.

You must enable the profiler on all MongoDB instances that could be related to a shard of our database. To check on which instances we should enable the profiler, I am going to use the pt-mongodb-summary tool. It shows us the information we need about our cluster:

./pt-mongodb-summary ./pt-mongodb-summary # Instances ############################################################################################## PID Host Type ReplSet Engine 11037 localhost:17001 SHARDSVR/PRIMARY r1 wiredTiger 11065 localhost:17002 SHARDSVR/SECONDARY r1 wiredTiger 11136 localhost:17003 SHARDSVR/SECONDARY r1 wiredTiger 11256 localhost:17004 SHARDSVR/ARBITER r1 wiredTiger 11291 localhost:18001 SHARDSVR/PRIMARY r2 wiredTiger 11362 localhost:18002 SHARDSVR/SECONDARY r2 wiredTiger 11435 localhost:18003 SHARDSVR/SECONDARY r2 wiredTiger 11513 localhost:18004 SHARDSVR/ARBITER r2 wiredTiger 11548 localhost:19001 CONFIGSVR - wiredTiger 11571 localhost:19002 CONFIGSVR - wiredTiger 11592 localhost:19003 CONFIGSVR - wiredTiger

We have mongod service running on the localhost on ports 17001~17003 and 18001~18003.

Now, let’s enable the profiler for the samples database on those instances. For this example, I am going to set the profile level to “2”, to collect information about all queries.

for port in 17001 17002 17003 18001 18002 18003; do echo "db.setProfilingLevel(2);" | mongo localhost:${port}/samples; done Running pt-mongodb-query-profile

Now we are ready to get statistics about our queries. To run pt-mongodb-query-digest, we need to specify at least “host: port/database”, like:

./pt-mongodb-query-digest localhost:27017/samples

The output will be something like this (I am showing a section for only one query):

# Query 0: 0.27 QPS, ID 2c0e2f94937d6660f510adeea98618f3 # Ratio 1.00 (docs scanned/returned) # Time range: 2017-02-22 12:27:21.004 -0300 ART to 2017-02-22 12:28:00.867 -0300 ART # Attribute pct total min max avg 95% stddev median # ================== === ======== ======== ======== ======== ======== ======= ======== # Count (docs) 845 # Exec Time ms 99 1206 0 697 1 0 29 0 # Docs Scanned 7 594.00 0.00 75.00 0.70 0.00 7.19 0.00 # Docs Returned 7 594.00 0.00 75.00 0.70 0.00 7.19 0.00 # Bytes recv 0 8.60M 215.00 1.06M 10.17K 215.00 101.86K 215.00 # String: # Namespaces samples.col1 # Operation query # Fingerprint user_id # Query {"user_id":{"$gte":3506196834,"$lt":3206379780}}

From the output, we can see that this query was seen 97 times, and it provides statistics for the number of documents scanned/retrieved by the server, the execution time and size of the results. The tool also provides information regarding the operation type, the fingerprint and a query example to help to identify the source. 

By default, the results are sorted by query count. It can be changed by setting the --order-by parameter to: count, ratio, query-time, docs-scanned or docs-returned.

A “-” in front of the field name denotes the reverse order. Example:

--order-by=-ratio

When considering what ordering to use, you need to know if you are looking for the most common queries (-count), the most cache abusive (-docs-scanned), or the worst ratio of scanned to returned (-ratio)? Please note you may be tempted to use (-query-time), however you will find this almost always ends up being more queries affected by, but not causing, issues.

Conclusion

This is a new tool in the Percona Toolkit. We hope in the future we can make it grow like its big brother for MySQL (pt-query-digest). This tool helps DBAs and developers identify and solve bottlenecks, and keep servers running at top performance.

Open Source Databases on Big Machines: Disk Speed and innodb_io_capacity

March 1, 2017 - 3:00pm

In this blog post, I’ll look for the bottleneck that prevented the performance in my previous post from achieving better results.

The powerful machine I used in the tests in my previous post has a comparatively slow disk, and therefore I expected my tests would hit a point when I couldn’t increase performance further due to the disk speed.

Hardware configuration:

Processors: physical = 4, cores = 72, virtual = 144, hyperthreading = yes
Memory: 3.0T
Disk speed: about 3K IOPS
OS: CentOS 7.1.1503
File system: XFS

Versions tested and configuration: same as in the first post of this series (check the post for specifics).

Even though I expected my tests would stop increasing in performance due to the disk speed, I did not observe high IO rates in the iostat output. I already tested with a full data set that fits in memory. In this case, write performance only affected data flushes and log writes. But we should still see a visible decrease in speed. So I decided to try RW tests totally in memory. I created a ramdisk and put the MySQL datadir on it. Surprisingly, results on the SSD and ramdisk did not differ.

I asked my colleagues from “Postgres Professional” to test PostgreSQL with the ramdisk. They got similar results:

It’s interesting that the value of innodb_io_capacity does not have any effect on this situation. Data for the graph below was taken when I ran tests on ramdisk. I wanted to see if I could control the IO activity of a disk, which is extremely fast by default, using this variable.

This totally contradicts all my past experiences with smaller machines. Percona re-purposed the machine with a faster disk (which I used before, described in this post), so I used a similar one with slower disk speed.

Hardware configuration:

Processors: physical = 2, cores = 12, virtual = 24, hyperthreading = yes
Memory: 47.2G
Disk speed: about 3K IOPS
OS: Ubuntu 14.04.5 LTS (trusty)
File system: ext4

Again, in this case innodb_io_capacity benchmarks with a smaller number of CPU cores showed more predictable results.

Conclusion:

Both MySQL and PostgreSQL on a machine with a large number of CPU cores hit CPU resources limits before disk speed can start affecting performance. We only tested one scenario, however. With other scenarios, the results might be different.

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Save

Visit Percona Store


General Inquiries

For general inquiries, please send us your question and someone will contact you.