Emergency

Strange iowait peaks on CPU Usage

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Strange iowait peaks on CPU Usage

    We noticed strange peaks on CPU Usage. We do not see anything strange in server behavior, we also don't see these peaks in Zabbix and sysstat daily output.
    When I set interval to 1 sec, it shows iowait usage at level 20000-30000%. Can it be a bug somewhere in node_exporter or pmm itself?

    pmm-client-1.1.1-1.x86_64

  • #2
    Hi Victor,

    we also don't see these peaks in Zabbix and sysstat daily output.
    it is expected because PMM very accurate, it reads statuses one per second.
    usual monitoring tools (like zabbix and sysstat) show average value for a minute and it is hard to troubleshoot database load based on such average values.

    Code:
    it shows iowait usage at level 20000-30000%
    as you know, pmm-client use prometheus/node_exporter for collecting cpu usage.
    I inspected node_exporter code
    https://github.com/prometheus/node_e...x.go#L101-L121
    looks like it just reads /proc/stat file and pass kernel counter values "as is" to prometheus, without any additional processing.
    so it is cannot be node_exporter error.

    can you share screenshot with cpu graph from prometheus?
    Code:
    http://pmm-server-ip/prometheus/graph?g0.range_input=1h&g0.expr=node_cpu%7Binstance%3D%22some-server%22%7D&g0.tab=0
    please replace "some-server" by problematic server hostname (it is visible in grafan dropdown list)

    Comment


    • #3
      Well, this graph looks quite strange and useless I've expanded the time period to 6 hours, but it's the same as for 1h.

      Comment


      • #4
        1. do you use virtualization with shared cpu core usage? (when one core sometimes shared between several VMs)
        2. can you try to use the following Query in graph?
        Code:
        sum(rate(node_cpu{instance="$host", mode!="idle"}[$interval])) by (mode)* 100 / scalar(sum(rate(node_cpu{instance="$host"}[$interval]))) or sum(irate(node_cpu{instance="$host", mode!="idle"}[5m])) by (mode) * 100 / scalar(sum(irate(node_cpu{instance="$host"}[5m])))
        it is needed to click on graph name, open for edit, open "metrics" tab and replace Query.
        please find screenshot
        query.png
        Attached Files

        Comment


        • #5
          Originally posted by Mykola View Post
          1. do you use virtualization with shared cpu core usage? (when one core sometimes shared between several VMs)
          2. can you try to use the following Query in graph?
          1. No, pmm-client is set up on the real server.
          2. Is it a new feature? I see only 'View' and 'Share' links, and no 'Edit'. I work with admin account

          Comment


          • #6
            what version of PMM Server do you use?
            can you click on Grafana logo and check user name?

            Comment


            • #7
              Originally posted by Mykola View Post
              what version of PMM Server do you use?
              can you click on Grafana logo and check user name?
              We just upgraded to 1.1.2 and nothing changed. Is there a hardcoded check for username or user id? We accidentally deleted the default admin account just after installing and creating custom accounts with admin rights.

              Comment


              • #8
                did you set SERVER_PASSWORD variable during container run?

                Comment


                • #9
                  Originally posted by Mykola View Post
                  did you set SERVER_PASSWORD variable during container run?
                  No, these variables are not mentioned here, so we didn't know about it. We should use it to access Edit tab?

                  Comment


                  • #10
                    No, it is not needed to use SERVER_PASSWORD (but recommended, see https://www.percona.com/doc/percona-.../security.html for more details)
                    i just wondered about authentication settings.

                    in 1.1.2 without SERVER_PASSWORD option - any unauthorised user should have permissions to modify graphs, so can you try to press "sign out" button and modify graphs?

                    Comment


                    • #11
                      After signing out it doesn't allow me to do anything but seeing /graph/login page

                      Comment


                      • #12
                        interesting... can you try to open "Incognito Window" in you browser and try to open http://PMM-Server-IP/graph/ url? it should work without auth under guest user.

                        can you share output of the following command?
                        Code:
                        docker inspect pmm-server

                        Comment


                        • #13
                          Originally posted by Mykola View Post
                          interesting... can you try to open "Incognito Window" in you browser and try to open http://PMM-Server-IP/graph/ url? it should work without auth under guest user.

                          can you share output of the following command?
                          Code:
                          docker inspect pmm-server
                          No, I'm still redirected to /graph/login
                          Code:
                          [
                              {
                                  "Id": "7ea43a1490b83e332f7313ecf090f24333dfee9e2924d9302dbabe4e34480f29",
                                  "Created": "2017-04-19T11:45:00.618581204Z",
                                  "Path": "/opt/entrypoint.sh",
                                  "Args": [],
                                  "State": {
                                      "Status": "running",
                                      "Running": true,
                                      "Paused": false,
                                      "Restarting": false,
                                      "OOMKilled": false,
                                      "Dead": false,
                                      "Pid": 9526,
                                      "ExitCode": 0,
                                      "Error": "",
                                      "StartedAt": "2017-04-19T11:45:06.156467447Z",
                                      "FinishedAt": "0001-01-01T00:00:00Z"
                                  },
                                  "Image": "sha256:38b9f4754f773c192d0e72ea65a400ee3b42e34b4a9a9505e278bdc0453b31c6",
                                  "ResolvConfPath": "/var/lib/docker/containers/7ea43a1490b83e332f7313ecf090f24333dfee9e2924d9302dbabe4e34480f29/resolv.conf",
                                  "HostnamePath": "/var/lib/docker/containers/7ea43a1490b83e332f7313ecf090f24333dfee9e2924d9302dbabe4e34480f29/hostname",
                                  "HostsPath": "/var/lib/docker/containers/7ea43a1490b83e332f7313ecf090f24333dfee9e2924d9302dbabe4e34480f29/hosts",
                                  "LogPath": "",
                                  "Name": "/pmm-server",
                                  "RestartCount": 0,
                                  "Driver": "devicemapper",
                                  "MountLabel": "",
                                  "ProcessLabel": "",
                                  "AppArmorProfile": "",
                                  "ExecIDs": null,
                                  "HostConfig": {
                                      "Binds": null,
                                      "ContainerIDFile": "",
                                      "LogConfig": {
                                          "Type": "journald",
                                          "Config": {}
                                      },
                                      "NetworkMode": "default",
                                      "PortBindings": {
                                          "80/tcp": [
                                              {
                                                  "HostIp": "",
                                                  "HostPort": "81"
                                              }
                                          ]
                                      },
                                      "RestartPolicy": {
                                          "Name": "always",
                                          "MaximumRetryCount": 0
                                      },
                                      "AutoRemove": false,
                                      "VolumeDriver": "",
                                      "VolumesFrom": [
                                          "pmm-data"
                                      ],
                                      "CapAdd": null,
                                      "CapDrop": null,
                                      "Dns": [],
                                      "DnsOptions": [],
                                      "DnsSearch": [],
                                      "ExtraHosts": null,
                                      "GroupAdd": null,
                                      "IpcMode": "",
                                      "Cgroup": "",
                                      "Links": null,
                                      "OomScoreAdj": 0,
                                      "PidMode": "",
                                      "Privileged": false,
                                      "PublishAllPorts": false,
                                      "ReadonlyRootfs": false,
                                      "SecurityOpt": null,
                                      "UTSMode": "",
                                      "UsernsMode": "",
                                      "ShmSize": 67108864,
                                      "Runtime": "docker-runc",
                                      "ConsoleSize": [
                                          0,
                                          0
                                      ],
                                      "Isolation": "",
                                      "CpuShares": 0,
                                      "Memory": 0,
                                      "CgroupParent": "",
                                      "BlkioWeight": 0,
                                      "BlkioWeightDevice": null,
                                      "BlkioDeviceReadBps": null,
                                      "BlkioDeviceWriteBps": null,
                                      "BlkioDeviceReadIOps": null,
                                      "BlkioDeviceWriteIOps": null,
                                      "CpuPeriod": 0,
                                      "CpuQuota": 0,
                                      "CpusetCpus": "",
                                      "CpusetMems": "",
                                      "Devices": [],
                                      "DiskQuota": 0,
                                      "KernelMemory": 0,
                                      "MemoryReservation": 0,
                                      "MemorySwap": 0,
                                      "MemorySwappiness": -1,
                                      "OomKillDisable": false,
                                      "PidsLimit": 0,
                                      "Ulimits": null,
                                      "CpuCount": 0,
                                      "CpuPercent": 0,
                                      "IOMaximumIOps": 0,
                                      "IOMaximumBandwidth": 0
                                  },
                                  "GraphDriver": {
                                      "Name": "devicemapper",
                                      "Data": {
                                          "DeviceId": "103",
                                          "DeviceName": "docker-253:1-3222577957-9b9a13bb42ea97578920df36577feea484dc7247056b586ec055a9490c0dae8a",
                                          "DeviceSize": "10737418240"
                                      }
                                  },
                                  "Mounts": [
                                      {
                                          "Name": "3ade038e4a2d102bca1d525c7adced6f3529581882feb16b909d44c0c185c759",
                                          "Source": "/var/lib/docker/volumes/3ade038e4a2d102bca1d525c7adced6f3529581882feb16b909d44c0c185c759/_data",
                                          "Destination": "/opt/consul-data",
                                          "Driver": "local",
                                          "Mode": "",
                                          "RW": true,
                                          "Propagation": ""
                                      },
                                      {
                                          "Name": "a7d74f024ed1f72f555620ccf3f9fb133685c00a42ab662fb0c4ec7b7b5aa05b",
                                          "Source": "/var/lib/docker/volumes/a7d74f024ed1f72f555620ccf3f9fb133685c00a42ab662fb0c4ec7b7b5aa05b/_data",
                                          "Destination": "/opt/prometheus/data",
                                          "Driver": "local",
                                          "Mode": "",
                                          "RW": true,
                                          "Propagation": ""
                                      },
                                      {
                                          "Name": "cfaa7da6208461687c48fc1877aaba076d6143345b3e612d334bd6c91f774a51",
                                          "Source": "/var/lib/docker/volumes/cfaa7da6208461687c48fc1877aaba076d6143345b3e612d334bd6c91f774a51/_data",
                                          "Destination": "/var/lib/grafana",
                                          "Driver": "local",
                                          "Mode": "",
                                          "RW": true,
                                          "Propagation": ""
                                      },
                                      {
                                          "Name": "41a2981c4be53d214c3d0d094cd31a47f1f2e635b5f68069062fb44b3c562c31",
                                          "Source": "/var/lib/docker/volumes/41a2981c4be53d214c3d0d094cd31a47f1f2e635b5f68069062fb44b3c562c31/_data",
                                          "Destination": "/var/lib/mysql",
                                          "Driver": "local",
                                          "Mode": "",
                                          "RW": true,
                                          "Propagation": ""
                                      }
                                  ],
                                  "Config": {
                                      "Hostname": "7ea43a1490b8",
                                      "Domainname": "",
                                      "User": "",
                                      "AttachStdin": false,
                                      "AttachStdout": false,
                                      "AttachStderr": false,
                                      "ExposedPorts": {
                                          "443/tcp": {},
                                          "80/tcp": {}
                                      },
                                      "Tty": false,
                                      "OpenStdin": false,
                                      "StdinOnce": false,
                                      "Env": [
                                          "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
                                      ],
                                      "Cmd": [
                                          "/opt/entrypoint.sh"
                                      ],
                                      "Image": "percona/pmm-server:1.1.2",
                                      "Volumes": null,
                                      "WorkingDir": "/opt",
                                      "Entrypoint": null,
                                      "OnBuild": null,
                                      "Labels": {
                                          "build-date": "20161214",
                                          "license": "GPLv2",
                                          "name": "CentOS Base Image",
                                          "vendor": "CentOS"
                                      }
                                  },
                                  "NetworkSettings": {
                                      "Bridge": "",
                                      "SandboxID": "b8b8dac97f23ffcb59d6cf62ce2d5af8f90bc724ff20dd652d022ba445745e21",
                                      "HairpinMode": false,
                                      "LinkLocalIPv6Address": "",
                                      "LinkLocalIPv6PrefixLen": 0,
                                      "Ports": {
                                          "443/tcp": null,
                                          "80/tcp": [
                                              {
                                                  "HostIp": "0.0.0.0",
                                                  "HostPort": "81"
                                              }
                                          ]
                                      },
                                      "SandboxKey": "/var/run/docker/netns/b8b8dac97f23",
                                      "SecondaryIPAddresses": null,
                                      "SecondaryIPv6Addresses": null,
                                      "EndpointID": "3d2968d423f39e0b77af96e326483578662467ca656c87c6bc629b29b2f757c8",
                                      "Gateway": "172.17.0.1",
                                      "GlobalIPv6Address": "",
                                      "GlobalIPv6PrefixLen": 0,
                                      "IPAddress": "172.17.0.2",
                                      "IPPrefixLen": 16,
                                      "IPv6Gateway": "",
                                      "MacAddress": "02:42:ac:11:00:02",
                                      "Networks": {
                                          "bridge": {
                                              "IPAMConfig": null,
                                              "Links": null,
                                              "Aliases": null,
                                              "NetworkID": "e3bf926620d6b36c55bd4828888a61b69968b2bb0b0f7a43560e5944ddf1d5d3",
                                              "EndpointID": "3d2968d423f39e0b77af96e326483578662467ca656c87c6bc629b29b2f757c8",
                                              "Gateway": "172.17.0.1",
                                              "IPAddress": "172.17.0.2",
                                              "IPPrefixLen": 16,
                                              "IPv6Gateway": "",
                                              "GlobalIPv6Address": "",
                                              "GlobalIPv6PrefixLen": 0,
                                              "MacAddress": "02:42:ac:11:00:02"
                                          }
                                      }
                                  }
                              }
                          ]

                          Comment


                          • #14
                            lets try to add admin permission for your user

                            you can open connection to grafana database with this command
                            Code:
                            docker exec -it pmm-server sqlite3 /var/lib/grafana/grafana.db
                            you can list users with following commands
                            Code:
                            .mode column
                            .header on
                            SELECT * FROM user;
                            add admin permissions for your user with the following command
                            Code:
                            UPDATE user SET is_admin=1 WHERE login='your-user';

                            Comment


                            • #15
                              Originally posted by Mykola View Post
                              Code:
                              UPDATE user SET is_admin=1 WHERE login='your-user';
                              Well, both our users already have this flag is_admin = 1 in database. Is it a mistery?
                              UPD:
                              Code:
                              UPDATE user SET id = 1 WHERE id = 2
                              helped me. Now I see Edit tab
                              Last edited by Victor Safronov; 04-21-2017, 06:22 AM.

                              Comment

                              Working...
                              X