Emergency

seeing gaps during data collection in PMM dashboard

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • seeing gaps during data collection in PMM dashboard

    Hello,
    We've installed PMM and it looks great! But the thing is that we see some gaps in between the charts as showing in the attachment.
    From client side pmm-mysql-queries.log, we see these warnings:

    2017/10/11 22:37:40.590505 WARNING data-sender Waiting for API to ack qan_1507761420012724193: read tcp x.x.x.x:50024->x.x.x.x:80: i/o timeout
    2017/10/11 22:37:49.293295 WARNING data-sender Waiting for API to ack qan_1507761420012724193: read tcp x.x.x.x:50026->x.x.x.x:80: i/o timeout
    2017/10/11 22:47:07.464413 WARNING data-sender Waiting for API to ack qan_1507761960013420750: read tcp x.x.x.x:50046->x.x.x.x:80: i/o timeout
    2017/10/11 22:54:28.530116 WARNING data-sender Waiting for API to ack qan_1507762440012218450: read tcp x.x.x.x:50062->x.x.x.x:80: i/o timeout
    2017/10/11 22:54:37.078760 WARNING data-sender Waiting for API to ack qan_1507762440012218450: read tcp x.x.x.x:50064->x.x.x.x:80: i/o timeout


    What could cause this? Is there any parameter we should tune?

    Thank you

  • #2
    Please share the output of pmm-admin check-network executed from the client side showing gaps in PMM Server.

    Can you also share the full command you've used to create the pmm-server container?

    Comment


    • #3
      Hi, sorry, I was out of town.

      Here is the command output:

      pmm-admin check-network
      PMM Network Status

      Server | x.x.x.x
      Client | x.x.x.x

      * Client --> Server
      --------------- -------------
      SERVER SERVICE CONNECTIVITY
      --------------- -------------
      Consul API 🙂
      QAN API 🙂
      Prometheus API 🙂

      Connection duration | 259.56µs
      Request duration | 365.425µs
      Full round trip | 624.985µs

      * Client <-- Server
      ------- ---------- ---------------------- -------------
      METRIC NAME PROMETHEUS ENDPOINT REMOTE STATE
      ------- ---------- ---------------------- -------------
      linux host1 x.x.x.x:42000 😡
      mysql host1 x.x.x.x:42002 🙂

      For endpoints in problem state, please check if the corresponding local service is running ("pmm-admin list").
      If yes and the endpoint references to a remote machine, ensure that machine is accessible from the current system.

      If all endpoints are down here and "pmm-admin list" shows they are up,
      check the firewall settings whether this system allows incoming connections from server by addressort in question.


      Here is the command we used to create the container:

      docker create \
      > -v /opt/prometheus/data \
      > -v /opt/consul-data \
      > -v /var/lib/mysql \
      > --name pmm-data \
      > percona/pmm-server:1.0.4 /bin/true

      docker run -d \
      > -p 80:80 \
      > --volumes-from pmm-data \
      > --name pmm-server \
      > --restart always \
      > percona/pmm-server:1.0.4


      Thank you

      Comment

      Working...
      X