Emergency

Lost data when deploying PMM into vmware machines

Collapse
X
  • Filter
  • Time
  • Show
Clear All
new posts

  • Lost data when deploying PMM into vmware machines

    This is what my grafana dashboard looks like:


    I deployed it as docker image first, the uncontinuously lines came out and I thought it may be the heavy load.
    Then I tried to deploy it to ESXi virtual machine and increased the CPUs&Mem, the same happened.

    At the end, I run in VirtualBox, everything is perfect, the graph seems continuously.

    Then I realized that my docker host is a ESXi vm. So I get to this conclusion: PMM-Server does not work well with vmware machines.

    Could someone help me what have I miss ?

  • #2
    Hi seanlook,

    Prometheus fetches data every 1s from all mysql hosts, so it requires very good network performance and very fast VM reaction.
    if ESXi has any network/CPU/memory delays Prometheus can ignores monitoring results.

    please check state of Targets - http://PMM-SERVER-IP/prometheus/targets, all of them should be "UP".
    please try to refresh this page at least 3-5 times with several seconds delay between refresh.

    Comment


    • #3
      Originally posted by Mykola View Post
      Hi seanlook,

      Prometheus fetches data every 1s from all mysql hosts, so it requires very good network performance and very fast VM reaction.
      if ESXi has any network/CPU/memory delays Prometheus can ignores monitoring results.

      please check state of Targets - http://PMM-SERVER-IP/prometheus/targets, all of them should be "UP".
      please try to refresh this page at least 3-5 times with several seconds delay between refresh.

      Thanks for your reply. I see lots of `UNKNOW` state.
      I can tell pmm-server and client network traffic status is not the proplem. MySQL instance node load is not heavy.

      I tried to visit client port:
      https://pmm-client-ip:42000/metrics -- fast
      https://pmm-client-ip:42002/metrics-mr -- very slow

      It seems say that all mysqld_exporter response slowly at the same time.

      Comment


      • #4
        /metrics-mr fetch time should be less than 1 seconds

        can you show output of the following command?

        Code:
          wget https://PMM-CLIENT-IP:42002/metrics-mr --no-check-certificate
        I am wondering response size and time consumed to fetch.

        Comment


        • #5
          As attached.

          Linux metrics should be independent with MySQL metrics. But both OS panel and MySQL panel data lost.

          I've changed the scrape_interval=10s, scrape_timeout=5s, still uncontinuously. Within virtualbox, nothing lost.

          Comment


          • #6
            can you share /var/log/prometheus.log ?

            Comment

            Working...
            X