GET 24/7 LIVE HELP NOW

Announcement

Announcement Module
Collapse
No announcement yet.

pt-Archiver very slow!!

Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • pt-Archiver very slow!!

    I am trying to archive a 21 Million row table using pt-archiver on a 512MB Amazon EC2 instance
    and it is running very slow
    the table is actively written into by my applications users as we are archiving it, could this be slowing down the archiving?

    here are my archiving options

    pt-archiver -u $MyUSER -p$MyPASS --source h=$MyHOST,D=$DEST_DB,t=$DEST_TABLE --purge --primary-key-only --why-quit --progress 10000 --statistics --charset utf8 --retries 3 --commit-each --limit 3000 --where "REQUEST_TIME <= DATE_SUB(NOW(), INTERVAL 60 DAY)"


    here is the log output when i kill the job after few hours

    TIME ELAPSED COUNT
    2014-04-29T15:46:07 0 0
    2014-04-29T16:17:21 1874 10000
    2014-04-29T16:36:18 3011 20000
    2014-04-29T16:55:19 4152 30000
    2014-04-29T17:14:17 5289 40000
    2014-04-29T17:42:40 6992 50000
    2014-04-29T18:11:04 8697 60000
    2014-04-29T18:33:02 10015 70000
    2014-04-29T18:52:03 11156 80000
    2014-04-29T19:15:55 12588 90000
    2014-04-29T19:33:00 13613 100000
    2014-04-29T19:57:07 15059 110000
    2014-04-29T20:25:27 16760 120000
    2014-04-29T20:44:56 17929 130000
    2014-04-29T21:06:48 19241 140000
    2014-04-29T21:35:12 20945 150000
    2014-04-29T21:52:58 22011 160000
    2014-04-29T22:20:35 23667 170000
    2014-04-29T22:39:26 24799 180000



  • #2
    Another run and same scenario, it slows down after a particular point
    TIME ELAPSED COUNT
    2014-05-01T12:23:05 0 0
    2014-05-01T12:23:08 2 10000
    2014-05-01T12:23:10 4 20000
    2014-05-01T12:23:13 7 30000
    2014-05-01T12:23:15 9 40000
    2014-05-01T12:23:17 12 50000
    2014-05-01T12:23:20 14 60000
    2014-05-01T12:23:22 17 70000
    2014-05-01T12:23:25 19 80000
    2014-05-01T12:23:27 22 90000
    2014-05-01T12:23:30 24 100000
    2014-05-01T12:23:32 27 110000
    2014-05-01T12:23:35 29 120000
    2014-05-01T12:23:48 42 130000
    2014-05-01T12:24:12 66 140000
    2014-05-01T12:24:36 90 150000
    2014-05-01T12:25:01 116 160000
    2014-05-01T12:25:26 140 170000
    2014-05-01T12:25:49 164 180000
    2014-05-01T12:26:14 188 190000
    2014-05-01T12:26:38 212 200000
    2014-05-01T12:27:02 237 210000
    2014-05-01T12:27:27 261 220000
    2014-05-01T12:27:51 285 230000
    2014-05-01T12:28:15 309 240000
    2014-05-01T12:28:39 334 250000
    2014-05-01T12:29:04 359 260000
    2014-05-01T12:29:28 383 270000
    2014-05-01T12:29:53 407 280000
    2014-05-01T12:30:17 432 290000
    2014-05-01T12:30:42 456 300000
    2014-05-01T12:31:06 481 310000
    2014-05-01T12:31:31 505 320000
    2014-05-01T12:31:55 529 330000
    2014-05-01T13:12:23 2958 340000
    2014-05-01T13:54:45 5499 350000
    2014-05-01T14:32:16 7750 360000
    2014-05-01T15:09:48 10002 370000
    2014-05-01T15:40:48 11863 380000
    2014-05-01T16:18:20 14114 390000


    Comment


    • #3
      another output
      Starting PT-Archiver NOW
      TIME ELAPSED COUNT
      2014-05-03T21:30:08 0 0
      2014-05-03T21:30:33 24 50000
      2014-05-03T21:30:55 46 100000
      2014-05-03T21:31:17 68 150000
      2014-05-03T21:31:39 90 200000
      2014-05-03T21:32:00 112 250000
      2014-05-03T21:32:21 133 300000
      2014-05-03T21:32:42 154 350000
      2014-05-03T21:33:04 176 400000
      2014-05-03T21:33:26 197 450000
      2014-05-03T21:33:47 218 500000
      2014-05-03T21:34:08 239 550000
      2014-05-03T21:34:29 260 600000
      2014-05-03T21:34:50 282 650000
      2014-05-03T21:35:13 305 700000
      2014-05-03T23:47:39 8250 750000
      2014-05-04T06:56:48 34000 800000
      2014-05-04T12:45:48 54940 850000
      2014-05-04T19:19:14 78546 900000
      2014-05-05T02:33:44 104616 950000
      2014-05-05T10:19:10 132542 1000000
      2014-05-05T15:52:15 152526 1032065


      Comment

      Working...
      X