Announcement

Announcement Module
Collapse
No announcement yet.

mysql TABLE optimisation

Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • mysql TABLE optimisation

    Hi All (please help)

    my mysql DB uas a table of the following structure:


    CREATE TABLE IF NOT EXISTS `content` (
    `VID` bigint(20) NOT NULL auto_increment,
    `UID` bigint(20) NOT NULL default '0',
    `provider` char(2) default NULL,
    `title` varchar(120) NOT NULL default '',
    `description` text NOT NULL,
    `featuredesc` text NOT NULL,
    `keyword` text NOT NULL,
    `channel` varchar(255) NOT NULL default '0|',
    `vdoname` varchar(40) NOT NULL default '',
    `flvdoname` varchar(40) default NULL,
    `adult` char(3) default 'no',
    `duration` float NOT NULL default '0',
    `space` bigint(20) NOT NULL default '0',
    `type` varchar(7) NOT NULL default '',
    `addtime` varchar(20) default NULL,
    `adddate` date NOT NULL default '0000-00-00',
    `record_date` date NOT NULL default '0000-00-00',
    `location` text NOT NULL,
    `country` varchar(120) NOT NULL default '',
    `vkey` varchar(20) NOT NULL default '',
    `viewnumber` bigint(10) NOT NULL default '0',
    `viewtime` datetime NOT NULL default '0000-00-00 00:00:00',
    `com_num` int(cool: NOT NULL default '0',
    `fav_num` int(cool: NOT NULL default '0',
    `featured` char(3) NOT NULL default 'no',
    `ratedby` bigint(10) NOT NULL default '0',
    `rate` float NOT NULL default '0',
    `filehome` varchar(120) NOT NULL default '',
    `be_comment` char(3) NOT NULL default 'yes',
    `be_rated` char(3) NOT NULL default 'yes',
    `embed` varchar(cool: NOT NULL default 'enabled',
    `voter_id` varchar(200) NOT NULL default '',
    `contentowner` varchar(120) default NULL,
    `adurl` varchar(255) default NULL,
    `admsg` text,
    `AnonName` varchar(255) default 'anonymous',
    PRIMARY KEY (`VID`),
    UNIQUE KEY `vkey` (`vkey`),
    KEY `VID` (`VID`)
    ) ENGINE=MyISAM DEFAULT CHARSET=latin1 AUTO_INCREMENT=79890 ;

    when i search this table using PHP using simple queries, my CPU utilisation grows exponentially. just 1 search query on this table causes my CPU to spike to 30%... and when i have multiple queries the CPU utilisation grows to 300%..

    could somone help me optimise this table please...
    (the number of rows in this table is > 100K)

  • #2
    logging slow queries on this DB.. here's what I got:
    (a few examples)

    SELECT keyword from content where adult<>"yes" and type='public' order by rand() ASC LIMIT 0, 25;
    # Time: 071122 4:18:34
    # User@Host: username[username] @ localhost []
    # Query_time: 2 Lock_time: 0 Rows_sent: 0 Rows_examined: 79270

    SELECT * from content where type='public' and (keyword like '%harbhajan%' or keyword like '%mann%'or keyword like '%man%'or keyword like '%jatt%'or keyword like '%sam%'or keyword like '%latest%'or keyword like '%punjabi%'or keyword like '%song%'or keyword like '%mitti%'or keyword like '%wajan%'or keyword like '%mardi%'or keyword like '%wajde%'or keyword like '%dhol%'or keyword like '%naal%'or keyword like '%algose%'or keyword like '%tumbi%'or keyword like '%upload%'or keyword like '%miti%'or keyword like '%vaja%'or keyword like '%vajan%') and adult<>"yes" order by VID asc limit 14619, 5;
    # Time: 071122 4:20:03
    # User@Host: username[username] @ localhost []
    # Query_time: 2 Lock_time: 0 Rows_sent: 10 Rows_examined: 79685

    Comment


    • #3
      i modified my table and added:

      KEY `INDEX` (`adult`,`featured`,`type`,`addtime`)


      still no use....

      here's my PHP select statements:
      $sql1="SELECT * from content where type='public' and adult='yes' and featured='yes' order by addtime desc limit 12";
      $rs1=$conn->Execute($sql1);
      if($rs1->recordcount()>0)
      $featuredcontent = $rs1->getrows();
      STemplate::assign('featuredcontent',$featuredconte nt); ---> i use smarty for the presentation...

      Comment


      • #4
        And your last query didn't gain speed?


        As for the two others.
        1.
        Try not to use negations like adult <> 'yes'.
        Change them to adult = 'no' instead.
        Index usage is much improved.

        2.
        An "... ORDER BY RAND()" can not use an index. Hence it will always require CPU to order the records. But note that the amount of records that needs sorting is determined from how many that match the WHERE and/or that are left after a GROUP BY statement since sorting is one of the last steps in a query execution.
        The only thing you can do to speed this query up is to make sure that you have a sort_buffer_size setting that is large enough to store the temporary table in RAM instead of having to write it to disk.

        Comment


        • #5
          didnt help... here's what I have now:

          1 PHP page:
          content as follows:

          include("include/config.php");
          include("include/function.php");
          $mydate= date('Y-m-d');

          $sql="SELECT CHID, name from channel";
          $rs = $conn->Execute($sql);
          $channels = $rs->getrows();
          STemplate::assign('channels',$channels);

          if($_SESSION['filterstatus']=="on"){
          $sql1="SELECT * from content where type='public' and (adult<>'yes' or adult is NULL) and featured=\"yes\" order by addtime desc limit 12";
          } else {
          $sql1="SELECT VID, title, from content where type='public' and adult='yes' and featured='yes' order by addtime desc limit 12";
          }
          $rs1=$conn->Execute($sql1);
          if($rs1->recordcount()>0)
          $featuredcontent = $rs1->getrows();
          STemplate::assign('featuredcontent',$featuredconte nt);


          STemplate::assign('page', 'index');
          STemplate::display('head.tpl');
          STemplate::display('err_msg.tpl');
          STemplate::display('index.tpl');
          STemplate::display('footer.tpl');
          ?>


          here's my function.php
          function insert_id_to_thumb($a)
          {
          global $config,$conn;

          $VID=$a[vid];
          $provider=$a[provider];
          $flvdoname=$a[flvdoname];
          $img=$a[img];
          if($img=="") $img=rand(1, 3);
          if($provider=="yt") $thumburl="http://img.youtube.com/vi/".$flvdoname."/".$img.".jpg";
          elseif($provider=="myspace") $thumburl="http://myspaceurl.com/vi/".$flvdoname."/".$img.".jpg";
          else $thumburl="/thumb/".$img."_".$VID.".jpg";
          return $thumburl;
          }

          function insert_video_channel($a)
          {
          global $conn;
          if($a[tbl]=="")$sqlx="channel from content where VID='$a[vid]'";
          else $sqlx="channel from $a[tbl] where GID='$a[gid]'";
          $sql="select $sqlx";
          $rs=$conn->execute($sql);
          $a=$rs->fields[channel];
          if($a!="")
          {
          $temp=explode("|",$a);
          if(count($temp)>=1) for($i=1;$i $sql="select CHID,name from channel where CHID=$temp[0] $list";
          $rsx=$conn->execute($sql);
          $res=$rsx->getrows();
          return $res;
          }
          }

          function insert_comment_count($a)
          {
          global $conn;
          $sql="select count(*) as ttl from comments where VID='$a[vid]'";
          $rs=$conn->execute($sql);
          return $rs->fields[ttl];
          }


          ?>


          and here's my index.tpl (smarty template)
          {section name=i loop=$channels}
        • {$channels[i].name}

        • {/section}


          {section name=i loop=12 start=0 max=12}
          {insert name=id_to_thumb assign=thumburl vid=$featuredcontent[i].VID provider=$featuredcontent[i].provider flvdoname=$featuredcontent[i].flvdoname}"{$featuredcontent[i].title|escape:"quotes"|escape:"htmlall
          {$featuredcontent[i].title|truncate:17: "...":true}

          {insert name=comment_count assign=commentcount vid=$featuredcontent[i].VID}Comments: {$commentcount}

          {insert name=video_channel assign=channel vid=$featuredcontent[i].VID}{section name=k loop=$channel start=0 max=1}{$channel[k].name}{/section}
          {/section}


          with this simple query: hre's what TOP shows me when i hit refresh on my page... just once:

          top - 11:27:48 up 7 days, 18:53, 1 user, load average: 0.00, 0.00, 0.00
          Tasks: 119 total, 2 running, 117 sleeping, 0 stopped, 0 zombie
          Cpu(s): 3.1%us, 4.4%sy, 0.0%ni, 92.5%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
          Mem: 32553000k total, 32021216k used, 531784k free, 1162548k buffers
          Swap: 2031608k total, 144k used, 2031464k free, 1640916k cached

          PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
          28452 mysql 18 0 184m 28m 3600 S 29 0.1 0:04.56 mysqld
          28504 apache 15 0 195m 12m 2992 S 2 0.0 0:00.06 httpd
          28498 apache 16 0 195m 11m 2580 S 1 0.0 0:00.03 httpd
          1 root 15 0 10312 684 572 S 0 0.0 0:01.65 init
          2 root RT 0 0 0 0 S 0 0.0 0:00.39 migration/0


          could someone help me structure my table right and enable me to display this 1 page without much CPU consumtion.. I can run with that.. but this 1 query is killing me...

        Comment


        • #6
          HELP !!! anybody !! somebody !!

          Comment


          • #7
            Please don't post a lot of PHP code because it really doesn't say anything.

            Do as you did posting the CREATE TABLE and then post the specific query that takes time and the explain plan for it.

            As for your CPU usage it is 7.5% not 29%. If you look at the top of the "top" output it says

            Cpu(s): 3.1%us, 4.4%sy, 0.0%ni, 92.5%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st

            which means that 7.5% is used and 92.5% is idle.

            The reason why mysqld is reporting 29% in the list is that the mysql process is in fact a _lot_ of threads and a rounding error for each thread makes the cpu usage to rise. But that is purely a display problem.
            The summarized cpu usage for the system is what you should look at and 7.5% isn't that high.

            Comment


            • #8
              One comment on the table design...

              Recommend splitting the table up so that there is a main
              table with a few commonly accessed fields and a child (dependent)
              table that has all the less commonly accessed fields. The goal
              is to slim down the number of columns in the main table.

              The row size is sooo big for this table that it will be
              slow to search as not many rows can be packed into a 16k
              block with a large row like this. This means that it will
              take a lot of disk accesses to search this table. Making
              it skinnier will improve search speed.

              You would want to put any fields that are searched for in
              the main table, or any fields that are always accessed in
              the main table.

              Comment

              • Working...
                X