One of our customers was setting up box for MySQL LVM backups. The box had 2 RAID volumes (internal drives and enclosure) with database being stored on enclosure drives and internal drives used pretty much for OS and logs (database grew too large and so was moved to enclosure completely).
As database was large and active I thought it would make sense to have LVM undo space located on the internal drives so LVM and backup has less backup on the running system.
As the system was configured creating snapshot volume with 100G of undo space failed complaining there is not enough space available on the volume….
The problem was internal and external volumes were configured as different “volume groups”. Volume groups in LVM live completely isolated life and you can’t use space from one volume group for snapshot for another.
What should have done instead is creating single volume group and having RAID volumes as physical volumes.
If you also have done such mistake there is vgmerge command which allows to merge volume groups together without data loss. (Though I took backup anyway just to be on the safe side)
P.S That would be interested to benchmark one day how LVM performs in shapshot mode in case undo space is stored on the same device vs different device.
About the Author
Peter ZaitsevPeter managed the High Performance Group within MySQL until 2006, when he founded Percona. Peter has a Master's Degree in Computer Science and is an expert in database kernels, computer hardware, and application scaling.
I have made the same mistake recently. But in my case creating a single volume group was no option, and I think this is often the case. I’ll explain.
The server in question was configured with 6 physical harddrives. The first two harddrives (146GB@10k) were on the first channel of the hardware RAID controller and the other four drives (72GB@15k) on the second channel. Next we created 3 hardware RAID1 volumes, giving a 146GB system volume and two 72GB data volumes. Finally we merged the two 72GB data volumes into one using software RAID0, because some benchmarking showed that software RAID0 performed better than hardware RAID0 in our case.
Now onto the LVM configuration. We had only two physical volumes left: a ‘slow’ 146GB system volume and a ‘fast’ 72GB data volume. The idea was to put the snapshots for the data volume onto the system volume. We came to the same conclusion as you: creating a snapshot for a logical volume must be done within the same volume group. Merging the two physical volumes into one volume group was no option because of the difference in drive speeds and intention of those physical volumes.
The end result is that we stuck to two physical volumes, two volume groups and two logical volumes. We had to reduce the size of the data logical volume to be able to make snapshots. Most of the part of the system volume is now unused and a waste of money.
Anyone know of a different way we could have solved this?
M. van der Klip,
I do not understand why was not it option ?
If you have 2 physical volumes in one volume group you can still have separate logical volumes created on these volume groups.
Hmmmm.. I setup the exact same configuration last week but haven’t tried snapshotting yet. And now I stumble on this. Thanks! That’s gonna save me some time. Long live the RSS feed.
As far as I know it is not possible to fix a logical volume to a specific physical volume, without creating multiple volume groups. We NEEDED to ensure the data to be on the fast disks. Could we have done this otherwise in your opinion? I would be interested in hearing so, because that would mean we could use the wasted space after all…
Well,
You can use the fact logical volume groups do not roam around 🙂
Create slow volume (w OS first), when create temporary volume to consume all space on slow physical volume. Add fast physical volume to the same volume group. Create your MySQL LVM Volume on it. Now drop the temporary “space consumer” you had on the slow volume and use that space for undo.
In fact, lvextend accepts a name of PV as last parameter, so you can have many disks in VG and still have control over which LVOL is on which PV.
But I would suggest to have separate system VG anyway. It’s much more manageable and easier to restore in case of any troubles.
Marki,
Ah right. Silly me. I see lvcreate also allows that 🙂
I think using different VG or same depends on your priorities. If you want to say move shelf of drives between servers you would want it to be separate volume group while if you would like have more flexible space usage between physical volumes one group can be your choice.
Ah, great, that lvmerge/lvcreate parameter seems to do the trick indeed! 🙂
In my experience, lvm snapshots have a serious performance impact for writes to the parent LVM. Just for kicks I tested unpacking a bz2 of the linux kernel in different scenarios. Command was akin to:
time tar xvfj linux.tar.bz2 ; time sync
No snapshot: 17s
Regular snapshot: 61s
Snapshot with chunk size of 512k: 47s
I unmount/mounted the filesystem between each test to clear the fs cache.
Thanks for info.
Did you use LVM undo on the same or different physical device ?
In general it would be interesting to see the data on workload more typical for database which is quite different. For example innodb log is circulary written.