Guide to LVM
From Initq
This guide shows how to work with LVM (Logical Volume Management) on Linux. It also describes how to use LVM together with RAID1.
Install all the tools
I installed all tools we need during the course of this guide on the Debian Etch system.
apt-get install lvm2 dmsetup mdadm reiserfsprogs xfsprogs
Creating the LVM partition
Let's first check and see what we have on our system.
[root@initq ~]# fdisk -l Disk /dev/sda: 160.0 GB, 160041885696 bytes 255 heads, 63 sectors/track, 19457 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x0004e546 Device Boot Start End Blocks Id System /dev/sda1 * 1 17150 137757343+ 83 Linux /dev/sda2 17151 19457 18530977+ 82 Linux swap / Solaris Disk /dev/sdb: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdb1 1 243205 1953544131 83 Linux Disk /dev/sdc: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x5ac88679 Device Boot Start End Blocks Id System /dev/sdc1 1 60801 488384001 83 Linux Disk /dev/sdd: 1500.3 GB, 1500320366592 bytes 64 heads, 32 sectors/track, 1430817 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Disk identifier: 0x064038e9 Device Boot Start End Blocks Id System /dev/sdd1 1 1430817 1465156592 83 Linux Disk /dev/sde: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0xa4b57300 Device Boot Start End Blocks Id System /dev/sde1 1 60801 488384001 83 Linux
Let's work on sdb the 500 gig drive. As you can see there is no partition on the drive and even if there was, please delete it.
[root@initq ~]# fdisk /dev/sdb The number of cylinders for this disk is set to 60801. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Command (m for help): p Disk /dev/sdb: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdb1 1 243205 1953544131 83 Linux
I always print this menu with the m command to make sure i am using the right letter.
Command (m for help): m Command action a toggle a bootable flag b edit bsd disklabel c toggle the dos compatibility flag d delete a partition l list known partition types m print this menu n add a new partition o create a new empty DOS partition table p print the partition table q quit without saving changes s create a new empty Sun disklabel t change a partition's system id u change display/entry units v verify the partition table w write table to disk and exit x extra functionality (experts only)
We now delete the old partition.
Command (m for help): d Selected partition 1
Then we will add a new partition.
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-60801, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-60801, default 60801):
Using default value 60801
Now we need to change that partition to the LVM partition. I know the partition id so i type it in but if you don't then use the L command to list all the partition id's.
Command (m for help): t Selected partition 1 Hex code (type L to list codes): 8e Changed system type of partition 1 to 8e (Linux LVM)
We now print the partition and then write it to the disk.
Command (m for help): p Disk /dev/sdb: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdb1 1 60801 488384001 8e Linux LVM Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks.
Check your cration one more time.
[root@initq ~]# fdisk -l /dev/sdb Disk /dev/sdb: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdb1 1 60801 488384001 8e Linux LVM
We just created LVM on /dev/sdb. We will now do the exact same thing to /dev/sdc and /dev/sde.
[root@initq ~]# fdisk -l /dev/sdb /dev/sdc /dev/sde Disk /dev/sdb: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/sdb1 1 60801 488384001 8e Linux LVM Disk /dev/sdc: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x5ac88679 Device Boot Start End Blocks Id System /dev/sdc1 1 60801 488384001 8e Linux LVM Disk /dev/sde: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0xa4b57300 Device Boot Start End Blocks Id System /dev/sde1 1 60801 488384001 8e Linux LVM
Creating Physical Volumes
pvcreate and pvremove are the two commands to create and remove physical volumes.
[root@initq ~]# pvcreate /dev/sdb1 /dev/sdc1 /dev/sde1 Physical volume "/dev/sdb1" successfully created Physical volume "/dev/sdc1" successfully created Physical volume "/dev/sde1" successfully created [root@initq ~]# pvremove /dev/sdb1 /dev/sdc1 /dev/sde1 Labels on physical volume "/dev/sdb1" successfully wiped Labels on physical volume "/dev/sdc1" successfully wiped Labels on physical volume "/dev/sde1" successfully wiped [root@initq ~]# pvcreate /dev/sdb1 /dev/sdc1 /dev/sde1 Physical volume "/dev/sdb1" successfully created Physical volume "/dev/sdc1" successfully created Physical volume "/dev/sde1" successfully created
Display the Volumes
The pvdisplay command will display all the volumes you created.
[root@initq ~]# pvdisplay "/dev/sdb1" is a new physical volume of "465.76 GB" --- NEW Physical volume --- PV Name /dev/sdb1 VG Name PV Size 465.76 GB Allocatable NO PE Size (KByte) 0 Total PE 0 Free PE 0 Allocated PE 0 PV UUID 6G9Yhg-bl1R-3DSG-UClz-VwSr-QO5D-MIszqL "/dev/sdc1" is a new physical volume of "465.76 GB" --- NEW Physical volume --- PV Name /dev/sdc1 VG Name PV Size 465.76 GB Allocatable NO PE Size (KByte) 0 Total PE 0 Free PE 0 Allocated PE 0 PV UUID rZYDPG-HgGk-9Mz0-Ic30-HNER-ym3x-YK1mGF "/dev/sde1" is a new physical volume of "465.76 GB" --- NEW Physical volume --- PV Name /dev/sde1 VG Name PV Size 465.76 GB Allocatable NO PE Size (KByte) 0 Total PE 0 Free PE 0 Allocated PE 0 PV UUID iJ1Chh-18HX-pEGq-7fZb-6san-HSfN-UGipG0
Volume Group Creation
[root@initq ~]# vgcreate initq_vg /dev/sdb1 /dev/sdc1 /dev/sde1 Volume group "initq_vg" successfully created
Display the Volume group
[root@initq ~]# vgdisplay --- Volume group --- VG Name initq_vg System ID Format lvm2 Metadata Areas 3 Metadata Sequence No 1 VG Access read/write VG Status resizable MAX LV 0 Cur LV 0 Open LV 0 Max PV 0 Cur PV 3 Act PV 3 VG Size 1.36 TB PE Size 4.00 MB Total PE 357702 Alloc PE / Size 0 / 0 Free PE / Size 357702 / 1.36 TB VG UUID Vd65gI-1TL6-6XiD-KY2G-bwIk-wKJK-VtD8zw
Another command to display vg info is vgscan.
[root@initq ~]# vgscan Reading all physical volumes. This may take a while... Found volume group "initq_vg" using metadata type lvm2
Rename, Delete and Create Volume Group
[root@initq ~]# vgdisplay --- Volume group --- VG Name initq_vg System ID Format lvm2 Metadata Areas 3 Metadata Sequence No 1 VG Access read/write VG Status resizable MAX LV 0 Cur LV 0 Open LV 0 Max PV 0 Cur PV 3 Act PV 3 VG Size 1.36 TB PE Size 4.00 MB Total PE 357702 Alloc PE / Size 0 / 0 Free PE / Size 357702 / 1.36 TB VG UUID Vd65gI-1TL6-6XiD-KY2G-bwIk-wKJK-VtD8zw [root@initq ~]# vgrename initq_vg initq_new Volume group "initq_vg" successfully renamed to "initq_new" [root@initq ~]# vgdisplay --- Volume group --- VG Name initq_new System ID Format lvm2 Metadata Areas 3 Metadata Sequence No 2 VG Access read/write VG Status resizable MAX LV 0 Cur LV 0 Open LV 0 Max PV 0 Cur PV 3 Act PV 3 VG Size 1.36 TB PE Size 4.00 MB Total PE 357702 Alloc PE / Size 0 / 0 Free PE / Size 357702 / 1.36 TB VG UUID Vd65gI-1TL6-6XiD-KY2G-bwIk-wKJK-VtD8zw [root@initq ~]# vgremove initq_new Volume group "initq_new" successfully removed [root@initq ~]# vgdisplay [root@initq ~]# vgcreate initq_vg /dev/sdb1 /dev/sdc1 /dev/sde1 Volume group "initq_vg" successfully created [root@initq ~]# vgdisplay --- Volume group --- VG Name initq_vg System ID Format lvm2 Metadata Areas 3 Metadata Sequence No 1 VG Access read/write VG Status resizable MAX LV 0 Cur LV 0 Open LV 0 Max PV 0 Cur PV 3 Act PV 3 VG Size 1.36 TB PE Size 4.00 MB Total PE 357702 Alloc PE / Size 0 / 0 Free PE / Size 357702 / 1.36 TB VG UUID GAQwr0-hP12-OuvU-LGre-CxVp-HZHM-YvZfpi
Create Logical Volume
Next we create our logical volume called junk (50 Gig).
[root@initq ~]# lvcreate --name junk --size 50G initq_vg Logical volume "junk" created [root@initq ~]# lvdisplay --- Logical volume --- LV Name /dev/initq_vg/junk VG Name initq_vg LV UUID 3G50Vp-bV5E-eah2-9fh2-fe8Y-LiHF-My2AiK LV Write Access read/write LV Status available # open 0 LV Size 50.00 GB Current LE 12800 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 252:0 [root@initq ~]# lvscan ACTIVE '/dev/initq_vg/junk' [50.00 GB] inherit
Extend and Reduce Volume size
[root@initq ~]# lvscan ACTIVE '/dev/initq_vg/junk' [50.00 GB] inherit [root@initq ~]# [root@initq ~]# [root@initq ~]# lvextend -L55G /dev/initq_vg/junk Extending logical volume junk to 55.00 GB Logical volume junk successfully resized [root@initq ~]# lvscan ACTIVE '/dev/initq_vg/junk' [55.00 GB] inherit [root@initq ~]# lvreduce -L30G /dev/initq_vg/junk WARNING: Reducing active logical volume to 30.00 GB THIS MAY DESTROY YOUR DATA (filesystem etc.) Do you really want to reduce junk? [y/n]: y Reducing logical volume junk to 30.00 GB Logical volume junk successfully resized [root@initq ~]# lvscan ACTIVE '/dev/initq_vg/junk' [30.00 GB] inherit
Create File system on Logical Volume
[root@initq ~]# mkfs.ext3 /dev/initq_vg/junk
mke2fs 1.41.2 (02-Oct-2008)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
3276800 inodes, 13107200 blocks
655360 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
400 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 37 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
Mount The file system
[root@initq ~]# mkdir /junk [root@initq ~]# mount /dev/init initctl initq_vg/ [root@initq ~]# mount /dev/init initctl initq_vg/ [root@initq ~]# mount /dev/initq_vg/junk /junk [root@initq ~]# mount /dev/sda1 on / type ext3 (rw,relatime) none on /proc type proc (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) /dev/sdd1 on /initq type ext3 (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) 192.168.0.15:/qasket on /qasket type nfs (rw,addr=192.168.0.15) //leechfinger.com/shared on /mnt/leechfinger type cifs (rw,mand) /dev/mapper/initq_vg-junk on /junk type ext3 (rw)
[root@initq ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 130G 13G 111G 10% /
/dev/sdd1 1.4T 838G 538G 61% /initq
192.168.0.15:/qasket 1.2T 871G 294G 75% /qasket
//leechfinger.com/shared
49G 7.2G 41G 15% /mnt/leechfinger
/dev/mapper/initq_vg-junk
50G 180M 47G 1% /junk
Resizing your Logical Volume
You should always umount the filesystem before you make any changes. First we will make the changes without umount.
[root@initq ~]# lvextend -L55G /dev/initq_vg/junk
Extending logical volume junk to 55.00 GB
Logical volume junk successfully resized
[root@initq ~]# lvdisplay
--- Logical volume ---
LV Name /dev/initq_vg/junk
VG Name initq_vg
LV UUID 3G50Vp-bV5E-eah2-9fh2-fe8Y-LiHF-My2AiK
LV Write Access read/write
LV Status available
# open 1
LV Size 55.00 GB
Current LE 14080
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 252:0
[root@initq ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 130G 13G 111G 10% /
/dev/sdd1 1.4T 838G 538G 61% /initq
192.168.0.15:/qasket 1.2T 871G 294G 75% /qasket
//leechfinger.com/shared
49G 7.2G 41G 15% /mnt/leechfinger
/dev/mapper/initq_vg-junk
50G 180M 47G 1% /junk
[root@initq ~]# e2fs
e2fsck e2fsck.static
[root@initq ~]# e2fsck -f /dev/initq_vg/junk
e2fsck 1.41.2 (02-Oct-2008)
/dev/initq_vg/junk is mounted.
WARNING!!! Running e2fsck on a mounted filesystem may cause
SEVERE filesystem damage.
Do you really want to continue (y/n)? no
check aborted.
[root@initq ~]# resi
resize2fs resize_reiserfs
[root@initq ~]# resize2fs /dev/initq_vg/junk
resize2fs 1.41.2 (02-Oct-2008)
Filesystem at /dev/initq_vg/junk is mounted on /junk; on-line resizing required
old desc_blocks = 4, new_desc_blocks = 4
Performing an on-line resize of /dev/initq_vg/junk to 14417920 (4k) blocks.
The filesystem on /dev/initq_vg/junk is now 14417920 blocks long.
[root@initq ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 130G 13G 111G 10% /
/dev/sdd1 1.4T 838G 538G 61% /initq
192.168.0.15:/qasket 1.2T 871G 294G 75% /qasket
//leechfinger.com/shared
49G 7.2G 41G 15% /mnt/leechfinger
/dev/mapper/initq_vg-junk
55G 180M 52G 1% /junk
You should always umount the filesystem before you resize it. Here are the commands you need to achieve that.
- umount /dev/initq_vg/junk
- lvextend -L55G /dev/initq_vg/junk
- e2fsck -f /dev/initq_vg/junk
- resize2fs /dev/initq_vg/junk
How to replace a drive in the Volume Group
- fisk /dev/sdf; d; n; t 8e; w;
- pvcreate /dev/sdf1
- vgextend fileserver /dev/sdf1
- pvmove /dev/sdb1 /dev/sdf1
- vgreduce fileserver /dev/sdb1
- pvremove /dev/sdb1
Remove all LVM's
- df -h
- umount /all-lvm's
- lvremove /dev/initq_vg/junk
- lvdisplay
- vgremove initq_vg
- vgdisplay
- pvremove /dev/sdb1 /dev/sdc1
- pvdisplay
- mv /etc/fstab_original /etc/fstab
- shutdown -r now
LVM over Raid
Now we will show you how to do LVM over raid drive. In the below diagram, we first create two raid drives md0 and md1 and then we create our physical volumes and then one volume group. Then we create our logical volumes on that volume group.
File:Lvm scheme full raid1.png
We are going to show you two methods of creating LVM over raid. First method will be straight forward where we start from scratch as if there was nothing on our drives and we were not concerned about the data. The second method will show you ways on how to safely protect your data while creating a raid and under an existing lvm.
LVM on Raid Complete command set
- always do the following two commands on all new harddrives to ensure that there is no junk in the first sector.
- dd if=/dev/zero of=/dev/diskname bs=1k count=1
- blockdev -rereadpt /dev/diskname
- create fd partition on all 4 drives, sda, sdb, sdc and sdd.
- fdisk /dev/sda; d; n; t fd; w;
- create Raid 1 on sda1 and sdb1
- mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1
- create Raid 1 on sdc1 and sdd1
- mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sdc1 /dev/sdd1
- create lvm partition on /dev/md0 and md1
- fdisk /dev/md0; d; n; t 8e; w;
- create physical volume
- pvcreate /dev/md0 /dev/md1
- create volume group
- vgcreate fileserver /dev/md0 /dev/md1
- create logical volume on volume group fileserver
- lvcreate --name junk --size 50G fileserver
- create file system on the partition before you could mount
- mkfs.ext3 /dev/fileserver/junk
- mount the partition
- mkdir /junk; mount /dev/fileserver/junk /junk
- check raid
- cat /proc/mdstat
That is it, now you have a LVM on top of a Raid 1.
Move Live volumes to Raid drives
Lets set up lvm first.
pvcreate /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 vgcreate fileserver /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 lvcreate --name share --size 40G fileserver lvcreate --name backup --size 5G fileserver lvcreate --name media --size 1G fileserver
Make the file systems on each one.
mkfs.ext3 /dev/fileserver/share mkfs.xfs /dev/fileserver/backup mkfs.reiserfs /dev/fileserver/media
mount them all.
mount /dev/fileserver/share /var/share mount /dev/fileserver/backup /var/backup mount /dev/fileserver/media /var/media
Now we remove two drives so we can start making our Raid 1.
pvmove /dev/sdc1 vgreduce fileserver /dev/sdc1 pvremove /dev/sdc1 pvmove /dev/sde1 vgreduce fileserver /dev/sde1 pvremove /dev/sde1
- fdisk /dev/sdc; d;n; t fd; w;
- fdisk /dev/sde; d; n; t fd; w;
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdc1 missing mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sde1 missing
pvcreate /dev/md0 /dev/md1 pvmove /dev/sdb1 /dev/md0 pvmove /dev/sdd1 /dev/md1 vgreduce fileserver /dev/sdb1 /dev/sdd1 pvremove /dev/sdb1 /dev/sdd1
- fdisk /dev/sdb; d;n; t fd; w;
- fdisk /dev/sdd; d; n; t fd; w;
mdadm --manage /dev/md0 --add /dev/sdb1 mdadm --manage /dev/md1 --add /dev/sdd1
You can now check you mdadm and mounts.
- cat /proc/mdstat
- mount
- df -h
Turn off Clustering
If your lvm is stuck in clustering then run:
- lvmconf --disable-cluster
This will make the appropriate changes to the lvm.conf file (locking_type = 1)
then turn off:
- clvmd
If the above method does not work then do the following.
- edit /etc/lvm/lvm.conf and put locking_type = 0
- vgchange -c n appl_2 (appl_2) is the vgname
- fix up your /etc/fstab and reboot.
That will bring the mounts back to normal not cluster state.
Cheat Sheet
- echo "- - -" > /sys/class/scsi_host/host0/scan
- pvcreate /dev/<somethging>
- vgextend <vg name> /dev/<something> or vgcreate <vg_name> /dev/<something>
- lvextend -L+12G /dev/vg_name or lvcreate --name <lv name> --size 12G vg_name
- resize2fs /dev/vg_name/_lv_name
Other commands to find space issues
- du -sh *
- fuser -c /tmp
- lsof | grep /tmp
