Reclaim disk space from a sparse image file (qcow2/ vmdk)

  • 17

Reclaim disk space from a sparse image file (qcow2/ vmdk)

Get Social!

western-digital-diskSparse disk image formats such as qcow2 only consume the physical disk space which they need. For example, if a guest is given a qcow2 image with a size of 100GB but has only written to 10GB then only 10GB of physical disk space will be used. There is some slight overhead associated, so the above example may not be strictly true, but you get the idea.

Sparse disk image files allow you to over allocate virtual disk space – this means that you could allocate 5 virtual machines 100GB of disk space, even if you only have 300GB of physical disk space. If all the guests need 100% of their 100GB disk space then you will have a problem. If you use over allocation of disk space you will need to monitor the physical disk usage very carefully.

There is another problem with sparse disk formats, they don’t automatically shrink. Let’s say you fill 100GB of a sparse disk (we know this will roughly consume 100GB of physical disk space) and then delete some files so that you are only using 50GB. The physical disk space used should be 50GB, right? Wrong. Because the disk image doesn’t shrink, it will always be 100GB on the file system even if the guest is now using less. The below steps will detail how to get round this issue.

On Linux

We need to fill the disk of the guest with zero’s (or any other character) so that the disk image can be re-compressed.

In a terminal, run the below command until you run out of disk space. Before running this, be sure to stop any applications running on the guest otherwise errors may result.

Once the command errors out (this may take a while depending on your disk image size and physical disk speed) delete the file.

Shutdown the guest and follow the steps below under All OS’s.

On Windows

You will need to download a tool called sdelete from Microsoft which is will fill the entire disk with zeros which can be re-compressed later.

Download: http://technet.microsoft.com/en-gb/sysinternals/bb897443.aspx

Once you have downloaded and extracted sdelete, open up a command prompt and enter the following. This assumes that sdelete was extracted into c:\ and c:\ is the disk you would like to use to reclaim space

Once this completes (this may take a while depending on your disk image size and physical disk speed), shutdown the guest and follow the below steps under All OS’s.

All OS’s

The rest of the process is done on the host so open up a terminal window and SSH to your Proxmox host. Move to the directory where the disk image is stored and run the below commands.

Make sure you have shut down the virtual machine which is using the qcow2 image file before running the below commands.

The above commands move the original image file, and then re-compress it to it’s original name. This will shrink the qcow2 image to consume less physical disk space.

You can now start the guest and check that everything is in working order. If it is, you can remove the original_image.qcow2_backup file.


17 Comments

helpme!

9-Sep-2014 at 8:55 pm

hi,
         sorry for my English, hope you can help. By hize error “dd if=/dev/zero of=/mytempfile” on my server host and not the virtual machine, now I have no available space and errors accumulate, no way to reverse this situation?

    james.coyle

    10-Sep-2014 at 7:10 am

    You just need to remove the file you created.

    rm /mytempfile

helpme!

10-Sep-2014 at 3:11 pm

James, muchas gracias por la respuesta pensaba que había borrado la tabla de particiones o algo por el estilo ya que al hacer “fdisk -l” podía ver en las particiones “Disk /dev/mapper/pve-data doesn’t contain a valid partition table” y como es un servidor en producción no podia reiniciarlo para probar si bootea, Al mirar otro servidor Proxmox ( que uso para virtualizar con KVM) veo que todos son iguales. Asi que me quedo mas tranquilo. Muchas gracias.

H1de0

15-Apr-2015 at 12:05 pm

You sir just saved my day:D Until today growing qcow2 sparse images where a major problem on my servers.

One thing I would add, is the info that the “qemu-img convert -O” command does not remove the original image, but creates a new compressed one while leaving the old one unchanged.

Ron Arts

2-Jun-2015 at 12:22 pm

Your Windows command is wrong. It should be:

sdelete -z c:

    james.coyle

    2-Jun-2015 at 12:34 pm

    Updated typo – thanks.

    Greg Guilmette

    2-Sep-2015 at 7:43 pm

    I disagree… for the purpose of this article, “-c” should be used for zeroing out the slack space. If you use “-z”, you get random strings overwriting the space for security purposes.
    The “-z” would not be conducive to making the space compressible/reclaimable.

      John Airey

      5-Dec-2016 at 12:02 pm

      sdelete64.exe /?

      SDelete v2.0 – Secure file delete
      Copyright (C) 1999-2016 Mark Russinovich
      Sysinternals – http://www.sysinternals.com

      usage: sdelete [-p passes] [-r] [-s] [-q] […]
      sdelete [-p passes] [-z|-c [percent free]]
      sdelete [-p passes] [-z|-c]
      -c Clean free space. Specify an option amount of space
      to leave free for use by a running system.
      -p Specifies number of overwrite passes (default is 1)
      -r Remove Read-Only attribute
      -s Recurse subdirectories
      -z Zero free space (good for virtual disk optimization)
      -nobanner Do not display the startup banner and copyright message.

      Disks must not have any volumes in order to be cleaned.

dudleyperkins

26-Jun-2015 at 2:10 pm

I’d like to add one thing:

Proxmox creates qcow2-disks with pre-allocated metadata (afaik for performance reasons) if you use the convert command without the option “-o reallocation=metadata” you’ll loose that advantage(?)

What’s your opinion on that?

    james.coyle

    29-Jun-2015 at 3:26 pm

    Hi,

    I agree – you sacrifice a small amount of space for the gain of not needing to create the metadata for the disk.

    You should probably include the switch you specified (-o reallocation=metadata) if you want to maintain performance at the cost of a small amount of disk space. This would likely only be an issue if you’re going to be writing large amounts of data to the disk after this process.

    I’ll try and do some benchmarking shortly and write another post.

      dudleyperkins

      14-Jul-2015 at 1:11 pm

      If you have time for that – this would be very interesting, thank you

Duncan

27-Apr-2016 at 10:19 pm

This doesn’t appear to work for me. I did the dd route but the qemu-img info shows the resulting .qcow still having the same bloated 80G virtual size, even though the file itself is only 1.5G.

    BaRoN.su

    27-May-2016 at 11:35 am

    That’s exactly how sparse file is supposed to work. It only consumes small amount of disk space and gets bigger and bigger as needed.

AMIT PATHANIA

13-Oct-2016 at 8:46 am

I have deleted one VM from kvm using virt manager but the disk space allotted to the VM is not returned back ie it’s not visible.How to regain the disk space ?

Shane

2-Feb-2017 at 7:44 pm

You might wish to check out the “virt-sparsify” command which is included in the “libguestfs-tools”. “virt-sparsify” is designed to operate on a wide range of Virtual Machines. More details can be found at: http://libguestfs.org/virt-sparsify.1.html

Roshan Singh

30-Mar-2017 at 5:52 pm

How safe it is to do it for virtualized domain controller (Windows 2008 R2) ?

We have 600GB DC (which was filer server as well) and we moved all 400GB files from it to another VM. Now it is 200GB but as you said actually it is still 600GB if you move it or take its image backup.

Please assist

    james.coyle

    31-Mar-2017 at 1:10 pm

    Should be fine – as always, take a backup first.

Leave a Reply