OpenVZ disk space issues

OpenVZ disk space issues

Get Social!

proxmox logo gradI started to receive an error in one of my Proxmox OpenVZ containers which indicated that the quota had been exceeded for disk space. Immediately I logged into the container and performed a df -h to view the disk usage. To my surprise only 50% of the disk space had been used.

Error in the logs:

kernel: VZ QUOTA: file softlimit exceeded for id=103

OpenVZ containers have two storage related limits;

  • Disk space – this is the amount of space which can be used by the container, usually measured in gigabytes.
  • inodes – this loosely translates to the number of files which can be stored on the filesystem.

Below is an example of the disk based limits configuration.

DISKSPACE="10485760:11534336"
DISKINODES="2000000:2200000"

The problem with my container was that although I wasn’t using all the allocated disk space, I was using all of the allocated inodes as I had a java application running which created 1000s of new small files every hour. CCTV recording software such as Zoneminder would likely have the same problem as it creates many small image files when recoding movement from a camera.

You can test how many inodes are in use by running df -i on the container.  An example below of a very low inode usage:

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/simfs 10000000 116024 9883976 2% /

Increase disk space and inode quota

Proxmox does not allow you to change the inodes limits directly from the GUI. You will need to edit the configuration file for the container directly on the OS or use the vz command. The config files are stored in /etc/vz/conf by default and named VMID.conf. Edit the required config file for your container and increase the DISKSPACE and DISKINODES values as required.

vi /etc/vz/conf/103.conf
DISKSPACE="10485760:11534336"
DISKINODES="4000000:4400000"

OpenVZ has the concept of HARD and SOFT limits. A soft limit will trigger a warning but allow the applications to continue to use disk space/ inodes however when the hard limit is hit new requests for storage will be denied. A hard limit is like a physical disk being full.

DISKSPACE="SOFT:HARD"
DISKINODES="SOFT:HARD"

You can also use the word UNLIMITED to remove any virtual limits on storage however you will still be constrained by the underlying physical storage limits.

You will need to restart your container for the settings to take effect.

To avoid having to restart your container, you can use the vzctl command to apply the new limits immediately, even to a running container.

vzctl set 103 --diskspace 45G:49G --save
vzctl set 103 --diskinodes 4000000:4400000 --save

Backup all Proxmox OpenVZ containers in one go

Category : How-to

Get Social!

proxmox logo gradThe below script is a bash script which works with Proxmox and the OpenVZ commands to backup all known containers to a specified folder.

You will need to set BACKUP_PATH to be the folder where you would like the backups to be stored and COMPRESS to a value which specifies the compression used. COMPRESS values can be:

  • 0 – no compression.
  • 1 – default compression (usually lzo).
  • gzip – gzip compression.
  • lzo – lzo compression.

Paste the below file into /bin/backup_all and make sure it’s executable.

vi /bin/backup-all
#!/bin/bash
#
# Filename : backup_all
# Description : Backup all OpenVZ containers in Proxmox
# Author : James Coyle
#
# Version:
# -Date      -Author     -Description
# 20-11-2013 James Coyle Initial
#
#

BACKUP_PATH=/var/lib/vz/dump
COMPRESS="lzo"

# Check dir exists
if [ ! -d $BACKUP_PATH ]; then
  echo "The directory $BACKUP_PATH does not exist."
  exit 99
fi

IFS=$'\n'
VMARRAY=($(vzlist -a -H))
VMIDS=""

for V in ${VMARRAY[@]}
do
  VMIDS="$VMIDS ${V:7:3}"
done

if [ -n $VMIDS ]; then
  vzdump $VMIDS --dumpdir $BACKUP_PATH --mode snapshot --compress $COMPRESS --remove 0
fi

echo "Backup of VMID(s) $VMIDS complete."

Restore a single Proxmox OpenVZ Container From The command Line

Get Social!

proxmox logo gradI mostly use Proxmox from the command line, or terminal, and I have created a few scripts to perform common and repetitive tasks.

The below script will restore a single OpenVZ container to the latest backup file available in the dump directory. The scripts takes a parameter for the container VMID to restore from backup. If the container exists, it will be stopped and removed before restoring the latest backup file available in the backup directory.

The script iterates through all of your backup files and only restores the latest based on the date in the file name, and not the date of the file creation or modified.

You will need to set the BACKUP_PATH variable to the location of your backup folder with no trailing slash, and BACKUP_EXT with the extension used for your chosen backup format.

If you save this script in the /bin then you can call the script from the terminal without having to move to the scripts directory. Create the file and paste the below script into it.

vi /bin/restore_one
#!/bin/bash
#
# Filename : restore_one
# Description : Restores a single OpenVZ Proxmox container to the latest backup file
#               available in the dump folder.
# Author : James Coyle
#
# Version:
# -Date -Author -Description
# 01-11-2013 James Coyle Initial
#
#

BACKUP_PATH=/var/lib/vz/dump
BACKUP_EXT=tar.lzo

# Do not change
SEARCH_PATH=$BACKUP_PATH/vzdump-openvz-$1-*.$BACKUP_EXT

function display-useage
{
  echo "Useage $0 [vmid to restore]"
  echo "Example: $0 999"

}

# Check dir exists
if [ ! -d $BACKUP_PATH ]; then
  echo "The directory $BACKUP_PATH does not exist"
  exit 99
fi

# Check if argument is present
if [ -z "$1" ]
then
  echo "Argument not present."
  display-useage
  exit 99
fi

# Check if vmid is available, on or off
VMON=$(vzlist | grep -P "[ ]+$1[ ]+")
VMOFF=$(vzlist --stopped | grep -P "[ ]+$1[ ]+")

if [ -n "$VMON" ]; then
  echo "Requesting stop of container."
  vzctl stop $1
  echo "Requesting deletion of container."
  vzctl delete $1
elif [ -n "$VMOFF" ]; then
  echo "Container is stopped."
  echo "Requesting deletion of container."
  vzctl delete $1
else
  echo "Container is not live."
fi

# Get unique VMIDs
for F in $SEARCH_PATH
do
  FILENAME=${F##*/}
  FILE_DATE=${FILENAME:18:19}
  FILE_DATE=${FILE_DATE//[_\-]/}

  if [ -z "$BACKUP_FILE" ]; then
    BACKUP_FILE=$F
  fi

  TEST_FILENAME=${BACKUP_FILE##*/}
  TEST_FILE_DATE=${TEST_FILENAME:18:19}
  TEST_FILE_DATE=${TEST_FILE_DATE//[_\-]/}
  if [ "$FILE_DATE" -gt "$TEST_FILE_DATE" ]; then
    BACKUP_FILE=$F
  fi
done

if [ -n $BACKUP_FILE ]; then
  # Restore VM
  echo "Restoring $1 with $BACKUP_FILE..."
  vzrestore $BACKUP_FILE $1
else
  echo "No backup file for VMID $1 exists."
fi

Make the script executable using chmod.

chmod +x /bin/restore_one

Use the below command, and substitute [VMID] with the container VMID to restore, to run the script.

restore_one [VMID]

See my other script on restoring multiple OpenVZ containers in Proxmox.


Restore all Proxmox OpenVZ Containers From The command Line

Get Social!

proxmox logo gradI use Proxmox to host a development environment using OpenVZ containers. I take frequent backups of all OpenVZ containers in the event I need to roll back any development work.

The below script restores all OpenVZ containers which are available in the backup folder, but not available in the Proxmox GUI. Using this script, you can remove the containers in Proxmox which you would like to restore and run the script to restore the latest backup.

The script iterates through all of your backup files and only restores the latest based on the date in the file name.

You will need to set the BACKUP_PATH variable to the location of your backup folder with no trailing slash, and BACKUP_EXT with the extension used for your chosen backup format.

If you save this script in the /bin then you can call the script from the terminal without having to move to the scripts directory. Create the file and paste the below script into it.

vi /bin/restore_all
#!/bin/bash
#
# Filename : restore_all
# Description : Restores all missing OpenVZ containers in Proxmox to the latest version available in the dump folder.
# Author : James Coyle
#
# Version:
# -Date       -Author      -Description
# 01-11-2013  James Coyle  Initial
#
#

BACKUP_PATH=/var/lib/vz/dump
BACKUP_EXT=tar.lzo

# Do not change
SEARCH_PATH=$BACKUP_PATH/vzdump-openvz-*.$BACKUP_EXT

# Check dir exists
if [ ! -d $BACKUP_PATH ]; then
  echo "The directory $BACKUP_PATH does not exist"
  exit 99
fi

# Get unique VMIDs
for F in $SEARCH_PATH
do
  FILENAME=${F##*/}
  FILE_DATE=${FILENAME:18:19}
  FILE_DATE=${FILE_DATE//[_\-]/}
  VMID=${FILENAME:14:3}
  if [ -n $FILE_VIMS[$VMID] ]; then
    FILE_VIMS[$VMID]=$F
  fi

  TEST_FILENAME=${FILE_VIMS[$VMID]##*/}
  TEST_FILE_DATE=${TEST_FILENAME:18:19}
  TEST_FILE_DATE=${TEST_FILE_DATE//[_\-]/}
  if [ "$FILE_DATE" -gt "$TEST_FILE_DATE" ]; then
    FILE_VIMS[$VMID]=F
  fi
done

# Restore VM
for I in ${!FILE_VIMS[*]}
do
  echo "Restoring $I with ${FILE_VIMS[$I]}..."
  vzrestore ${FILE_VIMS[$I]} $I
done

Make the script executable using chmod.

chmod +x /bin/restore_all

Use the below command to run the script and restore all containers which are missing from backup.

restore_all

Mount NFS volume in a Proxmox OpenVZ container

Category : How-to

Get Social!

openvz-logo-150px_new_3There are various options for adding additional storage to an OpenVZ container. You can add additional storage to the containers root volume to simply increase the overall storage available to the container. For external storage, or storage on another disk to the root partition of the container there are bind mounts.

With some light work you can also use NFS mounts inside an OpenVZ container. Before NFS will work in a container a command needs to be ran on the host to enable NFS features in the container.

If you do not enable NFS on the container, you will get the following error:

mount: unknown filesystem type 'nfs'

Open a Terminal on the host machine and run the below command to check that the modules are loaded in the kernel:

modprobe nfs

Then run the below command to enable NFS on the container. Make sure container is turned off or restart the container after issuing the command.

vzctl set 998 --features "nfs:on" --save

This writes a change to the containers config file. To avoid using the command, you could simply edit the config file directly and add the below text to the bottom of the file:

FEATURES="nfs:on"

Start the container and make sure that the required packages are installed.

apt-get install nfs-common

If you do not have the required packages installed you may receive the following error

mount: wrong fs type, bad option, bad superblock on 192.168.50.252:/dspool/compressed,
 missing codepage or helper program, or other error
 (for several filesystems (e.g. nfs, cifs) you might
 need a /sbin/mount.<type> helper program)
 In some cases useful info is found in syslog - try
 dmesg | tail or so

Finally, run the mount command to mount your NFS directory.

mount -t nfs 10.10.10.5:/storage/compressed /mnt/testmount

NFS Mount Error on OEL

Category : How-to

Get Social!

oelI have been using an Oracle Enterprise Linux (OEL) OpenVZ container in Proxmox for installing some Oracle software. The easiest and least interfering way I have found is to mount an NFS share which contains the binaries directly to the container. See my NFS blog post for more information on setting up an OpenVZ container for NFS.

After the service and container was set up for NFS I tried to mount the NFS share. I received the below error after running the mount command.

[root@localhost ~]# mount -a
mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: an incorrect mount option was specified

This was because the rpcbind daemon was not running. Run the below command to start the rpcbind daemon.

service rpcbind start

The service will display that the daemon is now running.

Starting rpcbind: [ OK ]

You can set rpcbind to start automatically by adding it to the startup group.

chkconfig rpcbind on

Retry the mount command and your NFS share should be mounted.

mount -a

You may need to check that you have the NFS client packages installed.

yum install nfs-utils

Visit our advertisers

Quick Poll

What type of VPN protocol do you use?

Visit our advertisers