Category Archives: How-to

How to Cluster Proxmox

Tags :

Category : How-to

Get Social!

proxmox logo grad

Clustering is a technique to group together multiple machines or devices providing common parameters, such as storage, over all machines in the cluster. Proxmox can be clustered to provide a common interface to control all machines in the cluster.

Automatic failover of machines can also be achieved with a Proxmox cluster however this requires significant setup and it not available out of the box. I will discuss VM failover in a future blog post.

Create a new Cluster

To join one or more nodes to a Proxmox cluster we must first create a cluster on one of the machines. Run the below command on your first Proxmox node and replace [CLUSTER_NAME] with a name for your cluster.

pvecm create [CLUSTER_NAME]

Check the status of a Cluster

Once a cluster is created, you can check the status using the status command.

pvecm status

You can use the nodes command to view all nodes in the cluster.

pvecm nodes

Add a new Node to the Cluster

Adding a new node to a Proxmox cluster is easy and only requires the hostname or IP address of the primary cluster node. It is highly recommended to use a hostname or DNS entry over an IP address to shield any IP address changes.

On a non-clustered node, log in and run the pvecm add command and specify the IP address or hostname. You can use the IP address or hostname of any already clustered Proxmox node.

pvecm add [HOSTNAME_OF_CLUSTER]

You must not have any virtual machines/ OpenVZ containers running on the node which you are adding to the cluster. All nodes should also be running the same component versions of Proxmox to ensure compatibility.

OpenVZ disk space issues

Tags :

Category : How-to

Get Social!

proxmox logo gradI started to receive an error in one of my Proxmox OpenVZ containers which indicated that the quota had been exceeded for disk space. Immediately I logged into the container and performed a df -h to view the disk usage. To my surprise only 50% of the disk space had been used.

Error in the logs:

kernel: VZ QUOTA: file softlimit exceeded for id=103

OpenVZ containers have two storage related limits;

  • Disk space – this is the amount of space which can be used by the container, usually measured in gigabytes.
  • inodes – this loosely translates to the number of files which can be stored on the filesystem.

Below is an example of the disk based limits configuration.

DISKSPACE="10485760:11534336"
DISKINODES="2000000:2200000"

The problem with my container was that although I wasn’t using all the allocated disk space, I was using all of the allocated inodes as I had a java application running which created 1000s of new small files every hour. CCTV recording software such as Zoneminder would likely have the same problem as it creates many small image files when recoding movement from a camera.

You can test how many inodes are in use by running df -i on the container.  An example below of a very low inode usage:

Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/simfs 10000000 116024 9883976 2% /

Increase disk space and inode quota

Proxmox does not allow you to change the inodes limits directly from the GUI. You will need to edit the configuration file for the container directly on the OS or use the vz command. The config files are stored in /etc/vz/conf by default and named VMID.conf. Edit the required config file for your container and increase the DISKSPACE and DISKINODES values as required.

vi /etc/vz/conf/103.conf
DISKSPACE="10485760:11534336"
DISKINODES="4000000:4400000"

OpenVZ has the concept of HARD and SOFT limits. A soft limit will trigger a warning but allow the applications to continue to use disk space/ inodes however when the hard limit is hit new requests for storage will be denied. A hard limit is like a physical disk being full.

DISKSPACE="SOFT:HARD"
DISKINODES="SOFT:HARD"

You can also use the word UNLIMITED to remove any virtual limits on storage however you will still be constrained by the underlying physical storage limits.

You will need to restart your container for the settings to take effect.

To avoid having to restart your container, you can use the vzctl command to apply the new limits immediately, even to a running container.

vzctl set 103 --diskspace 45G:49G --save
vzctl set 103 --diskinodes 4000000:4400000 --save

Gitlab error “fatal: The remote end hung up unexpectedly”

Tags :

Category : How-to

Get Social!

gitlabI have been using Gitlab for a while now to run a local Git server for storing code and documentation of the projects I work on. Gitlab is the open source version of Github which you can run on your own environments .

After upgrading to version 6.3 I received an error when checking in larger projects to the Gitlab server.

The below shows the git push command and the error which occurred.

git push origin master
Counting objects: 47, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (43/43), done.
Writing objects: 100% (47/47), 2.91 MiB | 204.00 KiB/s, done.
Total 47 (delta 0), reused 0 (delta 0)
error: RPC failed; result=22, HTTP code = 411
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly
Everything up-to-date

The solution is the run the below command on the client to increase the postBuffer size before trying to re-run the git push. Use the below command to set the postBuffer size to 100MB.

git config http.postBuffer 104857600

Then retry the git push and all should be working.

git push origin master
Counting objects: 47, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (43/43), done.
Writing objects: 100% (47/47), 2.91 MiB | 0 bytes/s, done.
Total 47 (delta 0), reused 0 (delta 0)

Backup all Proxmox OpenVZ containers in one go

Tags :

Category : How-to

Get Social!

proxmox logo gradThe below script is a bash script which works with Proxmox and the OpenVZ commands to backup all known containers to a specified folder.

You will need to set BACKUP_PATH to be the folder where you would like the backups to be stored and COMPRESS to a value which specifies the compression used. COMPRESS values can be:

  • 0 – no compression.
  • 1 – default compression (usually lzo).
  • gzip – gzip compression.
  • lzo – lzo compression.

Paste the below file into /bin/backup_all and make sure it’s executable.

vi /bin/backup-all
#!/bin/bash
#
# Filename : backup_all
# Description : Backup all OpenVZ containers in Proxmox
# Author : James Coyle
#
# Version:
# -Date      -Author     -Description
# 20-11-2013 James Coyle Initial
#
#

BACKUP_PATH=/var/lib/vz/dump
COMPRESS="lzo"

# Check dir exists
if [ ! -d $BACKUP_PATH ]; then
  echo "The directory $BACKUP_PATH does not exist."
  exit 99
fi

IFS=$'\n'
VMARRAY=($(vzlist -a -H))
VMIDS=""

for V in ${VMARRAY[@]}
do
  VMIDS="$VMIDS ${V:7:3}"
done

if [ -n $VMIDS ]; then
  vzdump $VMIDS --dumpdir $BACKUP_PATH --mode snapshot --compress $COMPRESS --remove 0
fi

echo "Backup of VMID(s) $VMIDS complete."

Increase mdadm raid rebuild speed

Tags :

Category : How-to

Get Social!

Linux penguinmdadm can take quite a while when rebuilding arrays and the more disks there are, the longer it will take. As an example, it took 5 x 2TB WD green disks 3 days to complete.

There are two main ways to increase the rebuild speed of a mdadm array which are detailed below. Remember that it doesn’t matter what options you use here – you can still be limited by your CPU, memory and hard disk channel bandwidth.

Remove the mdadm rebuild speed restriction

mdadm has a parameter which restricts the speed of the rebuild process on each array. The idea behind this is to stop mdadm from making the rest of your system unusable by guaranteeing that it doesn’t consume all the available resource and IO bandwidth. The attributes can be increased to raise the speed limit on mdadm and reduce the time it takes to complete the rebuild process.

You will be editing your sysctl.conf file which holds many attributes for the Linux system. As always, and before editing any configuration file, you should take a backup.

cp /etc/sysctl.conf /etc/sysctl.conf_ORIG

Open the sysctl.conf file with a text editor.

vi /etc/sysctl.conf

Find or add the below parameter and set the value to the speed in KB/s which you would like to use. The below example sets the speed limit to 50 MB/s.

dev.raid.speed_limit_max = 51200

You will then need to load the settings using the sysctl command.

/sbin/sysctl -p

Add bitmap indexes to mdadm

Adding a bitmap index to a mdadm before rebuilding the array can dramatically speed up the rebuild process.

Use the below command to add a bitmap index to an array. The example assumes your array is found at /dev/md0.

mdadm --grow --bitmap=internal /dev/md0

Once the process has completed, use the below command to remove the mdadm bitmap index. Again, this example assumes your array is found at /dev/md0.

mdadm --grow --bitmap=none /dev/md0

 

Reverse Proxy Subsonic with Apache

Tags :

Category : How-to

Get Social!

SubsonicLogoSubsonic is a web-based media player for playing audio and video files through a web browser. You can reverse proxy Subsonic using Apache

See my blog post on using Apache as a reverse proxy for more detailed information on Apache configuration files.

The below configuration expects the backend Subsonic port to be non-ssl as the encryption will be offloaded to the Apache reverse proxy server. The reverse proxy URL will be encrypted and available on the default SSL port 443. This has the advantage of not using any CPU on the Subsonic server for encrypting traffic allowing it to concentrate on transcoding.

Add the below text to a new sites-available Apache configuration file.
vi /etc/apache2/sites-available/subsonic

<VirtualHost *:443>
    SSLEngine On
    SSLCertificateFile /ssl-certs/cert.cer
    SSLCertificateKeyFile /ssl-certs/key.key
    SSLProxyEngine on
    ServerAdmin [email protected]
    DocumentRoot /var/www
    ServerName subsonic.jamescoyle.net
    # Possible values include: debug, info, notice, warn, error, crit ,alert, emerg.
    LogLevel warn
    CustomLog ${APACHE_LOG_DIR}/subsonic-access.log combined
    ErrorLog ${APACHE_LOG_DIR}/subsonic-error.logProxyHTMLStripComments on
    <Location />
        ProxyRequests off
        RequestHeader unset Accept-Encoding
        ProxyPass http://192.168.1.22:4040/
        ProxyPassReverse http://192.168.1.22:4040/
        Order allow,deny
        Allow from all
    </Location>
<VirtualHost>

 

Enable the new site in Apache and reload the configuration.
a2ensite subsonic
service apache2 reload

« Previous Page — Next Page »

Visit our advertisers

Search

Quick Poll

Do you use GlusterFS in your workplace?
Results

Recent Posts

Visit our advertisers

Site Links
Blog
Rated Sites

JamesCoyle.net Limited