Proxmox OpenVZ SWAP and Performance

Proxmox OpenVZ SWAP and Performance

Get Social!

openvz-logo-150px_new_3 I have been having trouble with a Proxmox node which is only running OpenVZ containers however it is at the upper limit of its RAM resources. Over time, I noticed that Proxmox used SWAP (virtual memory, page file, etc), quite aggressively, to make sure there was always some RAM free. That sounds fantastic, and is just what I would expect the Proxmox server to be doing, except it does it all too well. Proxmox made sure that around 40% of the RAM on the host machine was free at the expense of moving many running processes across all the running containers to SWAP. This is how Linux works, by design, and is expected behaviour. Running processes which have memory which hasn’t been touched in a while have their memory moved to SWAP. This allows other applications which need the memory right now to use it and anything left over can be used as cache by the kernel. When a process with memory in SWAP needs to use that memory, it needs to be read from SWAP and back into memory so that it can be used. There is a huge overhead with this process and will often be noticed when you use a container which has not been used in a while – at the start everything will be slow until all the required memory items have been read from SWAP and put back into RAM. To help with this situation we can do two things:

  • Make sure SWAP is always on a fast disk with plenty of free IO bandwidth. On a small installation, this should not be on the same disk as your container file systems. SSDs can also bring a huge performance benefit over conventional mechanical drives.
  • Reduce the amount of RAM which Proxmox keeps free by making the algorithm which moves memory to SWAP less aggressive.

Move SWAP to fast storage

Generally, and when installing Proxmox for the first time a SWAP partition will be created on your hard disk. By default, this will be the same partition as your Proxmox operating system and your container storage. On a slow mechanical disk, this will result in far too much IO concurrency – that is different processes trying to read or write to a disk at the same time – which will massively affect server performance. One thing we can move to another disk is system wide swap.

You can either use a new file, disk, partition or block device for your new swap location. You will then need to turn your old SWAP device off to stop it from being used. Use the below examples to move your SWAP device.

See this post for a quick script to automatically create a SWAP file.

Make a new SWAP device as a file

Create a file on your file system and enable it to be used as a SWAP device. The below example uses the mount /mnt/swapdrive and the file swapfile to use as your new swap device with a size of 4096 MB.

dd if=/dev/zero of=/mnt/swapdrive/swapfile bs=1M count=4096

You will then need to format the file as SWAP with the below command.

mkswap /mnt/swapdrive/swapfile

Make a new SWAP device as a partition

Use the below command to use a drive partition as your new SWAP device. The below example uses /dev/sdc3 as your SWAP partition. You must have precreated this partition for it to be available.

mkswap /dev/sdc3
swapon /dev/sdc3

Turn a new SWAP device on

Once you have a new SWAP device created, either a file or a disk or partition you will need to enable it. Use the swapon command. The below shows an example of a file and disk partition command:

swapon /mnt/swapdrive/swapfile
swapon /dev/sdc3

Turn off the old SWAP device

To turn off the old SWAP device, first identify it using swapon -s.

swapon -s

Then, use the swapoff command to turn the device off. The below example is the default Proxmox SWAP device location.

swapoff /dev/mapper/pve-swap

Clear SWAP space without rebooting

You can clear your SWAP memory by turning the system wide SWAP memory off and then back on again. Run the below commands to turn off your system wide SWAP space forcing all the SWAP to be read back into RAM. You must have enough RAM for available on your system for this to work correctly. Once this has completed, run the second command to turn SWAP back on again. You can also use this to make your SWAP memory changes take effect.

swapoff -a 
swapon -a

Make the SWAP file persist after rebooting

To make sure your SWAP file is mounted the next time your machine reboots you’ll need to add an entry to the fstab file.

Open the fstab file with your text editor:

vi /etc/fstab

And add a line, similar to the below making sure the first attribute is the location of your newly created SWAP file.

/mnt/swapdrive/swapfile  swap  swap  defaults  0  0

Change the ‘swapiness’ setting

To change how aggressively Proxmox, or other Linux distribution, moves process memory to SWAP we have a swapiness attribute. The swapiness setting is a kernel setting which is permanently set in the /etc/sysctl.conf file, or temporarily using sysctl.

The swapiness setting takes a value between 0 and 100. Using 0 will virtually turn off using SWAP, except to avoid an out of memory exception (oom). Using a value of 100 will cause the system to use SWAP as often as possible and will likely degrade system performance servilely. A value of 60 is the default for Proxmox.

Change the swapiness value for the current boot

To change your swapiness value for the current boot, use the below command. The value will be reset after rebooting. The following example will set the swapiness value to 20.

sysctl -w vm.swappiness=20

Permanently change the swapiness value

Use the below command to permanently change your swapiness value. Note that this will not affect the current boot.

vi  /etc/sysctl.conf

And add the following to give a swapiness of 20

vm.swappiness=20

Iperf cheat sheet

Get Social!

iperfIperf is an Open source network bandwidth testing application, available on Linux, Windows and Unix. Iperf can be used in two modes, client and server. The server runs on the remote host and listens for connections from the client. The client is where you issue the bandwidth test parameters, and connect to a remote server.

Install Iperf on Ubuntu

You can use apt-get install to install Iperf in Ubuntu.

apt-get install iperf

Start server

To start Iperf in server mode, use the below command.

iperf -s

Start server in daemon mode

Running the server without daemon mode keeps the process running in the terminal. Use the -D switch to run it as a daemon in the background.

iperf -s -D

Connecting to server from client

Iperf needs to run on the local host in client mode, as well as in server mode on the remote host. To connect to the remote host, add it’s IP address after the -c switch.

iperf -c 10.1.1.50

Bi-directional simultaneous (test the speed both ways at the same time)

Use the -d switch to test in the network bandwidth in both directions. This will perform two tests; one from local host to remote host, and another from the remote host to the local host.

iperf -c 10.1.1.50 -d

 

Bi-directional  (test the speed both one after another)

Use the -r switch to test in the network bandwidth in both directions. This is similar to -d except the tests will be performed in sequence; first from local host to remote host, and another from the remote host to the local host.

iperf -c 10.1.1.50 -r

Change the window size

The TCP window size can be changed using the -w switch followed by the number of bytes to use. the below example shows a window size of 2KB. This can be used on either the server or the client.

iperf -c 10.1.1.50 -w 2048
 iperf -s -w 2048

Change the port

You must use the same port on both the client and the server for the two processes to communicate with each other. Use the -p switch followed by the port number to use on both the local and remote host.

iperf -c 10.1.1.50 -p 9000
iperf -s -p 9000

Change the test duration

The default test duration of Iperf is 10 seconds. You can override the default with the -t switch followed by the time in seconds the test should last.

iperf -s -t 60

UDP instead of TCP

The default protocol for Iperf to use is TCP. You can change this to UDP with the -u switch. You will need to run both the client and server in UDP mode to perform the tests.

iperf -s -u
iperf -c -u

The result will have an extra metric for the packet loss which should be as low as possible, otherwise the packets will have to be re-transmitted using more bandwidth.

Run multiple threads

Iperf can spawn multiple threads to simultaneously send and receive data. Use the -P switch followed by the number of threads to use.

iperf -c -P 4

Check the version of Iperf

Use the -v switch to see the version of Iperf you have installed.

iperf -v

See the full list of arguments

Use the -h switch to see the full list of arguments supported by Iperf.

iperf -h

 

Let me know in the comments if you think anything is missing.


iptables cheat sheet

Get Social!

Here are a few handy commands for using iptables. They are tailored for an OpenVZ container with a venet network interface but can easily be adapted to use your interface by replacing venet0 with your network interface.

For setting up iptables in an openVZ this blog post.

iptables console

Remove existing rules

You can easily delete all existing rules in iptables. Be careful using this command, there is no going back unless you have backed up your rules.

iptables -F

Backup and restore

Backup to file

Rules can easily be saved to an external file for backups or outputting for version control. This will save the rules to /etc/iptables.rules.

iptables-save -c > /etc/iptables.rules

Restore from file

Saved settings can be restored with the following command:

iptables-restore > /etc/iptables.rules

Change the default policy

The default policy can be changed to specify what should happen to traffic which doesn’t have a rule to explicitly define what to do. You can specify to ACCEPT, REJECT or DROP for INPUT, FORWARD and OUTPUT.

Change OUTPUT to ACCEPT

iptables -P OUTPUT ACCEPT

Change INPUT to DROP

iptables -P INPUT DROP

Apply the catchall rule

You can add a rule to the bottom of the rule book to choose what to do with traffic which doesn’t match any other rule. A common use would be to add a DROP as the last rule to drop any traffic which isn’t explicitly allowed by an earlier rule. You can specify to ACCEPT, REJECT or DROP for INPUT, FORWARD and OUTPUT as well as an interface.

ACCEPT INPUT on interface lo

iptables -A INPUT -i lo -j ACCEPT

BLOCK INPUT on interface venet0

iptables -A INPUT -i venet0 -j DROP

List active rules

You can list the active rules with -L, and -v for information on packets affected.

iptables -L -v

Enable established connections rule

Already established connections will not be affected by adding this inbound rule. Traffic affected by other outbound rules will also be honoured. If you add this rule, you won’t likely need to specify inbound rules for many outbound rules.

iptables -A INPUT -i venet0 -m state --state RELATED,ESTABLISHED -j ACCEPT

Adding new rules

New rules can be added to control how traffic passes through an interface. If you have not used the Established connections rule then you will also need to add an outgoing rule. If you do use the Established connections rule then you will only need the top command in each of the below headings.

Add rule for port 80 – such as Apache

iptables -A INPUT -i venet0 -p tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -o venet0 -p tcp --sport 80 -m state --state ESTABLISHED -j ACCEPT

Add rule for port 22 – SSH outbound connections

iptables -A OUTPUT -o venet0 -p tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -i venet0 -p tcp --sport 22 -m state --state ESTABLISHED -j ACCEPT

Add rule for 53 – DNS outbound

iptables -A OUTPUT -o venet0 -p udp -o eth0 --dport 53 -j ACCEPT
iptables -A INPUT -i venet0-p udp -i eth0 --sport 53 -j ACCEPT

Add rule for port 22 – SSH inbound connections

iptables -A INPUT -i venet0 -p tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -o venet0 -p tcp --sport 22 -m state --state ESTABLISHED -j ACCEPT

Add rule for outgoing http/ https

iptables -A OUTPUT -o venet0 -p tcp -m multiport --dport 80,443 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A INPUT -i venet0 -p tcp -m multiport --sports 80,443 -m state --state ESTABLISHED -j ACCEPT

Add rule for ping from remote to local

iptables -A INPUT -i venet0 -p icmp --icmp-type echo-request -j ACCEPT
iptables -A OUTPUT -o ventet -p icmp --icmp-type echo-reply -j ACCEPT

 Add rule for ping from local to remote

iptables -A OUTPUT -o venet0 -p icmp --icmp-type echo-request -j ACCEPT
iptables -A INPUT -i venet0 -p icmp --icmp-type echo-reply -j ACCEPT

Delete rule

The easiest way to remove a rule is to delete it by it’s rule position in the list. To find out the rules position run iptables -L with the line-number argument. The below command is to delete an INPUT command however you can easily change INPUT to OUTPUT as required.

iptables -L INPUT --line-numbers

Then run the delete command for the relevent direction. This will delete the 7th inbound rule.

iptables -D INPUT 7

Example deleting rule 1 for INPUT:

# iptables -L INPUT --line-numbers
Chain INPUT (policy DROP)
num  target     prot opt source               destination
1    ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh state NEW,ESTABLISHED
2    ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
3    ACCEPT     icmp --  anywhere             anywhere             icmp echo-reply
4    ACCEPT     icmp --  anywhere             anywhere             icmp echo-request

# iptables -D INPUT 1

# iptables -L INPUT --line-numbers
Chain INPUT (policy DROP)
num  target     prot opt source               destination
1    ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
2    ACCEPT     icmp --  anywhere             anywhere             icmp echo-reply
3    ACCEPT     icmp --  anywhere             anywhere             icmp echo-request

 

Let me know in the comments if you think anything is missing.


Visit our advertisers

Search

Quick Poll

Which type of virtualisation do you use?
  • Add your answer

Visit our advertisers