Proxmox 3.2 is now available with SPICE, Ceph and updated QEMU

Proxmox 3.2 is now available with SPICE, Ceph and updated QEMU

Category : Tech News

Get Social!

proxmox logo gradProxmox has today released a new version of Proxmox VE, Proxmox 3.2 which is available as either a downloadable ISO or from the Proxmox repository.

Hilights of this release include’;

  • Ceph has now been integrated to the Proxmox web GUI as well as a new CLI command created for creating Ceph clusters. See my post on Ceph storage in Proxmox for more information.
  • SPICE is now fully integrated as the console viewer however the original Java console is still the default. SPICE supports multiple monitors and all recent guest operating systems.
  • QEMU has been updated with better backups and a few new supported guest hardware devices, mostly for compatibility with VMWare.

You can download the ISO from Proxmox directly at the following link:
http://www.proxmox.com/downloads/item/proxmox-ve-3-2-iso-installer

If you already have Proxmox installed, you can use the below commands to automatically update your Proxmox servers to the latest 3.2 version from the terminal. Before updating, make sure all your VM’s have been stopped. Run the below commands on each server in your cluster.

Upgrade
apt-get update
apt-get dist-upgrade

Restart all Proxmox servers to complete the installation.


Add Create OpenVZ Template to the Proxmox Web GUI

Get Social!

proxmox logo gradCreating a template from an OpenVZ container is a very manual process. My biggest problem is that you have to have root access to the Proxmox hardware node in order to create a tar from the root of the CT. See How to make a new OpenVZ template for more information on manually creating a template.

proxmox-create-template-context-menu

I created a small code patch for the Proxmox API and web GUI to add a ‘create template’ feature for CTs. The code adds a context menu entry when you right click on a CT in the Proxmox web GUI.

Before using the feature, the CT must be shut down and any network interfaces removed. The feature presents the user with a dialogue box requesting which storage device the template should be saved to, and what it should be called.

Once the storage has been selected and the template has been given a name, a new ‘create template’ task is created which archives the root directory of the selected container and adds it to the cache folder of the selected storage.

proxmox-create-template-dialogue-box

The changes were declined by the Proxmox team on the grounds that creating a template is a technical process and may not result in creating a working, cloned instance. In addition, it is very easy to leave sensitive information in the CT which is the source of the template – all data on the CTs file system will be archived into the template making it available the next time a CT is created. If SSH keys are left on the CT, for example, then they will be available in the new CT also.

Because the feature was not accepted into the main distribution of Proxmox, I will maintain it myself and manually apply the patches to my Proxmox servers after every update. I have created a public repository on my Gitlab server that holds the git patch file which is available for everyone. 

If you accepts the risks mentioned above, and are happy to hack away at your Proxmox binaries, then you are welcome to try the patch for yourself.

You can download the patch and get more information on my public Proxmox Github page.

I should mention that this patch may not always be up to date. In addition, this is changing the actual Proxmox distribution files and as such may have unintended side effects. Please use these patches with caution and only in your development environments.

gitlabDownload

 


Ceph Storage on Proxmox

Get Social!

ceph-logoCeph is an open source storage platform which is designed for modern storage needs. Ceph is scalable to the exabyte level and designed to have no single points of failure making it ideal for applications which require highly available flexible storage.

Since Proxmox 3.2, Ceph is now supported as both a client and server, the client is for back end storage for VMs and the server for configuring storage devices. This means that a Ceph storage cluster can now be administered through the Proxmox web GUI and therefore can be centrally managed from a single location. In addition, as Proxmox now manages the Ceph server the config can be stored in Proxmox’ shared file system meaning that the configuration is immediately replicated throughout the entire cluster.

The below diagram shows the layout of an example Proxmox cluster with Ceph storage.

  • 2 nodes are used dedicated to running VMs and use the Ceph storage hosted by the other nodes.
  • Two networks are used, one for management and application traffic and one for Ceph traffic only. This helps to maintain sufficient bandwidth for storage requirements without affecting the applications which are hosted by the VMs.

ceph-infrastructure-proxmox

Before getting started with setting up the Ceph cluster, we need to do some preparation work. Make sure the following prerequisites are met before continuing the tutorial.

  • You have Proxmox cluster with the latest packages from the pvetest repository. You must have at least three nodes in your cluster. See How to set up a cluster.
  • SSH Keys are set up between all nodes in your cluster – Proxmox does this automatically as part of clustering but if you are using a different user, you may need to set them up manually.
  • NTP is set up on all nodes in your cluster to keep the time in sync. You can install it with: apt-get install ntp

promox-ceph-3-nodesThe rest of this tutorial will assume that you have three nodes which are all clustered into a single Proxmox cluster. I will refer to three host names which are all resolvable via my LAN DNS server; prox1, prox2 and prox3 which are all on the jamescoyle.net domain. The image to the left is what is displayed in the Proxmox web GUI and details all three nodes in a single Proxmox cluster. Each of these nodes has two disks configured; one which Proxmox is installed onto and provides a small ‘local’ storage device which is displayed in the image to the left and one which is going to be used for the Ceph storage. The below output shows the storage available, which is exactly the same on each host. /dev/vda is the root partition containing the Proxmox install and /dev/vdb is an untouched partition which will be used for Ceph.

root@prox1:~# fdisk -l | grep /dev/vd
Disk /dev/vdb doesn't contain a valid partition table
Disk /dev/mapper/pve-root doesn't contain a valid partition table
Disk /dev/mapper/pve-swap doesn't contain a valid partition table
Disk /dev/mapper/pve-data doesn't contain a valid partition table
Disk /dev/vda: 21.5 GB, 21474836480 bytes
/dev/vda1   *        2048     1048575      523264   83  Linux
/dev/vda2         1048576    41943039    20447232   8e  Linux LVM
Disk /dev/vdb: 107.4 GB, 107374182400 bytes

Now that I’ve set the scene, lets start to put together our Ceph cluster! Before using the new Proxmox web GUI you must run a few SSH commands to set up the initial Ceph instance.

Run the below command on all of the nodes which you will use as a Ceph server. This will download and set up the latest Ceph packages.

pveceph install

Create the Ceph config file by initialising pveceph. The Ceph config file will be created in your /etc/pve/ directory called ceph.conf. You should only run this on one node.

pveceph init --network 192.168.50.0/24

The next step is to set up the Ceph monitors for your cluster. So that you don’t have a single point of failure, you will need at least 3 monitors. You must also have an uneven number of monitors – 3, 5, 7, etc.

pveceph createmon

The rest of the configuration can be completed with the Proxmox web GUI. Log in to your Proxmox web GUI and click on one of your Proxmox nodes on the left hand side, then click the Ceph tab.

proxmox-ceph-status-tab

Next, you will add a disk to the Ceph cluster. Each disk creates to as an OSD in Ceph which is a storage object used later by the Ceph storage pool. Click on the Disks tab at the bottom of the screen and choose the disk you would like to add to the Ceph cluster. Click the Create: OSD button and click Create to create an OSD. Repeat these steps for each Proxmox server which will be used to host storage for the Ceph cluster.

ceph-create-osd

If the Create: OSD button is greyed out, it’s because the disk is not in a state where Ceph can use it. It’s likely because you have partitions on your disk. Run the fdisk command on the disk and press d to delete the partitions and w to save the changes. For example:

fdisk /dev/vdb

The last step in creating the Ceph storage cluster is to create a storage pool. Click the Pools tab and click Create. Enter the below values into the new Create Pool dialogue box:

  • Name: name to use for the storage pool.
  • Size: the number of replicas to use for a working cluster. A replica is the number of times the data is stored across nodes.
  • Min. Size: the minimum of replicas which can be used.
  • Crush RuleSet:
  • pg_num: this is the placement group count which you have to calculate based on the number os OSDs you have. To calculate your placement group count, multiply the amount of OSDs you have by 100 and divide it by the number of number of times each part of data is stored. The default is to store each part of data twice which means that if a disk fails, you won’t loose the data because it’s stored twice.For our example,3 OSDs * 100 = 300
    Divided by replicas, 300 / 2 = 150

ceph-create-pool

The Ceph storage pool is now set up and available to use for your KVM images. You can check the status of the Ceph storage pool by clicking on the Status tab.

ceph-status-screen

See my blog post on mounting Ceph storage on Proxmox.


Manually Mount a Ceph Storage Pool in Proxmox

Get Social!

ceph-logoThe latest BETA of Proxmox, and soon to be released 3.2 comes with the Ceph client automatically installed which makes mount Ceph storage pools painless.

You can mount the Ceph storage pool using the Proxmox web GUI.

You may need to copy the Ceph storage pool keyring from your Ceph server to your Proxmox server. If you use authentication on your Ceph storage pool, log in to your Ceph server via SSH. Run the below scp command to copy your Ceph keyring to your Proxmox server. You will need to replace [PROXMOX_SERVER] with the host name or IP address of your Proxmox server and [STORAGE_NAME] is the name you will use for your Proxmox storage mount which will be used in the web GUI.

scp /etc/ceph/ceph.client.admin.keyring [PROXMOX_SERVER]:/etc/pve/priv/ceph/[STORAGE_NAME].keyring

For example

scp /etc/ceph/ceph.client.admin.keyring prox3.jamescoyle.net:/etc/pve/priv/ceph/ceph_storage.keyring

To use the Proxmox GUI to mount a Ceph storage pool, login to the web GUI and click on the Datacenter folder in the top left, then click the Storage tab, Add and finally RBD.

proxmox-add-ceph-rbd

Enter the details of your Ceph storage pool in the new dialogue box which appears.

  • ID: the name which this storage mount will have in Proxmox. If you use authentication on your Ceph server, you will need to use the same name as the above  [STORAGE_NAME] field.
  • Pool: the Ceph storage pool name.
  • Monitor Host: the host or IP addresses of your Ceph monitor servers. You can enter a single host or multiple hosts separated by a semicolon [;].
  • User name: the named use to use when connecting to the Ceph storage pool. If you are not using user authentication, set this to the below value.
    admin (optional, default = admin)

proxmox-ceph-add-rbd

Don’t forget to make sure that the Proxmox host can reach the remote Ceph monitor hosts – by default a Ceph monitor runs on port 6789.

proxmox-ceph-storage-mount-rbd


Ceph Error “Warning: the ECDSA host key for ‘SERVER’ differs from the key for the IP address ‘IP'”

Category : How-to

Get Social!

ceph-logoI recently received the below error when installing Ceph on Proxmox which related the the key for the local machine. After running the ceph-deploy install command I was presented with an error and the Ceph command halted.

ceph-deploy install ceph1.jamescoyle.net ceph2.jamescoyle.net ceph3.jamescoyle.net

After agreeing to the initial prompt I received the below error.

Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ceph1.jamescoyle.net' (ECDSA) to the list of known hosts.
Warning: the ECDSA host key for 'ceph1.jamescoyle.net' differs from the key for the IP address '192.168.50.102'
Offending key for IP in /etc/ssh/ssh_known_hosts:2
Are you sure you want to continue connecting (yes/no)? yes
Write failed: Broken pipe
[ceph_deploy][ERROR ] RuntimeError: connecting to host: ceph1.jamescoyle.net resulted in errors: HostNotFound ceph1.jamescoyle.net

You need to remove the local key using ssh-keygen and then allow the machine to re-add the key when you first connect to it. The above error was complaining about the key for IP 192.168.50.102, therefore add this IP to the end of the below command.

ssh-keygen -R [IP]

Example:

ssh-keygen -R 192.168.50.102
/root/.ssh/known_hosts updated.
Original contents retained as /root/.ssh/known_hosts.old

Now rerun your Ceph command and accept the key warning and your Ceph command should complete.

 


Set up Fail2ban for Proxmox Web GUI

Get Social!

fail2ban_logoFail2ban is an application that scans log files in real time and bans malicious IP addresses based on a set of rules and filters you can set.

For this blog post, we’re going to look at capturing invalid login attempts to the Proxmox Web GUI and ban any IP addresses from accessing the Web GUI if they fail to authenticate 3 times from the same IP address.

Fail2ban is made up of three main component parts:

  • Filter – a Filter is a pattern or regular expression that we wish to search for in the log files. In our case, we want to search for the words ‘authentication failure’ in the log because that’s what the pvedaemon writes when a failed login attempt occurs.
  • Action – an Action is what we’ll do if the filter is found. What we need to do is ban any IP address where the filter is triggered 3 times.
  • Jail – a Jail in Fail2ban is the glue that holds it all together – this ties a Filter, together with an Action and the relevant log file.

Install Fail2ban

Installing Fail2ban on Debian/ Proxmox is as easy as it gets – just use the apt package manager.

apt-get install fail2ban

Fail2ban is mostly Python, so it’ll need to be installed on the system or apt-get  will install it as a dependency.

Note: by default Fail2ban will enable itself on SSH connections, blocking invalid IPs after 6 invalid attempts. 

Configure Fail2ban for the Proxmox Web GUI

There are several steps to setting up Fail2ban. As mentioned earlier in the post, we want to ban any users IP address from accessing the Proxmox Web GUI if they have failed to authenticate 3 times. We shouldn’t block them indefinitely because it may be a simple password issue that they can resolve with the account administrator. We’ll configure Fail2ban to ban failed attempts for an hour.

Because banning a user after 3 invalid attempts is a fairly basic thing in the world of Fail2ban, we won’t need to create an Action as listed above. We’ll need to create a Jail and a Filter.

The Jail

A Jail in Fail2ban is the core configuration that  combines a Filter, an Action (although this may be default Fail2ban behaviour) and a log file.

The default configuration for Fail2ban is found in /etc/fail2ban/jail.conf and contains many predefined entries for common processes such as FTP and Apache. We shouldn’t edit this file directly when adding new entries, instead, we should create the below file which will be used to override the default jail.conf.

vi /etc/fail2ban/jail.local

Add the following (this file may not already exist):

[proxmox-web-gui]
enabled  = true
port     = http,https,8006
filter   = proxmox-web-gui
logpath  = /var/log/daemon.log
maxretry = 3
bantime = 3600

The above entry has set a ruleset name of proxmox-web-gui, and the following:

  • enabled – this simply states that this ruleset is active.
  • port – set sthe port that any bans should act on
  • filter – this sets the file name of the filter that we’ll use to detect any login failures. More about this in the next section.
  • logpath – the name or pattern (for example /var/log/apache/*.log) of the log to monitor for the failed logins. This is the file that the above filter will work on.
  • maxretry – this is how many times should the filter detect a problem before starting the ban.
  • bantime – this is how long, in minutes, that the ban be in effect for.

The Filter

Now that we have specified the log file to look in we need to specify how to find the event we need to look for. For our example, Proxmox writes a specific string each time a failed login occurs which looks like the belew:

authentication failure; rhost=10.10.10.10 [email protected] msg=no such user ('[email protected]')

Our Filter, therefore, needs to look for this text and pull out the IP address.

Create a Filter file called proxmox-web-gui.conf in /etc/fail2ban/filter.d/.

vi /etc/fail2ban/filter.d/proxmox-web-gui.conf

Add the following:

[Definition]
failregex = pvedaemon\[[0-9]+\]: authentication failure; rhost=<HOST> user=.* msg=.*

This will match the text that Proxmox writes to the daemon.log file when a failed login is detected. It’s got a Fail2ban specific keyword <HOST> which is what’s used to indicate to Fail2ban where the offending IP address is in the log entry. Fail2ban can then block this IP address as indicated in our Jail file.

Testing Fail2ban Filters

Fail2ban provides a nice little utility to test your Filter definitions to make sure they are working as you intend. First things first – we need an entry in our log file for an invalid login attempt. Go to your Proxmox Web GUI and enter some invalid login credentials.

The command to use is fail2ban-regex which has two parameters; the log file location and the Filter location.

fail2ban-regex /var/log/daemon.log /etc/fail2ban/filter.d/proxmox-web-gui.conf

An example of the output is below. The text Success, the total number of match is 1 states that there is one match in the log for our pattern in the proxmox-web-gui.conf file.

fail2ban-regex /var/log/daemon.log /etc/fail2ban/filter.d/proxmox-web-gui.conf

Running tests
=============

Use regex file : /etc/fail2ban/filter.d/proxmox-web-gui.conf
Use log file   : /var/log/daemon.log


Results
=======

Failregex
|- Regular expressions:
|  [1] pvedaemon\[[0-9]+\]: authentication failure; rhost=<HOST> user=.* msg=.*
|
`- Number of matches:
   [1] 1 match(es)

Ignoreregex
|- Regular expressions:
|
`- Number of matches:

Summary
=======

Addresses found:
[1]
    10.27.4.98 (Fri May 29 12:31:14 2015)

Date template hits:
770 hit(s): MONTH Day Hour:Minute:Second
0 hit(s): WEEKDAY MONTH Day Hour:Minute:Second Year
0 hit(s): WEEKDAY MONTH Day Hour:Minute:Second
0 hit(s): Year/Month/Day Hour:Minute:Second
0 hit(s): Day/Month/Year Hour:Minute:Second
0 hit(s): Day/Month/Year Hour:Minute:Second
0 hit(s): Day/MONTH/Year:Hour:Minute:Second
0 hit(s): Month/Day/Year:Hour:Minute:Second
0 hit(s): Year-Month-Day Hour:Minute:Second
0 hit(s): Year.Month.Day Hour:Minute:Second
0 hit(s): Day-MONTH-Year Hour:Minute:Second[.Millisecond]
0 hit(s): Day-Month-Year Hour:Minute:Second
0 hit(s): TAI64N
0 hit(s): Epoch
0 hit(s): ISO 8601
0 hit(s): Hour:Minute:Second
0 hit(s): <Month/Day/Year@Hour:Minute:Second>

Success, the total number of match is 1

However, look at the above section 'Running tests' which could contain important
information.

Restart fail2ban for the new Jail to be loaded.

service fail2ban restart

To check your new Jail has been loaded, run the following command and look for the proxmox-web-gui Jail name next to Jail List.

fail2ban-client -v status
INFO   Using socket file /var/run/fail2ban/fail2ban.sock
Status
|- Number of jail:      2
`- Jail list:           ssh, proxmox-web-gui

Try to log into the Proxmox Web GUI with an incorrect user 3 and see your IP address appear in the Currently banned section.

fail2ban-client -v status proxmox-web-gui
INFO   Using socket file /var/run/fail2ban/fail2ban.sock
Status for the jail: proxmox-web-gui
|- filter
|  |- File list:        /var/log/daemon.log
|  |- Currently failed: 0
|  `- Total failed:     3
`- action
   |- Currently banned: 1
   |  `- IP list:       10.10.10.10
   `- Total banned:     1

 


Visit our advertisers

Quick Poll

What type of VPN protocol do you use?

Visit our advertisers