GlusterFS Error cannot open /dev/fuse

  • 0

GlusterFS Error cannot open /dev/fuse

Category : How-to

Get Social!

After installing glusterfs-client on my Debian server I received the below error when trying to mount a remote GlusterFS volume. The error indicates that the device at /dev/fuse cannot be found, however ls showed that it was available.

This was the error displayed in the Gluster log after running the mount command:

A quick check of the kernel fuse module using modprobe gave an error:

And some Googleing indicated that it’s because fuse-utils was missing. In my case it wasn’t.

Further investigation showed that the kernel had recently been updated, but the machine hadn’t been restarted so the latest installed kernel wasn’t the kernel that was running. There seemed to be some kind of mismatch between the loaded kernel and the fuse library.

A reboot of the machine fixed the issue – the fuse module loaded correctly and the Gluster mount executed without error.


  • 0

GlusterFS Mount failed. Please check the log file for more details.

Category : How-to

Get Social!

gluster-orange-antYou may get the following error when trying to mount a GlusterFS volume locally. The error displayed gives no indication why the volume failed to mount, but it does hint at where you can get more information about the error.

This is the error presented when running the mount command:

The log file could be in numerous places, depending on your Linux distribution and Gluster settings, however generally it will be in /var/log/glusterfs.

Take a look at the log file for further information on why the volume cannot be mounted. An example is included below, showing an issue with the fuse kernel module.

Your issue could vary, and as such we can’t cover every eventuality here. At least you now know how to get more details around your specific issue.


  • 0

Persistent Ceph Mount Point

Tags :

Category : How-to

Get Social!

ceph-logoOnce you’ve got a Ceph cluster up and running you’re going to want to mount it somewhere. This guide assumes that the mount point will be on a machine that isn’t running Ceph, however if you’re mounting the storage on one of the Ceph server nodes then you can skip the package installation steps.

Install the Ceph Client

Before we start mounting anything, we’re going to need the required software installed. Assuming you’re on Debian run the below commands to add the key and the software repository for the Ceph binaries.

Then run the apt-get commands to update your software index and install the Ceph binaries for the client.

Mount a Ceph device as a folder

Here we’re going to use /mnt/ha-pool as the mount point but you can change that to whatever you’d like. Run this command on any machine that you’d like to mount the Ceph volume on.

Then we need to export the key so that the ceph-client can authenticate with the Ceph daemon. You could turn authentication off, or even create a non-admin user secret but for this tutorial we’ll just use the admin user. Run this command on your admin machine for your Ceph cluster (NOT on the client you’re setting up the mount point).

You’ll be presented with a string of letters and numbers. Copy this and add it to a file stored on your Ceph client machine. This is the ‘password’ or secret that the Ceph client will use to authenticate with the Ceph server. Paste the string into a file – you can store this anywhere but we’ll use /etc/ceph/admin.secret.

Automatic mount

If you’d like the Ceph mount point to persist across client machine reboots then you’ll need to add an entry to /etc/fstab. Run the below command to add an entry to your fstab file so that the Ceph volume will be automatically mounted on machine start. This will mount the Ceph volume at /mnt/ha-pool and is referencing the Ceph monitor server nodes ceph1, ceph2 and ceph3 – make sure you change these values for your environment. You don’t have to specify more than one Ceph monitor server node, but it makes sense, just incase one of your nodes fails.

Then to mount the volume, run the below mount command

Manually mount filesystem

If you don’t need the mount to persist you can simply use the mount command. The parameters are very similar to the above section, with the Ceph monitor servers, secret file and mount point all specified. This will mount the Ceph volume at /mnt/ha-pool and is referencing the Ceph monitor server nodes ceph1, ceph2 and ceph3 – make sure you change these values for your environment.

Ceph mount ports and additional options

By default, and if left unspecified like the above examples, the Ceph client will use 6789 for your monitor server daemon. If you’ve specified a different port for your monitor daemon then you can specify them in the mount command. The same syntax can be used in your fstab.

You can also specify your secret key directly, rather than a file that contains it. I won’t go into the security implications of this here, but I’m sure you can imagine one or two. Again, the same syntax can be used in your fstab.


  • 5

Small Scale Ceph Replicated Storage

Category : How-to

Get Social!

I’ve written a few posts about Ceph, how it works and how it’s set up and it mostly revolves around large scale storage for storing things like virtual machines. This post will focus on using Ceph  provide fault tolerant storage for a small amount of data in a low resource environment. Because of this, the main focus has been moved away from performance and switched to:

  • availability – the storage should always be available and recoverable in the event of disaster
  • portability – the storage isn’t tied to a machine and can be moved with relative ease.
  • scalability – more machines can use the storage as required.

This tutorial will focus on a small scale Ceph setup, fit for something like a Raspberry Pi or low resource VPS. We’ll use 3 machines but you could easily add more machines if your scenario requires it.

If you are looking for a larger setup, then see this blog post on installing Ceph.

ceph-local

The above diagram shows the topology of the layout. Each machine will have a file /ceph-file that will be mounted as a block device on /dev/loop0 and that’s the space that will be assigned to Ceph. Ceph will replicate any data stored to the file and ensure the data is available to all Ceph clients. The Ceph storage will be accessed from a mountpoint at /mnt/ha-pool.

Ceph block device

The first step in creating a Ceph storage pool is to set aside some storage that can be used by Ceph. Ceph stores everything twice, by default, so whatever storage you provision will be halved. For this example we’re going to use a file created with dd as the Ceph storage device, however you could use a drive mounted in /dev/ if you have one. A whole drive is by far the preferred solution, however as I’ve stated, the main goal of this post isn’t just performance.

If you’re going to use a file for storage, follow my post on creating a block device from a file and mount it on loop0. Otherwise you can continue to the next step.

OpenVZ: if you’re using Ceph inside of an OpenVZ container, make sure you pass the loop device through to the container.

Installing Ceph

At this point it’s worth noting that Ceph, in addition to the application requirements, will use approximately 1MB of RAM for each GB of storage provisioned. This means that 1TB of provisioned storage (which in today’s world is rather small) would take 1GB of RAM plus the requirements of running the Ceph daemons. For our low memory footprint, only provision the storage that you’ll need.

Before starting the install, you’ll need a couple of things in place:

  • SSH Keys are set up between all nodes in your cluster – see this post for information on how to set up SSH Keys. For security it’s good practice to set up a new user on all machines you’re going to install Ceph onto and use it to run Ceph. The key should also be copied to all machines using the ssh-copy-id command.
  • NTP is set up on all nodes in your cluster to keep the time in sync. You can install it with: apt-get install ntp

The following commands are for installing Ceph on Debian (wheezy) and should be executed on all machines that need to run Ceph. In our example, these commands will be executed on Server 1Server 2 and Server 3.

First let’s add the release key and repositories to the apt package manager. Run the following as root:

Next let’s update our apt cache and install Ceph and a few other bits.

Setup and configuring for minimal resource requirements

The next step should be done on just one of your Ceph machines. This will create the monitor service and make each machine aware of the other machines running Ceph.

The command references each machine you’re going to be running Ceph on by hostname or DNS entry. Before running the command, make sure that all of your machines resolve via DNS or hosts file. Because I’m only running this in a lab, I’ve used the hosts file route and added an entry to each machine in the hosts file of all Ceph machines.

Add your Ceph machine IP and hostnames.

You can test that each machine can see the others by using the ping command. If it works then you should be in business!

Once you’re happy that all machines can reference the other machines then run the ceph-deploy command:

If you haven’t used your ssh keys since setting them up you may be presented with the following warning. Just type yes to continue.

We now need to configure Ceph with our low resource settings. These settings are not performance driven, but instead set to minimise system resources.

See ceph.conf for the script and add the content to the ceph.conf file

Create the initial mds daemons, monitor daemons and set the proper permissions on the keyring file.

Test Ceph is deployed and monitors are running

At this point it’s good to take a step back and check everything is up and running. We’ve still not assigned any storage to our Ceph cluster so we can’t run it yet, but we should have the monitor daemons running and the cluster configuration be deployed on all servers.

Run the below command and take a look at the output.

The output should show

As you can see, three Ceph servers are referenced on port 6789 which is the monitor daemon port number.

Add storage to the Ceph cluster

We’ve got our Ceph cluster, and we’ve got our storage device that we created as the first step, it’s time to put the two together. Run the below commands on the same machine that you ran the above steps on. You’ll need to replace /dev/sda with the block device on each ceph machines that you’d  like to use. Note that the block device (sda) does not need to be the same on all machines.

Or…

You can use a directory as storage for Ceph, rather than a block device.

If you’re following this tutorial and creating a loop device to use with Ceph then you’ll need to ensure there is a filesystem on the loop0 device and that it’s mounted. You can skip these next step if you are just using an existing directory.

Run the below commands (if you’re using a loop device) on each of the machines that has a loop device you’d like to use. We’re assuming that you’re loop device is loop0. For this example we’ll run it on each of the three machines; ceph1, ceph2 and ceph3.

You can use a directory path on the Ceph machine as the OSD device. This may be an option if you’re in an OpenVZ or Docker container that doesn’t allow you to pass through block devices.

And then activate the storage:

Mount a Ceph device as a folder

That’s the server side done! The last step to using our Ceph storage cluster is to mount the cluster to a mountpoint on the local filesystem. Here we’re going to use /mnt/ha-pool as the mount point but you can change that to whatever you’d like. Run these commands on any machines that you’d like to mount the Ceph volume on.

First create the mount point where the Ceph storage will be accessible from.

Then we need to export the key so that the ceph-client can authenticate with the Ceph daemon. You could turn authentication off, or even create a non-admin user secret but for this tutorial we’ll just use the admin user.

Then run the below command to add an entry to your fstab file so that the Ceph volume will be automatically mounted on machine start. This will mount the Ceph volume at /mnt/ha-pool.

Finally run the mount command

One last check to make sure you’re up and running:

And that’s it! You have a working Ceph cluster up and running!


  • 1

Use A File As A Linux Block Device

Get Social!

Just like when creating a SWAP file, you can create a file on a disk and present it as a block device. The block device would have a maximum file size of the backing file, and (as long as it’s not in use) be moved around like a normal file. For example, I could create a 1GB file on the filesystem and make Linux treat the file as a disk mounted in /dev/. And guess what – that’s what we’re going to do.

Create a file and filesystem to use as a block device

First off, use dd to create a 1GB file on an existing disk that we’ll use for our storage device:

Then ‘format’ the file to give it the structure of a filesystem. For this example we’re going to use ext4 but you could choose any filesystem that meets your needs.

You’ll be promoted with Proceed anyway?. Type y and press return to proceed with the process.

Mounting a loop device

Before mounting the file we need to check that there is a free /dev/loopX loopback device that we can use to represent our new block device.

Run the below command, and if there is any output then check if it’s one of your loop devices, which will more than likely reference /dev/loop as the mounted device. If you do have a reference to our loop device then see the below section on Unmounting a loop device, or choose a number higher than the highest listed loop device, for example: usually there are several loop devices, starting with loop0 and going up in value to loop1loop2, and so on.

Once you have the file that you’d like to mount and a free loop device then you can go ahead and mount the file as a block device. You have two options:

  1. Mount the file as a block device only
  2. Mount the file as a block device and mount the filesystem of it on a local mount point (eg. /mnt/mymountpoint).

For option 1; to only mount the file as a device in /dev/, run the below command and change /root/diskimage to the path of the file you’d like to mount. loop0 can also be incremented as explained above.

If you’d like this to be remounted after a machine reboot then add the above line to the rc.local file.

And add:

 

For option 2; to mount the file and the filesystem on it, use the mount command. You must have already created the mount point locally before running the command, as you would when mounting a disk or NFS share.

Then run the mount command and specify the loop device, the path of the file and the path to mount the filesystem on:

To check the file has been mounted you can use the df command:

Unmounting a loop device

If you’ve mounted the filesystem on the block device using the mount command then make sure it’s unmounted before proceeding.

To then free the loop0 device (or which ever loop device you’ve used) you’ll need the losetup command with the d switch.

 


  • 3

Manually Mount a Ceph Storage Pool in Proxmox

Get Social!

ceph-logoThe latest BETA of Proxmox, and soon to be released 3.2 comes with the Ceph client automatically installed which makes mount Ceph storage pools painless.

You can mount the Ceph storage pool using the Proxmox web GUI.

You may need to copy the Ceph storage pool keyring from your Ceph server to your Proxmox server. If you use authentication on your Ceph storage pool, log in to your Ceph server via SSH. Run the below scp command to copy your Ceph keyring to your Proxmox server. You will need to replace [PROXMOX_SERVER] with the host name or IP address of your Proxmox server and [STORAGE_NAME] is the name you will use for your Proxmox storage mount which will be used in the web GUI.

For example

To use the Proxmox GUI to mount a Ceph storage pool, login to the web GUI and click on the Datacenter folder in the top left, then click the Storage tab, Add and finally RBD.

proxmox-add-ceph-rbd

Enter the details of your Ceph storage pool in the new dialogue box which appears.

  • ID: the name which this storage mount will have in Proxmox. If you use authentication on your Ceph server, you will need to use the same name as the above  [STORAGE_NAME] field.
  • Pool: the Ceph storage pool name.
  • Monitor Host: the host or IP addresses of your Ceph monitor servers. You can enter a single host or multiple hosts separated by a semicolon [;].
  • User name: the named use to use when connecting to the Ceph storage pool. If you are not using user authentication, set this to the below value.

proxmox-ceph-add-rbd

Don’t forget to make sure that the Proxmox host can reach the remote Ceph monitor hosts – by default a Ceph monitor runs on port 6789.

proxmox-ceph-storage-mount-rbd