GlusterFS Error cannot open /dev/fuse

  • 0

GlusterFS Error cannot open /dev/fuse

Category : How-to

Get Social!

After installing glusterfs-client on my Debian server I received the below error when trying to mount a remote GlusterFS volume. The error indicates that the device at /dev/fuse cannot be found, however ls showed that it was available.

This was the error displayed in the Gluster log after running the mount command:

A quick check of the kernel fuse module using modprobe gave an error:

And some Googleing indicated that it’s because fuse-utils was missing. In my case it wasn’t.

Further investigation showed that the kernel had recently been updated, but the machine hadn’t been restarted so the latest installed kernel wasn’t the kernel that was running. There seemed to be some kind of mismatch between the loaded kernel and the fuse library.

A reboot of the machine fixed the issue – the fuse module loaded correctly and the Gluster mount executed without error.


  • 0

GlusterFS Error volume add-brick: failed: Pre Validation failed on BRICK is already part of a volume

Category : How-to

Get Social!

I received the below error today after I tried to add a ‘new’ brick to a GlusterFS volume. I’ve put the word ‘new’ into quotes because although the brick was new to the GlusterFS volume, the disk being added had been used as a brick before. The disk had all data removed from it, however somehow GlusterFS knew that the disk held some remnants of the previous brick.

The following command failed, trying to add the new brick:

The error received:

Solution!

The solution is to use setfattr to clear the hidden filesystem attributes containing GlusterFS information about the bricks previous life. Run the following commands on the server that has the drive that you’re trying to add as a new brick.

Of course, you should also ensure that the filesystem you’re adding is cleared, especially the .glusterfs hidden directory.

And that’s it! Try running the add-brick command again, and you should be in business.


  • 0

GlusterFS Mount failed. Please check the log file for more details.

Category : How-to

Get Social!

gluster-orange-antYou may get the following error when trying to mount a GlusterFS volume locally. The error displayed gives no indication why the volume failed to mount, but it does hint at where you can get more information about the error.

This is the error presented when running the mount command:

The log file could be in numerous places, depending on your Linux distribution and Gluster settings, however generally it will be in /var/log/glusterfs.

Take a look at the log file for further information on why the volume cannot be mounted. An example is included below, showing an issue with the fuse kernel module.

Your issue could vary, and as such we can’t cover every eventuality here. At least you now know how to get more details around your specific issue.


  • 2

Synchronise a GlusterFS volume to a remote site using geo replication

Get Social!

gluster-orange-antGlusterFS can be used to synchronise a directory to a remote server on a local network for data redundancy or load balancing to provide a highly scalable and available file system.

The problem is when the storage you would like to replicate to is on a remote network, possibly in a different location, GlusterFS does not work very well. This is because GlusterFS is not designed to work when there is a high latency between replication nodes.

GlusterFS provides a feature called geo replication to perform batch based replication of a local volume to a remote machine over SSH.

The below example will use three servers:

  • gfs1.jamescoyle.net is one of the two running GlusterFS volume servers.
  • gfs2.jamescoyle.net is the second of the two running GlusterFS volume servers. gfs1 and gfs2 both server a single GlusterFS replicated volume called datastore.
  • remote.jamescoyle.net is the remote file server which the GlusterFS volume will be replicated to.

GlusterFS uses an SSH connection to the remote host using SSH keys instead of passwords. We’ll need to create an SSH key using ssh-keygen to use for our connection. Run the below command and press return when asked to enter the passphrase to create a key without a passphrase. 

The output will look like the below:

Now you need to copy the public certificate to your remote server in the authorized_keys file. The remote user must be a super user (currently a limitation of GlusterFS) which is root in the below example. If you have multiple GlusterFS volumes in a cluster then you will need to copy the key to all GlusterFS servers.

Make sure the remote server has glusterfs-server installed. Run the below command to install glusterfs-server on remote.jamescoyle.net. You may need to use yum instead of apt-get for Red Hat versions of Linux.

Create a folder on remote.jamescoyle.net which will be used for the remote replication. All data which transferrs to this machine will be stored in this folder.

Create the geo-replication volume with Gluster and replace the below values with your own:

  • [SOURCE_DATASTORE] – is the local Gluster data volume which will be replicated to the remote server.
  • [REMOTE_SERVER] – is the remote server to receive all the replication data.
  • [REMOATE_PATH] – is the path on the remote server to store the files.

Example:

Sometimes on the remote machine, gsyncd (part of the GlusterFS package) may be installed in a different location to the local GlusterFS nodes.

Your log file may show a message similar to below:

In this scenario you can specify the config command the remote gsyncd location.

You will then need to run the start command to start the volume synchronisation.

You can view the status of your replication task by running the status command.

You can stop your volume replication at any time by running the stop command.


  • 1

Advanced GlusterFS Log Rotation

Category : How-to

Get Social!

gluster-orange-ant

If installing GlusterFS on Debian using the launchpad repository then a log rotate entry will be automatically set up. This entry handles all the logs created by the GlusterFS application by rotating them daily and keeping 7 days of old log files.

Other packages or install methods may  vary and log and configuration paths may be different.

On Debian and Ubuntu the log files are kept in the following location:

And the logrotate configuration files is found here:

The default configuration is to rotate the logs each day, compress old logs and keep them for 7 days. Here is the default configuration file:

You can edit this file directly to make any changes you may need in your GlusterFS environment. See my cheat sheet for details on the logrotate commands.


  • 2

Simple GlusterFS log rotation

Category : How-to

Get Social!

gluster-orange-ant

You’ll be glad to know that GlusterFS has built in log rotation! This means that you can use a simple gluster command to rotate the log for a specific volume. This is very helpful when troubleshooting where it can be easiest to truncate logs before regenerating the issue.

For enduring log rotation I recommend using logrotate which I will cover in a future blog post.

Logs are rotated per volume, so you will need to know the volume name before issuing the command to rotate the log. Use the below command to list all the volumes available on your server:

Use the gluster command, replacing [VOL_NAME] with the volume name of the log file you would like to rotate.

In the below example, the logs for volume datastore will be rotated.

Below is the result on the file system displayed with the ls command.

And that’s it! Your log file will be moved and have a time stamp appended, and a new log will be started for the volume.

See my other post on using logrotate for more advanced log rotation.