Author Archives: James Coyle

Gitlab error “fatal: The remote end hung up unexpectedly” Again

Category : How-to

Get Social!

gitlabI previously wrote in this blog post about how to fix an error with Gitlab. The error was presented when using the  git push command with a remote repository that uses the Gitlabs HTTP protocol and not the SSH protocol.

The following error was presented in the Git client when using the git push command:

git push -u origin master
Counting objects: 556, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (539/539), done.
efrror: RPC failed; result=22, HTTP code = 413367.00 KiB/s
atal: The remote end hung up unexpectedly
Writing objects: 100% (556/556), 1.45 MiB | 367.00 KiB/s, done.
Total 556 (delta 282), reused 0 (delta 0)
fatal: The remote end hung up unexpectedly
Everything up-to-date

It seems that the issue is Nginx, Gitlab’s HTTP server, is not configured to receive large amounts of data. We need to specify the attribute client_max_body_size in Gitlab’s Nginx configuration file and specify the maximum amount of data Nginx will accept.

Open the configuration file and find the line location @gitlab.

vi /etc/nginx/sites-available/gitlab

Add the client_max_body_size attribute and specify the size value to use.

client_max_body_size 20M

The M stands for megabyte – you can also use G for gigabytes.

If the size of your git push ever exceeds the above value, you will have to either increase the size further or reduce your git commit sizes.

Your completed /etc/nginx/sites-available/gitlab file should look like the below example which has a 20MB upload limit.

upstream gitlab {
  server unix:/home/git/gitlab/tmp/sockets/gitlab.socket;
}

server {
  listen *:80 default_server;         # e.g., listen 192.168.1.1:80; In most cases *:80 is a good idea
  server_name YOUR_SERVER_FQDN;     # e.g., server_name source.example.com;
  server_tokens off;     # don't show the version number, a security best practice
  root /home/git/gitlab/public;

  # Increase this if you want to upload large attachments
  # Or if you want to accept large git objects over http
  client_max_body_size 20m;

  # individual nginx logs for this gitlab vhost
  access_log  /var/log/nginx/gitlab_access.log;
  error_log   /var/log/nginx/gitlab_error.log;

  location / {
    # serve static files from defined root folder;.
    # @gitlab is a named location for the upstream fallback, see below
    try_files $uri $uri/index.html $uri.html @gitlab;
  }

  # if a file, which is not found in the root folder is requested,
  # then the proxy pass the request to the upsteam (gitlab unicorn)
  location @gitlab {
    proxy_read_timeout 300; # Some requests take more than 30 seconds.
    proxy_connect_timeout 300; # Some requests take more than 30 seconds.
    proxy_redirect     off;

    proxy_set_header   X-Forwarded-Proto $scheme;
    proxy_set_header   Host              $http_host;
    proxy_set_header   X-Real-IP         $remote_addr;
    proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;

    proxy_pass http://gitlab;
  }

  error_page 502 /502.html;
}

 


Windows Error: “The program can’t start because WMVCORE.DLL is missing from your computer” in Windows Server 2008″

Get Social!

I recently received the below error when running Blue Iris CCTV software on Windows Server 2008.

The program can’t start because WMVCORE.DLL is missing from your computer. Try reinstalling the program to fix this problem.

wmvcore.dll-missing

The error is caused by not having the Windows Media Player libraries installed. You can easily resolve the error by installing the Desktop Experience for Windows Server 2008 which includes Windows Media Player.

Click the start menu and start to type Turn Windows features on or off and click it when it appears in the menu.

turn-windows-features-on-or-off

Click Features and click Add Features. Find the Desktop Experience entry and tick the check box. Click Add Required Features in the new dialogue which pops up.

desktop-experience

Click Next and then Install to install the required features. Click Close and then Yes to restart your computer and complete the installation.

When your computer restarts, click Close and Windows Media Player will now be available.

desktop-experience-complete


Ceph Storage on Proxmox

Get Social!

ceph-logoCeph is an open source storage platform which is designed for modern storage needs. Ceph is scalable to the exabyte level and designed to have no single points of failure making it ideal for applications which require highly available flexible storage.

Since Proxmox 3.2, Ceph is now supported as both a client and server, the client is for back end storage for VMs and the server for configuring storage devices. This means that a Ceph storage cluster can now be administered through the Proxmox web GUI and therefore can be centrally managed from a single location. In addition, as Proxmox now manages the Ceph server the config can be stored in Proxmox’ shared file system meaning that the configuration is immediately replicated throughout the entire cluster.

The below diagram shows the layout of an example Proxmox cluster with Ceph storage.

  • 2 nodes are used dedicated to running VMs and use the Ceph storage hosted by the other nodes.
  • Two networks are used, one for management and application traffic and one for Ceph traffic only. This helps to maintain sufficient bandwidth for storage requirements without affecting the applications which are hosted by the VMs.

ceph-infrastructure-proxmox

Before getting started with setting up the Ceph cluster, we need to do some preparation work. Make sure the following prerequisites are met before continuing the tutorial.

  • You have Proxmox cluster with the latest packages from the pvetest repository. You must have at least three nodes in your cluster. See How to set up a cluster.
  • SSH Keys are set up between all nodes in your cluster – Proxmox does this automatically as part of clustering but if you are using a different user, you may need to set them up manually.
  • NTP is set up on all nodes in your cluster to keep the time in sync. You can install it with: apt-get install ntp

promox-ceph-3-nodesThe rest of this tutorial will assume that you have three nodes which are all clustered into a single Proxmox cluster. I will refer to three host names which are all resolvable via my LAN DNS server; prox1, prox2 and prox3 which are all on the jamescoyle.net domain. The image to the left is what is displayed in the Proxmox web GUI and details all three nodes in a single Proxmox cluster. Each of these nodes has two disks configured; one which Proxmox is installed onto and provides a small ‘local’ storage device which is displayed in the image to the left and one which is going to be used for the Ceph storage. The below output shows the storage available, which is exactly the same on each host. /dev/vda is the root partition containing the Proxmox install and /dev/vdb is an untouched partition which will be used for Ceph.

root@prox1:~# fdisk -l | grep /dev/vd
Disk /dev/vdb doesn't contain a valid partition table
Disk /dev/mapper/pve-root doesn't contain a valid partition table
Disk /dev/mapper/pve-swap doesn't contain a valid partition table
Disk /dev/mapper/pve-data doesn't contain a valid partition table
Disk /dev/vda: 21.5 GB, 21474836480 bytes
/dev/vda1   *        2048     1048575      523264   83  Linux
/dev/vda2         1048576    41943039    20447232   8e  Linux LVM
Disk /dev/vdb: 107.4 GB, 107374182400 bytes

Now that I’ve set the scene, lets start to put together our Ceph cluster! Before using the new Proxmox web GUI you must run a few SSH commands to set up the initial Ceph instance.

Run the below command on all of the nodes which you will use as a Ceph server. This will download and set up the latest Ceph packages.

pveceph install

Create the Ceph config file by initialising pveceph. The Ceph config file will be created in your /etc/pve/ directory called ceph.conf. You should only run this on one node.

pveceph init --network 192.168.50.0/24

The next step is to set up the Ceph monitors for your cluster. So that you don’t have a single point of failure, you will need at least 3 monitors. You must also have an uneven number of monitors – 3, 5, 7, etc.

pveceph createmon

The rest of the configuration can be completed with the Proxmox web GUI. Log in to your Proxmox web GUI and click on one of your Proxmox nodes on the left hand side, then click the Ceph tab.

proxmox-ceph-status-tab

Next, you will add a disk to the Ceph cluster. Each disk creates to as an OSD in Ceph which is a storage object used later by the Ceph storage pool. Click on the Disks tab at the bottom of the screen and choose the disk you would like to add to the Ceph cluster. Click the Create: OSD button and click Create to create an OSD. Repeat these steps for each Proxmox server which will be used to host storage for the Ceph cluster.

ceph-create-osd

If the Create: OSD button is greyed out, it’s because the disk is not in a state where Ceph can use it. It’s likely because you have partitions on your disk. Run the fdisk command on the disk and press d to delete the partitions and w to save the changes. For example:

fdisk /dev/vdb

The last step in creating the Ceph storage cluster is to create a storage pool. Click the Pools tab and click Create. Enter the below values into the new Create Pool dialogue box:

  • Name: name to use for the storage pool.
  • Size: the number of replicas to use for a working cluster. A replica is the number of times the data is stored across nodes.
  • Min. Size: the minimum of replicas which can be used.
  • Crush RuleSet:
  • pg_num: this is the placement group count which you have to calculate based on the number os OSDs you have. To calculate your placement group count, multiply the amount of OSDs you have by 100 and divide it by the number of number of times each part of data is stored. The default is to store each part of data twice which means that if a disk fails, you won’t loose the data because it’s stored twice.For our example,3 OSDs * 100 = 300
    Divided by replicas, 300 / 2 = 150

ceph-create-pool

The Ceph storage pool is now set up and available to use for your KVM images. You can check the status of the Ceph storage pool by clicking on the Status tab.

ceph-status-screen

See my blog post on mounting Ceph storage on Proxmox.


Create a 3 Node Ceph Storage Cluster

Category : How-to

Get Social!

ceph-logoCeph is an open source storage platform which is designed for modern storage needs. Ceph is scalable to the exabyte level and designed to have no single points of failure making it ideal for applications which require highly available flexible storage.

The below diagram shows the layout of an example 3 node cluster with Ceph storage. Two network interfaces can be used to increase bandwidth and redundancy. This can help to maintain sufficient bandwidth for storage requirements without affecting client applications.

ceph-infrastructure

This example will create a 3 node Ceph cluster with no single point of failure to provide highly redundant storage. I will refer to three host names which are all resolvable via my LAN DNS server; ceph1ceph2 and ceph3 which are all on the jamescoyle.net domain. Each of these nodes has two disks configured; one which runs the Linux OS and one which is going to be used for the Ceph storage. The below output shows the storage available, which is exactly the same on each host. /dev/vda is the root partition containing the OS install and /dev/vdb is an untouched partition which will be used for Ceph.

root@ceph1:~# fdisk -l | grep /dev/vd
Disk /dev/vdb doesn't contain a valid partition table
Disk /dev/mapper/pve-root doesn't contain a valid partition table
Disk /dev/mapper/pve-swap doesn't contain a valid partition table
Disk /dev/mapper/pve-data doesn't contain a valid partition table
Disk /dev/vda: 21.5 GB, 21474836480 bytes
/dev/vda1   *        2048     1048575      523264   83  Linux
/dev/vda2         1048576    41943039    20447232   8e  Linux LVM
Disk /dev/vdb: 107.4 GB, 107374182400 bytes

Before getting started with setting up the Ceph cluster, you need to do some preparation work. Make sure the following prerequisites are met before continuing the tutorial.

You will need to perform the following steps on all nodes in your Ceph cluster. First you will add the Ceph repositories and download the key to make the latest Ceph packages available. Add the following line to a new /etc/apt/sources.list.d/ file.

vi /etc/apt/sources.list.d/ceph.list

Add the below entry, save and close the file.

deb http://ceph.com/debian-dumpling/ wheezy main

Download the key from Ceph’s git page and install it.

wget -q -O- 'https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc' | apt-key add -

Update all local repository cache.

sudo apt-get update

Note: if you see the below code when running apt-get update then the above wget command has failed – it could be because the Ceph git URL has changed.

W: GPG error: http://ceph.com wheezy Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 7EBFDD5D17ED316D

Run the following commands on just one of your Ceph nodes. I’ll use ceph1 for this example. Update your local package cache and install the ceph-deploy command.

apt-get install ceph-deploy -y

Create the first Ceph storage cluster and specify the first server in your cluster by either hostname or IP address for [SERVER].

ceph-deploy new [SERVER]

For example

ceph-deploy new ceph1.jamescoyle.net

Now deploy Ceph onto all nodes which will be used for the Ceph storage.  Replace the below [SERVER] tags with the host name or IP address of your Ceph cluster including the host you are running the command on. See this post if you get a key error here.

ceph-deploy install [SERVER1] [SERVER2] [SERVER3] [SERVER...]

For example

ceph-deploy install ceph1.jamescoyle.net ceph2.jamescoyle.net ceph3.jamescoyle.net

Install the Ceph monitor and accept the key warning as keys are generated. So that you don’t have a single point of failure, you will need at least 3 monitors. You must also have an uneven number of monitors – 3, 5, 7, etc. Again, you will need to replace the [SERVER] tags with your server names or IP addresses.

ceph-deploy mon create [SERVER1] [SERVER2] [SERVER3] [SERVER...]

Example

ceph-deploy mon create ceph1.jamescoyle.net ceph2.jamescoyle.net ceph3.jamescoyle.net

Now gather the keys of the deployed installation, just on your primary server.

ceph-deploy gatherkeys [SERVER1]

Example

ceph-deploy gatherkeys ceph1.jamescoyle.net

It’s now time to start adding storage to the Ceph cluster. The fdisk output at the top of this page shows that the disk I’m going to use for Ceph is /dev/vdb, which is the same for all the nodes in my cluster. Using Ceph terminology, we will create an OSD based on each disk in the cluster. We could have used a file system location instead of a whole disk but, for this example, we will use a whole disk. Use the below command, changing [SERVER] to the name of the Ceph server which houses the disk and [DISK] to the disk representation in /dev/.

ceph-deploy osd --zap-disk create [SERVER]:[DISK]

For example

ceph-deploy osd --zap-disk create ceph1.jamescoyle.net:vdb

If the command fails, it’s likely because you have partitions on your disk. Run the fdisk command on the disk and press d to delete the partitions and w to save the changes. For example:

fdisk /dev/vdb

Run the osd command for all nodes in your Ceph cluster

ceph-deploy osd --zap-disk create ceph2.jamescoyle.net:vdb
ceph-deploy osd --zap-disk create ceph3.jamescoyle.net:vdb

We now have to calculate the number of placement groups (PG) for our storage pool. A storage pool is a collection of OSDs, 3 in our case, which should each contain around 100 placement groups. Each placement group will hold your client data and map it to an OSD whilst providing redundancy to protect against disk failure.

To calculate your placement group count, multiply the amount of OSDs you have by 100 and divide it by the number of number of times each part of data is stored. The default is to store each part of data twice which means that if a disk fails, you won’t loose the data because it’s stored twice.

For our example,

3 OSDs * 100 = 300
Divided by replicas, 300 / 2 = 150

Now lets create the storage pool! Use the below command and substitute [NAME] with the name to give this storage pool and [PG] with the number we just calculated.

ceph osd pool create [NAME] [PG]

For example

ceph osd pool create datastore 150
pool 'datastore' created

You have now completed the set up for the Ceph storage pool. See my blog post on mounting Ceph storage on Proxmox.


Manually Mount a Ceph Storage Pool in Proxmox

Get Social!

ceph-logoThe latest BETA of Proxmox, and soon to be released 3.2 comes with the Ceph client automatically installed which makes mount Ceph storage pools painless.

You can mount the Ceph storage pool using the Proxmox web GUI.

You may need to copy the Ceph storage pool keyring from your Ceph server to your Proxmox server. If you use authentication on your Ceph storage pool, log in to your Ceph server via SSH. Run the below scp command to copy your Ceph keyring to your Proxmox server. You will need to replace [PROXMOX_SERVER] with the host name or IP address of your Proxmox server and [STORAGE_NAME] is the name you will use for your Proxmox storage mount which will be used in the web GUI.

scp /etc/ceph/ceph.client.admin.keyring [PROXMOX_SERVER]:/etc/pve/priv/ceph/[STORAGE_NAME].keyring

For example

scp /etc/ceph/ceph.client.admin.keyring prox3.jamescoyle.net:/etc/pve/priv/ceph/ceph_storage.keyring

To use the Proxmox GUI to mount a Ceph storage pool, login to the web GUI and click on the Datacenter folder in the top left, then click the Storage tab, Add and finally RBD.

proxmox-add-ceph-rbd

Enter the details of your Ceph storage pool in the new dialogue box which appears.

  • ID: the name which this storage mount will have in Proxmox. If you use authentication on your Ceph server, you will need to use the same name as the above  [STORAGE_NAME] field.
  • Pool: the Ceph storage pool name.
  • Monitor Host: the host or IP addresses of your Ceph monitor servers. You can enter a single host or multiple hosts separated by a semicolon [;].
  • User name: the named use to use when connecting to the Ceph storage pool. If you are not using user authentication, set this to the below value.
    admin (optional, default = admin)

proxmox-ceph-add-rbd

Don’t forget to make sure that the Proxmox host can reach the remote Ceph monitor hosts – by default a Ceph monitor runs on port 6789.

proxmox-ceph-storage-mount-rbd


Ceph Error “Warning: the ECDSA host key for ‘SERVER’ differs from the key for the IP address ‘IP'”

Category : How-to

Get Social!

ceph-logoI recently received the below error when installing Ceph on Proxmox which related the the key for the local machine. After running the ceph-deploy install command I was presented with an error and the Ceph command halted.

ceph-deploy install ceph1.jamescoyle.net ceph2.jamescoyle.net ceph3.jamescoyle.net

After agreeing to the initial prompt I received the below error.

Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ceph1.jamescoyle.net' (ECDSA) to the list of known hosts.
Warning: the ECDSA host key for 'ceph1.jamescoyle.net' differs from the key for the IP address '192.168.50.102'
Offending key for IP in /etc/ssh/ssh_known_hosts:2
Are you sure you want to continue connecting (yes/no)? yes
Write failed: Broken pipe
[ceph_deploy][ERROR ] RuntimeError: connecting to host: ceph1.jamescoyle.net resulted in errors: HostNotFound ceph1.jamescoyle.net

You need to remove the local key using ssh-keygen and then allow the machine to re-add the key when you first connect to it. The above error was complaining about the key for IP 192.168.50.102, therefore add this IP to the end of the below command.

ssh-keygen -R [IP]

Example:

ssh-keygen -R 192.168.50.102
/root/.ssh/known_hosts updated.
Original contents retained as /root/.ssh/known_hosts.old

Now rerun your Ceph command and accept the key warning and your Ceph command should complete.

 


Visit our advertisers

Quick Poll

Which type of virtualisation do you use?
  • Add your answer

Visit our advertisers