Automated Bash MongoDB 3.2 Install Script for Debian/ Ubuntu

Automated Bash MongoDB 3.2 Install Script for Debian/ Ubuntu

Get Social!

mongodb-logoMongoDB is one of the leading noSQL breeds of database that’s been growing in popularity in recent years. The database is available in a ‘community edition’ that’s available for all to use freely.

The database setup and install is mostly straightforward however there are a few steps you have to take to get a MongoDB instance up and running.

The binaries aren’t shipped by the common Linux distributions so you’ll need to add the apt repositories hosted by mongodb.org. After installing the binaries there are a few config options that can be helpful to start with that are not contained in the default MongoDB install.

First off, create a file on your server called install_mongo.sh and copy the content of the script into it found in the below link.

MongoDB 3.2 Bash Install Script

vi install_mongo.sh

Then make the script executable and run it.

chmod +x install_mongo.sh
./install_mongo.sh

Once the script completes you’ll be able to connect to your MongoDB instance with adminadmin.

mongo admin -u admin -p admin

MongoDB shell version: 3.2.7
connecting to: admin
rs1:PRIMARY>

What the MongoDB install script does

The following is a brief outline of the steps the script takes:

  1. Add the apt repository from mongodb.org and associated key.
  2. Install the full mongo-org package containing these packages: mongodb-org-server, mongodb-org-mongos, mongodb-org-shell, mongodb-org-tools.
  3.  Add a basic config file that:
    1. Enforces the wiredTiger storage engine.
    2. Enables remote access by listening on ALL interfaces.
    3. Enables replication, even if it’s just a stand alone node.
    4. Enables user authentication.
  4. Initiates the server as a replication cluster.
  5. Adds a user for administration called admin with password admin.

 


Use DD to Quickly Benchmark Your CPU

Category : How-to

Get Social!

Let me start by saying… this is a quick and dirty method and shouldn’t be used for precise comparisons of CPU performance. That said, it’s perfectly adequate for approximating CPU performance, especially on Cloud hosted VPSs to ensure you’re getting the horsepower that you’re being promised.

The idea here is to force your machine to perform tasks that will be computationally expensive to force your CPU to work at 100 percent and become the bottleneck for the task (rather than disk I/ O, etc.). This task will then be timed. The shorter times will generally represent faster CPU’s and longer results would indicate a slower CPU.

[the_ad id=”2698″]

CPU benchmark

The md5sum command is a tool that creates an MD5 hash of some data. We can generate some data on the fly with dd and pipe it into the md5sum tool to create a computationally expensive task. We’ll limit the data to hash and time the length of time it takes to create the hash.

Run the below to start the test. If your result completes in under 2 seconds then increase the count=1k value to a higher value, for example count=10k.

dd if=/dev/zero bs=1M count=1k | md5sum

You’ll get an output similar to the below output.

1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 2.38909 s, 449 MB/s
cd573cfaace07e7949bc0c46028904ff  -

There are a couple of items that you’re interested in here, and the rest can be ignored.

  • 2.38909 s is the time it took in seconds for the operation to complete. This is the number to use for comparison with other machines – lower is better.
  • 449 MB/s is the speed that the data was fabricated and push into the md5sum tool to be hashed – higher is better.

 

CPU details with cpuinfo

Linux has various nuggets of information about your system available in the proc directory on a linux root partition. You can cat various files, such as /proc/cpuinfo, to see system specifications and metrics.

/proc/cpuinfo

The output will look similar to the below output that shows a Xeon CPU running at a clock speed of 2.50GHz.

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 62
model name      : Intel(R) Xeon(R) CPU @ 2.50GHz
stepping        : 4
microcode       : 0x1
cpu MHz         : 2500.000
cache size      : 30720 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht sysca
ll nx pdpe1gb rdtscp lm constant_tsc nopl xtopology eagerfpu pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx
 f16c rdrand hypervisor lahf_lm xsaveopt fsgsbase smep
bogomips        : 5000.00
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual

 


Use A File As A Linux Block Device

Get Social!

Just like when creating a SWAP file, you can create a file on a disk and present it as a block device. The block device would have a maximum file size of the backing file, and (as long as it’s not in use) be moved around like a normal file. For example, I could create a 1GB file on the filesystem and make Linux treat the file as a disk mounted in /dev/. And guess what – that’s what we’re going to do.

Create a file and filesystem to use as a block device

First off, use dd to create a 1GB file on an existing disk that we’ll use for our storage device:

dd if=/dev/zero of=/root/diskimage bs=1M count=1024

Then ‘format’ the file to give it the structure of a filesystem. For this example we’re going to use ext4 but you could choose any filesystem that meets your needs.

mkfs.ext4 /root/diskimage

You’ll be promoted with Proceed anyway?. Type y and press return to proceed with the process.

mke2fs 1.42.5 (29-Jul-2012)
/root/diskimage is not a block special device.

Proceed anyway? (y,n) y

Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
65536 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376

Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

Mounting a loop device

Before mounting the file we need to check that there is a free /dev/loopX loopback device that we can use to represent our new block device.

Run the below command, and if there is any output then check if it’s one of your loop devices, which will more than likely reference /dev/loop as the mounted device. If you do have a reference to our loop device then see the below section on Unmounting a loop device, or choose a number higher than the highest listed loop device, for example: usually there are several loop devices, starting with loop0 and going up in value to loop1loop2, and so on.

cat /proc/mounts | grep /dev/loop

Once you have the file that you’d like to mount and a free loop device then you can go ahead and mount the file as a block device. You have two options:

  1. Mount the file as a block device only
  2. Mount the file as a block device and mount the filesystem of it on a local mount point (eg. /mnt/mymountpoint).

For option 1; to only mount the file as a device in /dev/, run the below command and change /root/diskimage to the path of the file you’d like to mount. loop0 can also be incremented as explained above.

losetup /dev/loop0 /root/diskimage

If you’d like this to be remounted after a machine reboot then add the above line to the rc.local file.

vi /etc/rc.local

And add:

losetup /dev/loop0 /root/diskimage

 

For option 2; to mount the file and the filesystem on it, use the mount command. You must have already created the mount point locally before running the command, as you would when mounting a disk or NFS share.

mkdir /mnt/mymountpoint

Then run the mount command and specify the loop device, the path of the file and the path to mount the filesystem on:

mount -o loop=/dev/loop0 /root/diskimage /mnt/mymountpoint

To check the file has been mounted you can use the df command:

df -h | grep mymountpoint
/dev/loop0  976M  1.3M  924M  1% /mnt/mymountpoint

Unmounting a loop device

If you’ve mounted the filesystem on the block device using the mount command then make sure it’s unmounted before proceeding.

umount /mnt/mymountpoint

To then free the loop0 device (or which ever loop device you’ve used) you’ll need the losetup command with the d switch.

losetup -d /dev/loop0

 


Script To Create A Swapfile On Linux

Category : How-to

Get Social!

This 5 line script will quickly create a 512MB SWAP file that will be automatically mounted on each machine reboot. It assumes you don’t already have a swap file enabled.

The script creates a 512MB file called .swapfile on your root partition, makes it SWAP format and enables it as available system SWAP. An fstab entry is also added so that it’s mounted after a machine reboot.

You can find more detailed instructions and explanations on this blog post. With some file systems, fallocate may not work – again, take a look at  this for a work around.

fallocate -l 512M /.swapfile
chmod 600 /.swapfile
mkswap /.swapfile
swapon /.swapfile
echo "/.swapfile none swap defaults 0 0" >> /etc/fstab

How To Fix A Full /boot Partition on Linux

Get Social!

full-boot-mountUtilities such as apt-get generally install kernel updates by adding the new kernel to the Linux boot list and set it as the default. This means that the next time the system boots, the new kernel will be loaded. The problem is that the old kernel is still there (just incase the new one doesn’t work, you’ve got a fall back!), and the kernel before that, and the kernel before that…

The current 3.x kernel for Ubuntu is around 20MB so it doesn’t take long for the kernel updates to fill a tiny 200-or-so-Mb boot partition where they are stored.

The good news is that it’s easy to clear out the old updates, but it’s important to not remove the latest one that you’re using.

Which Kernel am I Running?

It’s quick and easy to see which kernel version you are running. Use the uname command with the -a switch and pay attention to the version numbers reported.

uname -a
3.19.0-47-generic

This shows that you’re using version 3.19.0-47.

List Currently Installed Kernels

Next you’ll need to list the kernel packages that are currently installed so that you can remove any outdated ones to free up the space on the /boot partition.

Run the below dpkg command to list the installed kernel packages and their versions.

dpkg --list 'linux-image*'

dpkg-kernel-package-listThe above output shows several versions of kernel that are all taking up space on the /boot partition however we only really need the current running version given by the above uname -a command. It’s a good idea to keep the last 2 kernels just incase you notice an issue down the line, but the above list is quite excessive.

Remove Unused Kernel Packages to Free Space on /boot

Once you’ve identified your current kernel and the kernel packages you have installed it’s time to remove the ones you don’t need.

Using apt-get enter the package names of the kernel packages to remove.

apt-get remove linux-image-3.19.0-25-generic

If you get an error running this command then see the next section.

note: the above screenshot of the installed packages cropped the full version name – it’s missing the generic part. You can use tab completion with the apt-get command, or use an asterisk after the version number to remove the required package. 

Errors Removing Packages

If you get an error warning about dependencies similar to the below then you may need to manually remove a few kernel packages to free up some space.

[root@server:~] $apt-get remove linux-image-extra-3.19.0-25-generic
Reading package lists... Done
Building dependency tree
Reading state information... Done
You might want to run 'apt-get -f install' to correct these:
The following packages have unmet dependencies.
 linux-image-generic-lts-vivid : Depends: linux-image-3.19.0-49-generic but it is not going to be installed
                                 Depends: linux-image-extra-3.19.0-49-generic but it is not going to be installed
                                 Recommends: thermald but it is not going to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).

The problem here is that the original update failed to install the latest version of the kernel (version 3.19.0-49-generic in this case) and apt-get doesn’t like doing anything else until that problem has been resolved.

First we need to free up some space. Carefully delete 2 of the older kernel packages with a command similar to the below, but with old version numbers from your system returned by the above section List Currently Installed Kernels.

rm /boot/*-3.19.0-25*
rm /boot/*-3.19.0-33*

Check and double check this command because there’s no going back once it’s ran! It’s perfectly safe to do as long as you are using old version numbers that you’re no longer using.

You can now run apt-get to complete the original upgrade now that it has the space. This will remove the error when trying to remove the unused kernel packages in the above section.

apt-get install -f

Now go back to the above section and remove the old kernel packages that you no longer need.


dd Cheat Sheet

Get Social!

dd is one of the most versatile IO tools available for Linux. It’s used in a variety of ways ranging from Disk Benchmarking through to creating SWAP files and copying downloaded disk images to physical disks.

dd takes the following common switches:

  • if is the input file name and location.
  • of is the name and location of the output file.
  • bs is the block size that will be used to read and/ or write the file. Increasing this can help with performance  or dictate how much data will be read or written.
  • count is the number of blocks that will be used.
  • seek is the number of blocks on the output file that will be skipped before writing any data.
  • skip is the number of blocks that will be skipped on the input file before starting to read data.
  • conv is a comma separated list of additional parameters that can be used. See the man dd for more information.

The below headings will list a few example uses of dd in a typical Linux environment.

Backup disk partition with dd

You can use dd to copy an entire disk partition to a virtual disk file. This can be useful for creating a backup or to clone the disk to another machine.

dd if=/dev/sda1 of=~/localdisk_sda1.img

You can use this method to read a CD-ROM, USB drive or Flash disk to a file in the same way – just make sure the device is inserted and point the if= part of the dd command to the relevant /dev/ device.

You could also compress the image as part of the process with gzip.

dd if=/dev/sda1 | gzip -c > ~/localdisk_sda1.img.gz

Restore disk partition with dd

Similar to the above command, you can use dd to replace a disk’s partition with a virtual disk file.

dd if=~/localdisk_sda1.img of=/dev/sda1

If you compressed the image then you can decompress it first all in one go:

gunzip -c ~/localdisk_sda1.img.gz | dd of=/dev/sda1

Create a fixed size file with dd

You can create a fixed size file with DD that will be created in the location you specify.

dd if=/dev/zero of=/root/test bs=1024 count=1

This will create a file in /root/test of 1024 bytes in size. Increase either bs or count to change the size of the file. The resulting size will be bs count. You can also use shorhand sizes such as K, M and G with bs, for example bs=1G,

dd if=/dev/zero of=upload_test bs=file_size count=1

Create a SWAP file with dd

dd can be used to create a SWAP file that can be used as a SWAP device by your computer. This is often needed with smaller instances on Cloud providers such as AWS.

The starting point is the same as the above command to create a file with the size that you’d like to use for swap. See my other blog post for more info.

Split a file with dd

dd can be used to read just part of a file, given offset and length coordinates. The below example will skip the first 100 bytes of the file and output the proceeding 10 bytes (byte 101 – 111).

dd if=filetosplit of=partfile bs=1 count=10 skip=100

You could repeat this process to split a large file into multiple smaller files, to be able to email it for example.

dd if=filetosplit of=partfile1 bs=1 count=100
dd if=filetosplit of=partfile2 bs=1 count=100 skip=100
dd if=filetosplit of=partfile3 bs=1 count=100 skip=200

Merge multiple files with dd

You can merge multiple files into a single file with dd. Following on from the above split example, the below will rejoin the 3 file parts into a single file.

dd if=partfile1 of=joinedfile bs=1 count=100
dd if=partfile2 of=joinedfile bs=1 count=100 seek=100
dd if=partfile3 of=joinedfile bs=1 count=100 seek=200

Convert text to lower case with dd

You can use the conv switch with dd to transform ascii text from upper case to lower case and visa-versa. Using lcase and ucase in the conv switch will instruct dd to convert the text as it’s written.

The below example will convert all characters in the filetoconvert.txt. file to lower case.

dd if=filetoconvert.txt of=convertedfile.txt conv=lcase

 


Visit our advertisers

Quick Poll

Which type of virtualisation do you use?
  • Add your answer

Visit our advertisers