Category Archives: How-to

Run Multiple Bash Commands In Parallel

Tags :

Category : How-to

Get Social!

Bash, whilst great for simple things, can be tricky to use more advanced programming techniques that are easily exposed in things like Java, or Go.

Multithreading is one such problem. I often find myself with a series of tasks to perform that I’d like to run in parallel up to a predefined concurrency threshold.

My recent task which I’ll use as an example was to run multiple curl commands against an endpoint. These commands were standalone in the fact that they could be executed in any order and would benefit from running several API calls at once.

The first step is to create your list of commands in a file. For this, I’ll use the echo and sleep commands to demonstrate.

Once you have your list of commands, it’s time to run them!

The first command cat /tmp/myCommands is simply the path to your list of commands to run. The only other part to worry about is the —max-proxcs=2 attribute of xargs – this is what defines the concurrency and therefore how many ‘threads’ will run at once. xargs will do the rest – each command in your source file will be executed with 2 running at once!

So there you have it – threaded command execution in Bash!


Oracle ORA-02287: sequence number not allowed here

Get Social!

I’ve recently hit an issue when trying to include a sequence.nextval in an Oracle database view. the database throws the following error:

I can see why that might not be allowed – each time you select from the view you’ll get a new sequence number making it a bit daft for most circumstances.

I say most because I’ve had a need recently; ETL logic at my current client is held in database views with a boilerplate set of code that wraps around that to perform the table load. That’s fine until you want to fabricate ID’s (SIDs) as part of the load.

Solution

The solution here was to create a utility package to wrap the sequence that could be referenced in the view. See the below example code:

You can now query the view without any pesky ORAs.

See the test script here.


Oracle View Sequence Test

Get Social!

 


Move Proxmox Container to Different Storage (Updated for LXC)

Get Social!

2015-03-05 00_18_04-Proxmox Virtual Environment storageThe Proxmox Web GUI does not give us the ability to migrate a container from one storage device to another directly. To move a container onto different storage we have to take a backup of the container and restore it to the same ID with a different storage device specified. This can be time laborious when working with several containers.

This is an update to the OpenVZ script found here.

The below script allows you to move an LXC container from one storage device to another. The process requires that the container be stopped, which the script will handle.

Save the below script into a file called migrate.

Set execution permissions on the script:

The script has several parameters which are detailed below:

  • -d is specified if you would like the script to delete the temporary backup after the process has completed. Leave this out if you would like the backup tar file to be kept, just in case anything goes wrong.
  • -s is required to specify the name of the target storage. You can find this from the Proxmox Web GUI.
  • -c is required for the container ID to migrate.

In addition, the script contains the variable TMP. This will be the location of the backup tar created as part of the migration process and must contain enough space to store the content of the container being migrated. You can change this to suit your environment.

Example command:

 


Reduce Proxmox LXC Backup Size and Time

Category : How-to

Get Social!

Proxmox backs up guests byte-for-byte in a compressed archive. Looking at LXC backups specifically, the file system is compressed into the target backup file with just a few exceptions – temp files aren’t included. You can also add your own exceptions by editing the vzdump.conf to exclude specific file patterns.

All that said, one of the biggest disk space wasters is the cache directory for apt which caches the installation packages for software you have installed. This can generally be safely removed on internet connected machines which will reduce your overall backup size.

For example, a newly created Debian LXC that’s been recently updated shows a total of 206MB of disk used.

After clearing this with the command apt-get clean we can see the space has mostly been freed.

Considering this whole container is only consuming approximately 1GB of disk space, 200MB is quite significant.

vzdump hooks

Now we can see how much space we can save, we need to make Proxmox issue the apt-get clean command before it creates the backup of our container.

vzdump, the utility which Proxmox uses to perform backups has the ability to call a script for various stages of the backup process – these stages are:

  • backup-start
  • backup-end
  • backup-abort
  • log-end
  • pre-stop
  • pre-restart
  • post-restart

We can use these hooks to run our own commands at any of these points of the backup. For the goal of this blog post, we want to run the apt-get clean command at the point of backup-start.

Create a script on your Proxmox host with the following content:

Now edit your vzdump.conf file and add the following line to point to your new script. Remember to change the location of where your script is – I’ve just saved mine in /root/.

 


Gitlab Runner Error: sudo: no tty present and no askpass program specified

Get Social!

After issuing the first build on a dynamically created Container I came across the following build error when running a command with sudo.

The error is caused by trying to run a command with sudo, however the calling user has not been authorised to use sudo. The error isn’t helpful, and doesn’t really spell out where to go, but adding the calling user to the sudoers file will save the day.

Solution

Open up the sudoers file for editing in your favorite editor.

And add your gitlab runner user to the bottom. If you installed your gitlab runner from the official apt repositories then your gitlab-runner process will run under the gitlab-runner user.

Add the following to the bottom of the file:

Retry your build and you should be back in business!