Parallel Replication In MariaDB

Parallel Replication In MariaDB

Tags :

Category : How-to

Get Social!

Parallel replication has been available in MariaDB since Version 10.0.5, however requires at least version 10.0.5 on both the Master and Slave for parallel replication to work.

Parallel replication can help speed up applying changes to a MariaDB slave server by applying several changes at once.

What is Parallel Replication?

MariaDB replicates data from a master to a slave by shipping all changes that have been applied to the master to the slave in a serialised file. The file is then read by the slave and each change is applied one at a time. A change may be a single row change, such as an INSERT, a DDL change or statement that is applied in it’s entirety such as INSERT INTO… SELECT. The bottle neck to this process is that the changes which need to be applied are read in serial – that is, one at a time.

Parallel replication tries to overcome this by applying DML statements in parallel by reading ahead in the relay log (the log on the slave with changes waiting to be applied) and giving work to each parallel worker to apply, in parallel! Each parallel worker has a cache that allows it to read ahead in the log and apply statements that can be applied in parallel – these are usually statements applied in a single transaction, or statements that have been committed in the same group.

The above diagrams show the differences between the different replication mode. Up to the SQL Thread things work in much the same way, however in parallel replication mode the SQL Thread behaves differently in that it moves work to the Worker threads rather than applying it directly itself.

Enable Parallel Replication

You will need MariaDB 10.0.5 or later running on both the master and the slave for parallel replication to be available.

Edit your MariaDB config file, my.cnf on some installations and edit or add the following parameter. 

slave-parallel-threads=12

This will enable 12 parallel workers on the database Slave which will be started when your slave server is next restarted and replication is enabled.

You can see if the required number of workers has been started by running show processlist which will show 12 processes running as system user with various State information.

You can see further information by running show slave status which will show you the replication type, how up to date the replication is and if there are any errors.

Run Multiple Bash Commands In Parallel

Tags :

Category : How-to

Get Social!

Bash, whilst great for simple things, can be tricky to use more advanced programming techniques that are easily exposed in things like Java, or Go.

Multithreading is one such problem. I often find myself with a series of tasks to perform that I’d like to run in parallel up to a predefined concurrency threshold.

My recent task which I’ll use as an example was to run multiple curl commands against an endpoint. These commands were standalone in the fact that they could be executed in any order and would benefit from running several API calls at once.

The first step is to create your list of commands in a file. For this, I’ll use the echo and sleep commands to demonstrate.

vi /tmp/myCommands

echo 1 && sleep 2
echo 2 && sleep 2
echo 3 && sleep 2
echo 4 && sleep 2
echo 5 && sleep 2
echo 6 && sleep 2
echo 7 && sleep 2
echo 8 && sleep 2
echo 9 && sleep 2
echo 0 && sleep 2

Once you have your list of commands, it’s time to run them!

cat /tmp/myCommands | while read n; do printf "%q\n" "$n"; done | xargs --max-procs=2 -I LC bash -c LC

The first command cat /tmp/myCommands is simply the path to your list of commands to run. The only other part to worry about is the —max-proxcs=2 attribute of xargs – this is what defines the concurrency and therefore how many ‘threads’ will run at once. xargs will do the rest – each command in your source file will be executed with 2 running at once!

So there you have it – threaded command execution in Bash!

Visit our advertisers

Search

Quick Poll

What type of VPN protocol do you use?
Results

Recent Posts

Visit our advertisers

Site Links
Blog
Rated Sites

JamesCoyle.net Limited