Author Archives: James Coyle

Speed up Multiple apt-get install Requests by Caching the Repository

Tags :

Category : How-to

Get Social!

Linux penguinapt-get is the tool used in Debian and Ubuntu to manage packages installed on the system. Each time an update is available, or you install a new package the files will be downloaded from one of the central repository servers out on the internet and installed on your system.

There are two main problems with this:

  • Your servers may not be on able to access the internet directly for security reasons
  • Installing the same package on multiple servers will result in downloading the package the same amount of times. This could be slow or expensive in terms of bandwidth.

To solve the problem you can mirror the source repository on your own local server and add that as a source for your servers to update from. The main issue with this is that each distribution has a huge catalog of package which would take up vast amounts of space. Multiply this by the different releases of operating system in your environment and you could be talking terabytes of space.

Various utilities have been created to work round this problem such as apt-proxy, apt-cacher and debproxy. These utilities work by only caching some of the most used packages and fetching the rest from the source.

The below example will use apt-cacher-ng which is a middle man who sits in between the server being updated and the source repository out on the internet. It chooses to cache some regularly or recently used files locally and will recall them when they are requested which greatly speeds up the process for the requesting machine. The cache is frequently cleared to make sure that disk space is only being used for the most necessary packages. This drastically reduces resource required to run the service, whilst speeding up package downloading and guaranteeing that all packages are available.

Setting up apt-cacher-ng server

The apt-cacher-ng utility sits on a server which must be able to access both the public network and any internal network which your other servers may sit on.

Run apt-get install to install the proxy utility.

apt-get install apt-cacher-ng

The default installation of apt-cacher-ng holds details of both Ubuntu and Debian source repositories and is ready to use.

If you need to change the settings of the application such as the port it listens on, edit the below file:

vi /etc/apt-cacher-ng/acng.conf

You can now access the web interface using using the local machine’s IP or host name and the port. The default port is 3142.

apt-cache-nr-homescreen

This page shows that apt-cacher-ng is working correctly and is ready to cache the first source requests.

The next step is to add the server location to your clients. Create the below file and add details of your caching server.

/etc/apt/apt.conf.d/02proxy

Add the below line and edit [SERVER_IP] and [SERVER_PORT] to match your apt-cacher-ng configuration.

Acquire::http { proxy "http://[SERVER_IP]:[SERVER_PORT]"; };

Eg.

Acquire::http { proxy "http://10.10.10.1:3142"; };

Finally, run the update command on your clients to cause the proxy to cache the package lists. Packages will also be cached soon as you start to install or updates packages on your client.

To make sure that apt-cacher-ng is doing it’s job, tail the log to make sure entries are appearing.

tail -f /var/log/apt-cacher-ng/apt-cacher.log

In addition, you can also view the webpage for statistics on cache hits and misses:

http://[SERVER_IP]:[SERVER_PORT]/acng-report.html?doCount=Count+Data#top

doskey in Windows is just like alias in Linux

Tags :

Category : How-to

Get Social!

doskey on Windows is very similar to alias on Linux, it allows you to set a term which will call a command and allow you to specify default arguments. You can specify that the command showdirs could call the command dir to list the content of a folder.

You may have guessed, the above use of doskey is unlikely and has no real benefit. A better use of the command would be to turn some of the windows commands into their Linux counterparts – such as ls into dir.

Run the below command in a command prompt to alias ls to run the command dir. The $* on the end are required so that any additional arguments are also passed to the dir command.

doskey ls=dir $*

The problem with this is that all of your alias commands will be lost when you close the cmd session. To make them persist we need to create a batch file and add the entry to the windows registry.

Create a new folder in the windows directory called bin and create a new batch file inside it.

C:\>mkdir c:\windows\bin
C:\>notepad.exe c:\windows\bin\doskey.bat

Add your entries to the batch file in the below format.

@echo off
doskey ls=dir $*
doskey mv=move $*
doskey cp=copy $*
doskey cat=type $*

Next, open up regedit.exe and add an entry to the batch file to make the doskey commands permanent for each cmd session.

HKEY_CURRENT_USER\Software\Microsoft\Command Processor

Add a new String Value called AutoRun and set the absolute path in the value of c:\windows\bin\doskey.bat.

The doskey.bat file will now be executed before opening a new cmd session which will set all of your alias ready for you to use.

 

Ignoring Files and Directories in Git with .gitignore

Tags :

Category : How-to

Get Social!

octocat-githubWith Git you are able to define file exceptions to exclude certain files and folders from git repository commits. You can create files which contain a list of patterns which git will check against on each git add and ignore any matching files.

You can create ignore pattern lists to ignore files on either a global scale which will affect all repositories on the system or limit it to a specific repository.

Both types of ignore use a .gitignore file which contains literal paths of files inside the repository or patterns which will be used to exclude matching files and directories.

You can skip to the bottom of the post for a few common examples.

.gitignore patterns

Patterns inside the .gitignore file are matched from the root directory of the git repository. Patterns are comprised of a wildcard character *, to match any character, and literal characters to match the exact phrase.

A typical example of using a .gitignore file would be to exclude all files ending in .log. The below pattern would be added to the .gitignore file

*.log

Or, as with something like log4j, your log files may include numbers at the end. This pattern will exclude any file names that contain .log.

*.log*

Another use is to exclude all files in a specific path, such as the application build directory. This will ignore the Build directory and everything within it.

/Build/*

A double asterisk (**) has its own special meaning and represents matching in all directories. For example, a/*/c would only match a single folder between a and b – a/this/b would match but /a/this/and/this/b would not match. Using a double asterisk would match in both scenarios. 

/src/**/tmp # exclude any /tmp files or folders at any level in the /src/ folder.

Single repository .gitignore

Add your patterns to the below file to add exclusions to affect only a singe git repository. You must make sure you have changed to the root directory of your repository, or include it in the file path.

vi /path/to/repository/.git/info/exclude

Global .gitignore

You must run a git config command to enable .gitignore to work across all local repositories. You can edit the ~/.gitignore path if required.

git config --global core.excludesfile ~/.gitignore

Once enabled, edit the ~/.gitignore file and add patterns which will affect the next git add command.

vi ~/.gitignore

For example, you may add a global gitignore entry for .bak files. Add the following line to you global gitignore file:

*.bak

You can use just one of the above methods or a combination of both gitignore methods on your git client.

Common .gitignore examples

Copy MySQL Database to Another Server

Tags :

Category : How-to

Get Social!

mysql-logoThe easiest way to do a one off move of a MySQL database from one server to another is to use the mysqldump utility. This utility will dump the entire database and it’s content to a file and which can then be imported it into the target database.

Before running the export, you should make sure you have enough free space to hold the database. The bigger the database, the more free space you will need for the SQL dump.

Run the below command to export your database from the source server. Replace [EXPORT_DATABASE] with the name of the database to export.

mysqldump -uroot -p [EXPORT_DATABASE] -r /location/to/export/to.sql

If required, move the SQL dump file to the target server. On the target server, run the below command to connect, create the database and import the data. Change [USERNAME] to the user you would like to connect as and [DATABASE_NAME] to the name of the databse where the dump file will be imported. The [DATABASE_NAME] will most likely be the same as on the source database however you are able to change it if required.

mysql -p -u [USERNAME] [DATABASE_NAME] < /location/to/export/to.sql

OpenSSL Certificate Cheat Sheet

Tags :

Category : Cheat Sheets

Get Social!

openssl-logoThese commands cover the basics of OpenSSL and are valid for either Windows or Linux with the exception that paths may need to be corrected for the respective platform.

Install OpenSSL

For windows http://www.openssl.org/related/binaries.html

For Ubuntu

sudo apt-get install openssl

Create Private Key

The last argument in the below line is the key length. This can be changed to 2048 or 4096 if required for better encryption.

openssl genrsa -des3 -out server.key 1024

Generate a CSR (Certificate Signing Request)

You will be asked for the details of the certificate such as domain name and address when running this command.

openssl req -new -key server.key -out server.csr

Remove Passphrase from Key

Some applications do not allow for the private key to have a passphrase. The below commands will remove the passphrase – be careful as it will mean the key is no longer protected and can be viewed by anyone with read access to the file.

openssl rsa -in server-with-passphrase.key -out server.key

Generating a Self-Signed Certificate

Once you have generated a key and CSR you will need to sign the request and generate the public certificate. If you do not have a certificate authority you can sign the certificate yourself. The below will generate a certificate which is valid for one year.

openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

Convert x509 to pem

openssl x509 -inform der -in server.crt -out server.pem

pkcs12 to pem – key only

Use the below command to extract only the key from a pkcs12 certificate.

openssl pkcs12 -nocerts -in c:\server.pfx -out c:\server-key.key

pkcs12 to pem – certificate only

Use the below command to extract only the public certificate from a pkcs12 certificate.

openssl pkcs12 -nokeys -in server.pfx -out server-cert.cer

Check a private key

You can check a private key with the below command.

openssl rsa -in privateKey.key -check

Check a certificate

Use the below command to check a certificate.

openssl x509 -in certificate.crt -text -noout

 

Bash Script to Create an SSL Certificate Key and Request (CSR)

Tags :

Category : How-to

Get Social!

padlockCreating multiple SSL certificates for web servers and application can be a repetitive task. Generally speaking, when creating these things manually you would follow the below steps:

  • Create a certificate key.
  • Create the certificate signing request (CSR) which contains details such as the domain name and address details.
  • Sign the certificate
  • Install the certificate and key in the application.

If nothing else, typing out the address and organisation for every certificate can be laborious.

The below script allows you to hard code many of the details to avoid the repetition and only specify the domain name as an argument. The script is dependent on openssl which can be installed using your distributions package manger or from their website. Use apt-get on Debian/ Ubuntu:

apt-get install openssl

Once you have openssl installed, copy the below script to a file called gen-cer.

vi gen-cer
#!/bin/bash

#Required
domain=$1
commonname=$domain

#Change to your company details
country=GB
state=Nottingham
locality=Nottinghamshire
organization=Jamescoyle.net
organizationalunit=IT
[email protected]

#Optional
password=dummypassword

if [ -z "$domain" ]
then
    echo "Argument not present."
    echo "Useage $0 [common name]"

    exit 99
fi

echo "Generating key request for $domain"

#Generate a key
openssl genrsa -des3 -passout pass:$password -out $domain.key 2048 -noout

#Remove passphrase from the key. Comment the line out to keep the passphrase
echo "Removing passphrase from key"
openssl rsa -in $domain.key -passin pass:$password -out $domain.key

#Create the request
echo "Creating CSR"
openssl req -new -key $domain.key -out $domain.csr -passin pass:$password \
    -subj "/C=$country/ST=$state/L=$locality/O=$organization/OU=$organizationalunit/CN=$commonname/emailAddress=$email"

echo "---------------------------"
echo "-----Below is your CSR-----"
echo "---------------------------"
echo
cat $domain.csr

echo
echo "---------------------------"
echo "-----Below is your Key-----"
echo "---------------------------"
echo
cat $domain.key

Make sure your script has execute permissions.

chmod +x gen-cer

You can then call the script with ./gen-cer and specify your domain name as an argument. For example:

./gen-cer mynewwebserver.jamescoyle.net

The script will then output the key as well as the CSR which you will need to submit to your certificate authority (CA).

« Previous Page — Next Page »

Visit our advertisers

Search

Quick Poll

Do you use ZFS on Linux?
Results

Recent Posts

Visit our advertisers

Site Links
Blog
Rated Sites

JamesCoyle.net Limited