Category Archives: How-to

Install AWS CodeDeploy Agent on Linux

Category : How-to

Get Social!

AWS CodeDeploy Agent is the agent that runs deploy jobs on EC2 instances. Before a CodeDeploy job will run you’ll need to make sure the agent is installed, running, and has the correct IAM permissions to execute.

Installation is straight forward on Linux and will have your agent up and running in no time.

The below example is based on Ubuntu, but the same steps would be used on other distributions, with the exception of the package manager for installing ruby.

As the root user, run the below commands. root is required because the deployment could be performing actions that require elevated privileges. Ruby is an installation dependency for AWS CodeDeploy and must be available before installing the agent itself.

apt update
apt -y install ruby

Once Ruby is installed we can download and install the CodeDeploy agent. The below agent is being downloaded from the eu-central-1 region, but you can replace the region with your local region if required. Other than saving bandwidth charges for the download (which will be tiny) there is no real reason to do so.

cd /tmp
wget https://aws-codedeploy-eu-central-1.s3.amazonaws.com/latest/install;
chmod +x ./install

./install auto

The final step is to start the agent and check that it’s running. A systemd entry will be added and needs to be called to start the agent.

service codedeploy-agent start

Finally, check that the agent is running by checking the log. You should be looking for a similar output to the below.

tail -f /var/log/aws/codedeploy-agent/codedeploy-agent.log

2019-11-24 07:30:54 INFO  [codedeploy-agent(31022)]: master 31017: Spawned child 1/1
2019-11-24 07:30:54 INFO  [codedeploy-agent(31022)]: On Premises config file does not exist or not readable
2019-11-24 07:30:54 INFO  [codedeploy-agent(31022)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandExecutor: Archives to retain is: 5}
2019-11-24 07:30:54 INFO  [codedeploy-agent(31022)]: Version file found in /opt/codedeploy-agent/.version with agent version OFFICIAL_1.0-1.1597_deb.
2019-11-24 07:30:54 INFO  [codedeploy-agent(31017)]: Started master 31017 with 1 children
2019-11-24 07:31:54 INFO  [codedeploy-agent(31022)]: [Aws::CodeDeployCommand::Client 200 61.547075 0 retries] poll_host_command(host_identifier:"xxxx")

See here for the installation steps combined into a single script.


Script To Install AWS CodeDeploy Agent on Linux

Get Social!

Here is the script for installing the AWS CodeDeploy agent on Ubuntu. See the full description here.

apt update
apt -y install ruby

cd /tmp
wget https://aws-codedeploy-eu-central-1.s3.amazonaws.com/latest/install;
chmod +x ./install

./install auto

service codedeploy-agent start

Storj Storage Node Docker-Compose file

Get Social!

Storj V3 is now in BETA and recruiting Storage Node operators. Since V3 of Storj, Docker is used exclusively to wrap up creating a new Storage Node into a simple, manageable container.

You’ll need to see the official docs for creating your identity certificates, but when it comes to creating your docker environment it couldn’t be more simple than using docker-compose. If you haven’t got docker-compose installed then check out this blog post.

Create a new folder and a docker-compose.yml with the below content.

mkdir storj
vi storj/docker-compose.yml
version: '3'
services:
  storagenode:
    image: storjlabs/storagenode:beta
    restart: unless-stopped
    ports:
        - 28967:28967
    volumes:
        - ./config/identity:/app/identity
        - ./data:/app/config
    environment:
        - WALLET=0x123456789
        - EMAIL=EMAIL
        - ADDRESS=external.url:28967
        - BANDWIDTH=10TB
        - STORAGE=1TB
        - STORJ_LOG_LEVEL=info

You’ll need to fill out the environment details to match your requirements, especially the WALLET and ADDRESS. You may want to redirect the volume elements to match your environment – the /app/config path should point to the disk that you’d like to use for storage (I know, the name is confusing) and the /app/identity path should point to your Storj identity certificates.

Run docker-compose up -d to fetch the images from the docker hub and create your Storage Node instance.


Apache Traffic Server (ATS) Returning 403 For DELETE HTTP Requests

Category : How-to

Get Social!

Here is a quick snippet which solves an issue I ran into today. I’ve recently set up Apache Traffic Server to reverse proxy requests to various Docker containers. It all works great and runs itself in Docker.

One thing, however, with a default install of Apache Traffic Server is that it doesn’t allow DELETE HTTP requests from any source other than localhost. Instead, the 403 Forbidden status code is returned which can cause some curious side effects for front end web applications.

The fix is simple enough, when you know where to look. ATS has a config file called ip_allow.config that controls, believe it or not, which http methods are allowed for different source IP addresses. The default file looks like this:

#
# ip_allow.config
#
# Documentation:
#    https://docs.trafficserver.apache.org/en/latest/admin-guide/files/ip_allow.config.en.html
#
# Rules:
# src_ip=<range of IP addresses> action=<action> [method=<list of methods separated by '|'>]
#
# Actions: ip_allow, ip_deny
#
# Multiple method keywords can be specified (method=GET method=HEAD), or
# multiple methods can be separated by an '|' (method=GET|HEAD).  The method
# keyword is optional and it is defaulted to ALL.
# Available methods: ALL, GET, CONNECT, DELETE, HEAD, OPTIONS,
# POST, PURGE, PUT, TRACE, PUSH
#
# Rules are applied in the order listed starting from the top.
# That means you generally want to append your rules after the ones listed here.
#
# Allow anything on localhost (this is the default configuration based on the
# deprecated CONFIG proxy.config.http.quick_filter.mask INT 0x482)
src_ip=127.0.0.1                                  action=ip_allow method=ALL
src_ip=::1                                        action=ip_allow method=ALL
# Deny PURGE, DELETE, and PUSH for all (this implies allow other methods for all)
src_ip=0.0.0.0-255.255.255.255                    action=ip_deny  method=PUSH|PURGE|DELETE
src_ip=::-ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff action=ip_deny  method=PUSH|PURGE|DELETE

Take a look at the bottom few lines. They state that PUSH, PURGE and DELETE should all be denied to all IP ranges.

To enable the DELETE http method from all IPs, simply remove the DELETE method from the bottom 2 lines. You should be left with something looking like this:

src_ip=127.0.0.1                                  action=ip_allow method=ALL
src_ip=::1                                        action=ip_allow method=ALL
# Deny PURGE, DELETE, and PUSH for all (this implies allow other methods for all)
src_ip=0.0.0.0-255.255.255.255                    action=ip_deny  method=PUSH|PURGE
src_ip=::-ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff action=ip_deny  method=PUSH|PURGE

It’s a curious default to have, but it could stop destructive API calls being made if endpoints were accidentally made public.


Mysql Data Types and Sizes for String/ Text

Category : How-to

Get Social!

MySQL or MariaDB has several data types for handling text-based characters. There are several data types for handling smaller strings such as CHAR and VARCHAR data types. For larger text-based strings there are BLOB based data types such as TEXT.

It’s worth noting at this point that the below-quoted sizes do not necessarily represent the number of characters they can hold. In addition, more recent versions of MySQL (version 5 and 8) counts characters when defining the length, however, prior to these versions byres were used.

The below table shows the ‘size’ of each data type – notice that some data types are mentioned in characters, and others in bytes. The number of characters are always used when defining a string data type in your DDL statement – for example, VARCHAR(10).

Data TypeSizeDescription
CHAR(n)255 charactersFixed-length character field. Rows are padded with whitespace to the defined length.
VARCHAR(n)65,535 bytes *Variable-length character field with no manipulation on INSERT or SELECT.
TINYTEXT255 bytesVariable-length and stored off-row. Can only be sorted and grouped by up to max_sort_length characters
TEXT(n)65,535 bytes Variable-length and stored off-row. Can only be sorted and grouped by up to max_sort_length characters
MEDIUMTEXT(n)16,777,215 bytes (16MB)Variable-length and stored off-row. Can only be sorted and grouped by up to max_sort_length characters
LONGTEXT(n)4,294,967,295 bytes (4GB)Variable-length and stored off-row. Can only be sorted and grouped by up to max_sort_length characters
ENUM2 bytesThe ENUM doesn’t need a size definition but can hold up to 65,535 values.

* The maximum row length in MySQL is 65,535 bytes – your total row size cannot exceed this byte value. Keep in mind that some character sets consume more than one byte per character. For example, utf8mb4 can take up to 4 bytes per character and therefore the maximum VARCHAR is approximately one-quarter of the maximum row size.


Docker Compose yml for Gitlab and Gitlab Runner

Get Social!
version: '3.5'
services:
  gitlab:
    image: gitlab/gitlab-ce:latest
    hostname: www.jamescoyle.net
    restart: unless-stopped
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        gitlab_rails['gitlab_shell_ssh_port'] = 8822
    ports:
      - "8000:80"
      - "8822:22"
    volumes:
      - ./config/gitlab:/etc/gitlab
      - ./data/gitlab:/var/opt/gitlab
      - ./logs:/var/log/gitlab
    networks:
      - gitlab

  gitlab-runner:
    image: gitlab/gitlab-runner:alpine
    restart: unless-stopped
    depends_on:
      - gitlab
    volumes:
      - ./config/gitlab-runner:/etc/gitlab-runner
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - gitlab

networks:
  gitlab:

Create a new directory and save the above file inside it as docker-compose.yml. You’ll need to replace the field hostname with the external URL that you’ll use to access your Gitlab instance.

mkdir gitlab
vi gitlab\docker-compose.yml

Run docker-compose up -d to fetch the images from the docker hub and create your Gitlab instance. You’ll be able to access Gitlab from a browser on port 8000 and the SSH on port 8822.


Visit our advertisers

Quick Poll

Do you use GlusterFS in your workplace?

Visit our advertisers