Author Archives: James Coyle

Storj Storage Node Docker-Compose file

Get Social!

Storj V3 is now in BETA and recruiting Storage Node operators. Since V3 of Storj, Docker is used exclusively to wrap up creating a new Storage Node into a simple, manageable container.

You’ll need to see the official docs for creating your identity certificates, but when it comes to creating your docker environment it couldn’t be more simple than using docker-compose. If you haven’t got docker-compose installed then check out this blog post.

Create a new folder and a docker-compose.yml with the below content.

mkdir storj
vi storj/docker-compose.yml
version: '3'
services:
  storagenode:
    image: storjlabs/storagenode:beta
    restart: unless-stopped
    ports:
        - 28967:28967
    volumes:
        - ./config/identity:/app/identity
        - ./data:/app/config
    environment:
        - WALLET=0x123456789
        - EMAIL=EMAIL
        - ADDRESS=external.url:28967
        - BANDWIDTH=10TB
        - STORAGE=1TB
        - STORJ_LOG_LEVEL=info

You’ll need to fill out the environment details to match your requirements, especially the WALLET and ADDRESS. You may want to redirect the volume elements to match your environment – the /app/config path should point to the disk that you’d like to use for storage (I know, the name is confusing) and the /app/identity path should point to your Storj identity certificates.

Run docker-compose up -d to fetch the images from the docker hub and create your Storage Node instance.


Apache Traffic Server (ATS) Returning 403 For DELETE HTTP Requests

Category : How-to

Get Social!

Here is a quick snippet which solves an issue I ran into today. I’ve recently set up Apache Traffic Server to reverse proxy requests to various Docker containers. It all works great and runs itself in Docker.

One thing, however, with a default install of Apache Traffic Server is that it doesn’t allow DELETE HTTP requests from any source other than localhost. Instead, the 403 Forbidden status code is returned which can cause some curious side effects for front end web applications.

The fix is simple enough, when you know where to look. ATS has a config file called ip_allow.config that controls, believe it or not, which http methods are allowed for different source IP addresses. The default file looks like this:

#
# ip_allow.config
#
# Documentation:
#    https://docs.trafficserver.apache.org/en/latest/admin-guide/files/ip_allow.config.en.html
#
# Rules:
# src_ip=<range of IP addresses> action=<action> [method=<list of methods separated by '|'>]
#
# Actions: ip_allow, ip_deny
#
# Multiple method keywords can be specified (method=GET method=HEAD), or
# multiple methods can be separated by an '|' (method=GET|HEAD).  The method
# keyword is optional and it is defaulted to ALL.
# Available methods: ALL, GET, CONNECT, DELETE, HEAD, OPTIONS,
# POST, PURGE, PUT, TRACE, PUSH
#
# Rules are applied in the order listed starting from the top.
# That means you generally want to append your rules after the ones listed here.
#
# Allow anything on localhost (this is the default configuration based on the
# deprecated CONFIG proxy.config.http.quick_filter.mask INT 0x482)
src_ip=127.0.0.1                                  action=ip_allow method=ALL
src_ip=::1                                        action=ip_allow method=ALL
# Deny PURGE, DELETE, and PUSH for all (this implies allow other methods for all)
src_ip=0.0.0.0-255.255.255.255                    action=ip_deny  method=PUSH|PURGE|DELETE
src_ip=::-ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff action=ip_deny  method=PUSH|PURGE|DELETE

Take a look at the bottom few lines. They state that PUSH, PURGE and DELETE should all be denied to all IP ranges.

To enable the DELETE http method from all IPs, simply remove the DELETE method from the bottom 2 lines. You should be left with something looking like this:

src_ip=127.0.0.1                                  action=ip_allow method=ALL
src_ip=::1                                        action=ip_allow method=ALL
# Deny PURGE, DELETE, and PUSH for all (this implies allow other methods for all)
src_ip=0.0.0.0-255.255.255.255                    action=ip_deny  method=PUSH|PURGE
src_ip=::-ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff action=ip_deny  method=PUSH|PURGE

It’s a curious default to have, but it could stop destructive API calls being made if endpoints were accidentally made public.


Mysql Data Types and Sizes for String/ Text

Category : How-to

Get Social!

MySQL or MariaDB has several data types for handling text-based characters. There are several data types for handling smaller strings such as CHAR and VARCHAR data types. For larger text-based strings there are BLOB based data types such as TEXT.

It’s worth noting at this point that the below-quoted sizes do not necessarily represent the number of characters they can hold. In addition, more recent versions of MySQL (version 5 and 8) counts characters when defining the length, however, prior to these versions byres were used.

The below table shows the ‘size’ of each data type – notice that some data types are mentioned in characters, and others in bytes. The number of characters are always used when defining a string data type in your DDL statement – for example, VARCHAR(10).

Data TypeSizeDescription
CHAR(n)255 charactersFixed-length character field. Rows are padded with whitespace to the defined length.
VARCHAR(n)65,535 bytes *Variable-length character field with no manipulation on INSERT or SELECT.
TINYTEXT255 bytesVariable-length and stored off-row. Can only be sorted and grouped by up to max_sort_length characters
TEXT(n)65,535 bytes Variable-length and stored off-row. Can only be sorted and grouped by up to max_sort_length characters
MEDIUMTEXT(n)16,777,215 bytes (16MB)Variable-length and stored off-row. Can only be sorted and grouped by up to max_sort_length characters
LONGTEXT(n)4,294,967,295 bytes (4GB)Variable-length and stored off-row. Can only be sorted and grouped by up to max_sort_length characters
ENUM2 bytesThe ENUM doesn’t need a size definition but can hold up to 65,535 values.

* The maximum row length in MySQL is 65,535 bytes – your total row size cannot exceed this byte value. Keep in mind that some character sets consume more than one byte per character. For example, utf8mb4 can take up to 4 bytes per character and therefore the maximum VARCHAR is approximately one-quarter of the maximum row size.


Docker Compose yml for Gitlab and Gitlab Runner

Get Social!
version: '3.5'
services:
  gitlab:
    image: gitlab/gitlab-ce:latest
    hostname: www.jamescoyle.net
    restart: unless-stopped
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        gitlab_rails['gitlab_shell_ssh_port'] = 8822
    ports:
      - "8000:80"
      - "8822:8822"
    volumes:
      - ./config/gitlab:/etc/gitlab
      - ./data/gitlab:/var/opt/gitlab
      - ./logs:/var/log/gitlab
    networks:
      - gitlab

  gitlab-runner:
    image: gitlab/gitlab-runner:alpine
    restart: unless-stopped
    depends_on:
      - gitlab
    volumes:
      - ./config/gitlab-runner:/etc/gitlab-runner
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - gitlab

networks:
  gitlab:

Create a new directory and save the above file inside it as docker-compose.yml. You’ll need to replace the field hostname with the external URL that you’ll use to access your Gitlab instance.

mkdir gitlab
vi gitlab\docker-compose.yml

Run docker-compose up -d to fetch the images from the docker hub and create your Gitlab instance. You’ll be able to access Gitlab from a browser on port 8000 and the SSH on port 8822.


Dockerfile for Apache Traffic Server (ATS)

Get Social!

Apache Traffic Server is an enterprise-grade proxy and caching server initially developed by Yahoo, then later made open source and managed by the Apache Foundation.

The below code is a Dockerfile that will download and build ATS on the latest Ubuntu base image. Currently, we’re using Apache Traffic Server version 8.0.5, but if you’d like to use a different version or check for a later version then you’ll need to replace the curl command with one of the downloads available from here.

Create a new folder on your Docker host and add the below text to the dockerfile.

mkdir ats
vi ats/dockerfile
FROM ubuntu:latest
# Update the package repository
RUN set -x \
 && DEBIAN_FRONTEND=noninteractive apt-get update \
 && DEBIAN_FRONTEND=noninteractive apt-get upgrade -y \
 && DEBIAN_FRONTEND=noninteractive apt-get install -y \
        curl \
        locales \
        build-essential \
        bzip2 \
        libssl-dev \
        libxml2 \
        libxml2-dev \
        libpcre3 \
        libpcre3-dev \
        tcl \
        tcl-dev \
        libboost-dev \
    # Configure locale
 && export LANGUAGE=en_US.UTF-8 \
 && export LANG=en_US.UTF-8 \
 && export LC_ALL=en_US.UTF-8 \
 && locale-gen en_US.UTF-8 \
 && DEBIAN_FRONTEND=noninteractive dpkg-reconfigure locales
    # Get ATS and build
    # http://www-eu.apache.org/dist/trafficserver/
RUN  mkdir /tmp/trafficserver \
 && cd /tmp/trafficserver \
 && curl -L http://www-eu.apache.org/dist/trafficserver/trafficserver-8.0.5.tar.bz2 | tar -xj --strip-components 1 \
 && ./configure \
 && make install \
 && make distclean \
 && cd / \
    # Clean-up
 && apt-get purge --auto-remove -y \
        curl \
        build-essential \
        bzip2 \
        libssl-dev \
        libxml2-dev \
        libpcre3-dev \
        tcl-dev \
        libboost-dev \
 && apt-get clean \
 && rm -rf /tmp/* /var/lib/apt/lists/*

RUN ln -s /usr/local/etc/trafficserver /etc/trafficserver
EXPOSE 8080

ENTRYPOINT ["/usr/local/bin/traffic_server"]

To build the Apache Traffic Server image, cd into the ats directory and issue the build command. The period (.) at the end of the build command is there on purpose – make sure you include it in your build command.

cd ats
docker build -t ats .

The build will take a few minutes, depending on your hardware, but will return you to the command line once completed.

Run your dockerfile and ATS will be available on port 8080, however, you’ll need to configure it as required. The config, such as remap.conf, is contained in /etc/trafficserver


Install Docker-Compose

Get Social!

Docker-compose is a utility used to create and manage multiple Docker containers together to form a service. It relies on Docker being installed on the same host so make sure that Docker is installed on your system beforehand. Once you have docker installed and tested you’re ready to begin.

It’s worth noting that docker-compose is often available with your Linux distributions package manager, such as apt or yum but you may find they’re often out of date.

Manually installing docker-compose

Manually installing docker-compose is made easy by the fact that the docker-compose binary is a single file. The process is simple; grab the binary, put it in the right place and make it executable.

In addition to Docker, you’ll need curl installed to download the docker-compose binaries.

apt install curl
# Or
yum install curl

Once you have curl installed it’s time to install docker-compose. The below link will install the latest ‘stable’ release – if you need a different release you’ll need to check the github page for the version number you require.

Run the below commands to download docker-compose and link the binary to your linux bin directory.

curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose

Test docker-compose

Finally, to test your docker-compose is working, run the below command to output the version of docker-compose. At the time of writing, the latest version was 1.24.1, but this will move as time goes on.

docker-compose --version
docker-compose version 1.24.1, build 4667896b


Visit our advertisers

Search

Quick Poll

How many Proxmox servers do you work with?

Visit our advertisers