Category Archives: Knowledge

  • 0

What is tmpfs?

Category : Knowledge

Get Social!

Linux penguinA disk drive storage is usually persistent, that is, anything that is written to it will always be there exactly as it was written until it’s deleted or modified by an application. Power failures or computer restarts will not (for the most part) effect the data on the storage disk. You are limited in size by the capacity of the disk and the read and write speed will vary depending on the type of drive you have. Generally storage drives tend to be high in capacity and slow in speed.

Does that make sense? Good. tmpfs is nothing like that. tmpfs, as the name suggests, is intended to be for temporary storage that is very quick to read and write from and does not need to persist across operating system reboots. tmpfs is used in Linux for /run/var/run and /var/lock to provide very fast access for runtime data and lock files. It is also often used for /tmp however it’s not always recommended.

tmpfs uses a combination of computer RAM and disk based SWAP space to create a filesystem, such as EXT4, that the operating system can use. Because tmpfs is located in RAM, it’s very fast to read and write data to and from it, several times faster than an SSD. As your computer runs out of RAM, some of the data in tmpfs will be flushed to the systems SWAP storage on disk. This will dramatically decrease the speed that the tmpfs can be used, but stop your computer from receiving out of memory errors.

See my other blog post on tmpfs vs ramfs for more information on tmpfs and an alternative. Or jump straight to creating your own tmpfs mount point.


  • 2

The Difference Between a tmpfs and ramfs RAM Disk

Get Social!

Linux penguinThere are two file system types built into most modern Linux distributions which allow you to create a RAM based storage area which can be mounted and used link a normal folder.

Before using this type of file system you must understand the benefits and problems of memory file system in general, as well as the two different types. The two types of RAM disk file systems are tmpfs and ramfs and each type has it’s own strengths and weaknesses.

See my other post for details on how to create a RAM disk in Linux.

What is a memory based file system (RAM disk)?

A memory based file system is something which creates a storage area directly in a computers RAM as if it were a partition on a disk drive. As RAM is a volatile type of memory which means when the system is restarted or crashes the file system is lost along with all it’s data.

The major benefit to memory based file systems is that they are very fast – 10s of times faster than modern SSDs. Read and write performance is massively increased for all workload types. These types of fast storage areas are ideally suited for applications which need repetitively small data areas for caching or using as temporary space. As the data is lost when the machine reboots the data must not be  precious as even scheduling backups cannot guarantee that all the data will be replicated in the even of a system crash.

tmpfs vs. ramfs

The two main RAM based file system types in Linux are tmpfs and ramfs. ramfs is the older file system type and is largely replaced in most scenarios by tmpfs.

ramfs

ramfs creates an in memory file system which uses the same mechanism and storage space as Linux file system cache. Running the command free in Linux will show you the amount of RAM you have on your system, including the amount of file system cache in use. The below is an example of a 31GB of ram in a production server.

Currently 8GB of file system cache is in use on the system. This memory is generally used by Linux to cache recently accessed files so that the next time they are requested then can be fetched from RAM very quickly. ramfs uses this same memory and exactly the same mechanism which causes Linux to cache files with the exception that it is not removed when the memory used exceeds threshold set by the system.

ramfs file systems cannot be limited in size like a disk base file system which is limited by it’s capacity. ramfs will continue using memory storage until the system runs out of RAM and likely crashes or becomes unresponsive. This is a problem if the application writing to the file system cannot be limited in total size. Another issue is you cannot see the size of the file system in df and it can only be estimated by looking at the cached entry in free.

tmpfs

tmpfs is a more recent RAM file system which overcomes many of the drawbacks with ramfs. You can specify a size limit in tmpfs which will give a ‘disk full’ error when the limit is reached. This behaviour is exactly the same as a partition of a physical disk.

The size and used amount of space on  a tmpfs partition is also displayed in df. The below example shows an empty 512MB RAM disk.

These two differences between ramfs and tmpfs make tmpfs much more manageable  however this is one major drawback; tmpfs may use SWAP space. If your system runs out of physical RAM, files in your tmpfs partitions may be written to disk based SWAP partitions and will have to be read from disk when the file is next accessed. In some environments this can be seen as a benefit as you are less likely to get out of memory exceptions as you could with ramfs because more ‘memory’ is available to use.

See my other post for details on how to create a RAM disk in Linux.


  • 0

Proxmox Firewall Rules

Category : Knowledge

Get Social!

proxmox logo gradProxmox uses various ports for communication between either a user and the server or between multiple servers in a cluster. You may need to proxy or port forward some of these ports for external access to Proxmox or, in more strict environments, add rules to allow inter machine communication.

For external access to Proxmox you will use either SSH, the web console or SPICE. The following ports are used for each of these services:

  • Web console: 8006 TCP – this is the console you open in your web browser to administer Proxmox. Note: VNC terminals will not work on this port alone.
  • VNC console: 5900 – 5999 TCP – this range of ports is used for the VNC console/ terminal. The reason there is a range is because each open session requires it’s own port meaning that you can have a total of 100 open VNC sessions at once.
  • SPICE: 3128 TCP – this is used if you use SPICE instead of the VNC console. SPICE requires a client to connect.
  • SSH: 22 TCP – this is required for accessing your Proxmox server using SSH.

An additional set of ports are required if you are using Proxmox in a cluster.

  • CMAN: 5404 and 5405 UDP – this is the cluster manager for the Proxmox cluster.

  • 2

Linux Runlevels explained

Tags :

Category : Knowledge

Get Social!

Linux penguinA Linux runlevel dictates the state that the machine is currently operating, and what applications or services should be running at that time. The runlevel is specified as a number between 0 and 6 inclusive.

As your system starts up, it will move up through the runlevels until it reaches it’s desired state. Generally headless installs will be considered fully started at 2 or 3, and desktop GUIs usually at 4 or 5. The following details what each of the default runlevels represent. These are the standard definitions which may be slightly different depending on your Linux distribution.

  • 0 – is described as Halt. your machine will halt when the runlevel is set to 0.
  • 1 – is single-user Mode which is used for administrative tasks before the non-essential services are started.
  • 2 – is the first multi-user mode runlevel and will start some non-essential services. This may, or may not contain networking depending on your Linux distribution.
  • 3 – is a multi-user mode which headless servers usually run at. All essential and non-essential services such as Apache HTTP Server should be running.
  • 4 – is a multi-user mode runlevel which is not used by the default operating system and can be used for user defined purposes.
  • 5 – is when GUI desktops are loaded such as Gnome or KDE.
  • 6 – is the reboot runlevel. the operating system will reboot when runlevel 6 is issued.

You can check your current runlevel with the runlevel command. the below example shows a runlevel of 2.

You can also change the current runlevel manually using the the init command. As an example, your machine will restart if you set the runlevel to 6. Use the following init command to set the runlevel to 6.

Each available runlevel has it’s own directory under /etc staring with rc. Inside the folder is a file, or symlink which controls the respective service. When you add a new service to your operating system startup you are simply adding the services control file to one or more of the runlevel folders.

 


  • 2

Windows Task Manager – what do the memory columns mean?

Get Social!

Screen Shot 2013-10-02 at 18.40.41

I have recently been using Windows to tune a Java application I created to move large amounts of data into a database. The application was very memory hungry as it implemented various queues to buffer incoming data before committing it to the data target.

This meant that I needed to understand the memory which was being used at any given point, both in the JVM and on the operating system itself. On the Windows side, the Task Manager has many extra columns which can be added to the Processes tab to see exactly what each process is doing, and how much resource it is costing to do it.

The CPU and IO columns should be fairly straight forward however the memory columns can be most confusing. Before we can get into it there are a few terms which you must understand

  • Physical memory – this is RAM, physical memory which you have installed in your machine. This is the fastest type of memory and also the most expensive. Physical memory will be used for all your processes memory requirements unless you run out or the operating systems decides to free some for other things.
  • Virtual memory – when you run out of physical memory and another process asks for more memory the operating system needs a mechanism to ensure the requirements can be met. Virtual memory is used much the same as RAM however it is stored on a disk. Disks, even SSDs, are very slow compared to RAM and come with a huge performance impact. Your operating system will move RAM to virtual memory when a process hasn’t been used for a while, or if sacrifices need to be made to free some RAM for something else.

There are many memory related columns in Windows Task Manager which all have subtly different meanings. Here are some of the most common and useful memory column definitions:

  • Memory – Working Set – is the total amount of memory used by the process which is private and unique to the process as well as shared memory used by the process such as shared libraries.
  • Memory – Private Working Set – is the amount of memory used by the process which is unique to the process and cannot be accessed by other processes.
  • Memory – Peak Working Set – is the most memory, both unique and shared, which has been used by the process since it started. This is reset each time the process ends.
  • Memory – Commit Size – is the amount of virtual memory which is reserved, or committed, for the process.
  • Page Faults – is the amount of times memory has been fetched from virtual memory because it was not found in physical memory. This is a counter and similar to Memory – Peak Working Set  in that it is reset each time the process ends.

By understanding an monitoring these values you can get an understanding of what a processes is doing with it’s memory.