Windows Task Manager – what do the memory columns mean?
Category : Knowledge
I have recently been using Windows to tune a Java application I created to move large amounts of data into a database. The application was very memory hungry as it implemented various queues to buffer incoming data before committing it to the data target.
This meant that I needed to understand the memory which was being used at any given point, both in the JVM and on the operating system itself. On the Windows side, the Task Manager has many extra columns which can be added to the Processes tab to see exactly what each process is doing, and how much resource it is costing to do it.
The CPU and IO columns should be fairly straight forward however the memory columns can be most confusing. Before we can get into it there are a few terms which you must understand
- Physical memory – this is RAM, physical memory which you have installed in your machine. This is the fastest type of memory and also the most expensive. Physical memory will be used for all your processes memory requirements unless you run out or the operating systems decides to free some for other things.
- Virtual memory – when you run out of physical memory and another process asks for more memory the operating system needs a mechanism to ensure the requirements can be met. Virtual memory is used much the same as RAM however it is stored on a disk. Disks, even SSDs, are very slow compared to RAM and come with a huge performance impact. Your operating system will move RAM to virtual memory when a process hasn’t been used for a while, or if sacrifices need to be made to free some RAM for something else.
There are many memory related columns in Windows Task Manager which all have subtly different meanings. Here are some of the most common and useful memory column definitions:
- Memory – Working Set – is the total amount of memory used by the process which is private and unique to the process as well as shared memory used by the process such as shared libraries.
- Memory – Private Working Set – is the amount of memory used by the process which is unique to the process and cannot be accessed by other processes.
- Memory – Peak Working Set – is the most memory, both unique and shared, which has been used by the process since it started. This is reset each time the process ends.
- Memory – Commit Size – is the amount of virtual memory which is reserved, or committed, for the process.
- Page Faults – is the amount of times memory has been fetched from virtual memory because it was not found in physical memory. This is a counter and similar to Memory – Peak Working Set in that it is reset each time the process ends.
By understanding an monitoring these values you can get an understanding of what a processes is doing with it’s memory.
Over the past few days I have been looking at alternative virtualisation servers to Proxmox VE, something which I have been using for the past few years. This has been prompted by Proxmox’ recent move to subscription based stable repositories and free test repositories. What this means is that unless I pay a subscription, I am no longer allowed to use the stable Proxmox binaries through apt-get and have to use potentially unsafe ‘test’ binaries. Of cause, I could build the latest stable binaries directly from git but then there is another problem – I don’t know what commit is marked as ‘stable’ to use as my build source.
I have been having trouble with a Proxmox node which is only running OpenVZ containers however it is at the upper limit of its RAM resources. Over time, I noticed that Proxmox used SWAP (virtual memory, page file, etc), quite aggressively, to make sure there was always some RAM free. That sounds fantastic, and is just what I would expect the Proxmox server to be doing, except it does it all too well. Proxmox made sure that around 40% of the RAM on the host machine was free at the expense of moving many running processes across all the running containers to SWAP. This is how Linux works, by design, and is expected behaviour. Running processes which have memory which hasn’t been touched in a while have their memory moved to SWAP. This allows other applications which need the memory right now to use it and anything left over can be used as cache by the kernel. When a process with memory in SWAP needs to use that memory, it needs to be read from SWAP and back into memory so that it can be used. There is a huge overhead with this process and will often be noticed when you use a container which has not been used in a while – at the start everything will be slow until all the required memory items have been read from SWAP and put back into RAM. To help with this situation we can do two things:
I have been using GlusterFS to replicate storage between two physical servers for two reasons; load balancing and data redundancy. I use this on top of a ZFS storage array as described in 
Proxmox 3.1 has implemented a new repository setup, as described in my