APR means tough life for garbage collector

X Erlang is designed to be as portable as possible and a few things would have been done differently, probably, improving performance along the way. I decided not to do this and stick to the original rule. When it comes to memory management the rule is this: use memory pools provided by APR (Apache Portable Runtime) only.

If you played with APR than you should be aware that the library uses a specific memory model which makes a lot of sense to the web server environment. Most of the time it prevents memory leaks. Memory is allocated extremely fast from "memory pools". Internally the allocation is just incrementing a a free memory pointer. However, freeing up memory is not that easy. It can be done by destroying the whole memory pool at once.

So far so good. Following the rule X Erlang creates an APR memory pool per process and quickly allocates the values from it. When it comes to garbage collection reachable values are copied to a a new pool and the original pool is destroyed. It means that, yes, X Erlang uses stop-the-world garbage collection and VM is stalled until all garbage is collected for the overgrown process.

There are two points to watch here.

The first is that in many cases variables reference the same locations in memory in X Erlang. If values are immutable there is no reason to copy them around. Thus during garbage collection we should discern the values which were already copied to the new pool. This would have been easy if APR had a way to say that a given pointer was allocated from a given pool. Unfortunately, this can not be done in APR. The approach adopted by X Erlang is to substitute the value in the current pool with a special structure (I call it a 'grave') with a special tag (a 'cross') and a 'buried' pointer to the copy of value in the new pool. The 'grave' structure occupies 4 bytes and the whole scheme works because all referenced values are at least 4 bytes long. In addition, APR by default aligns allocated memory on 8th boundary but this behavior can be changed.

The second is that is not easy to determine the 'size' of a memory pool. There is just no function for this. Technically, there is such a function in a debug version of the library but it determines the size of the pool by scanning it element by element - no use for the high-performance virtual machine :-). Thus X Erlang VM has to add up sizes of all allocations to keep track of the size of the current memory pool. It is ugly but there seems no other way around.

The garbage collection may not recover any memory at all. The size of the new pool may well be larger than the current one. This happens quite often in X Erlang (and OTP too?) and results in immediate termination of the process grown out of proportion.

There is also a lot to be said about handling literal values but it will be a subject matter of another post.