Limiting LXC Memory Usage

I’ve been playing around with LXC over the past few weeks and one of the things I tried out was limiting the memory that the container is allowed to use. I didn’t plan on explaining all the ins-and-outs of LXC here, but a short description is that LXC provides a virtualizedish environment that is more than a chroot gives you, but less than a full-blown virtual machine. If you want more details, please check out stgraber’s blog post about LXC in 12.04.

Kernel Configuration

The first thing you need to do in order to limit memory usage for LXC is make sure your kernel is properly configured, you need the following flag enabled:


If you plan on also limiting swap space usage, you’ll also need:


These flags are enabled for me in my 12.10 kernel (3.5.0-22) and so presumably you’ll have them in 12.04.

Setting the Cap

First, I’m going to create my container. Following the instructions from stgraber’s blog post, and calling the container “memlimit”:

sudo lxc-create -t ubuntu -n memlimit

Once the container is built, we need to modify the config files. Look at /var/lib/lxc/memlimit/config. We need to add a few lines to that file. I’m going to limit memory to 512M and total usage or memory + swap to 1G. Note the second setting is for overall memory + swap, not just swap usage.

lxc.cgroup.memory.limit_in_bytes = 512M
lxc.cgroup.memory.memsw.limit_in_bytes = 1G

Now let’s start the container and get some debug info out of it to make sure these were set:

sudo lxc-start -n memlimit -l debug -o debug.out

The debug.out file will show up wherever you ran lxc-start from, so let’s see if it picked up our limits:

lxc-start 1359136997.617 DEBUG lxc_conf - cgroup 'memory.limit_in_bytes' set to '512M'
lxc-start 1359136997.617 DEBUG lxc_conf - cgroup 'memory.memsw.limit_in_bytes' set to '1G'

Looks good to me!

Note, I tried setting this once to 1.5G and it seems that the fields are only happy with whole numbers, it complained about 1.5G. That error message appeared in the same debug log I used above.

A list of more of the values you can set in here is available here.

Measuring Memory Usage

The view of /proc/meminfo inside the container and outside the container are the same. This means that you cannot rely on tools like top to show how much memory the container is using. In other words, when run inside the container top will correctly only show processes inside the container, it will also show overall memory usage for the entire system. To get valid information, instead we need to examine some files in /sys:

Current memory usage:

Current memory + swap usage:

Maximum memory usage:

Maximum memory + swap usage:

You can use expr to show it as kb or mb which is easier to read for me:

expr `cat memory.max_usage_in_bytes` / 1024

What Happens When the Limit is Reached?

When the cap is reached, the container simply behaves as if the system ran out of memory. Calls to malloc will start failing (returning -1), leading to strange and bad things happening. Dialog boxes may not open, you may not be able to save files, and more than likely where people didn’t bother to check the returned value from malloc (aka, everyone), you’ll get segfaults. You can alleviate the pressure like normal systems do, by enabling swap inside the container, but once that limit is reached, you’ll have the same problem. In this case the host system’s kernel should start firing up the OOM killer and killing stuff inside the container.

Here is my extremely simple test program to drive up memory usage, install gcc in your container and you can try it too:


int main(void) {
    int i;
    for (i=0; i<65536; i++) {
        char *q = malloc(65536);
        printf ("Malloced: %ld\n", 65536*i);

You can simply compiled with: gcc -o foo foo.c

I used the following simple shell construct to watch the memory usage. This needs to be run outside the container and I ran it from the /sys directory mentioned above:

while true; do echo -n "Mem Usage (mb): " \&\& expr `cat memory.usage_in_bytes` / 1024 / 1024; echo -n "Mem+swap Usage (mb): " \&\& expr `cat memory.memsw.usage_in_bytes` / 1024 / 1024; sleep 1; done

With the above shell script runnint, I fired up a bunch of copies of foo one bye one. Here’s the memory usage from that script:

Running a few copies:

Mem+swap Usage (mb): 825
Mem Usage (mb): 511
Mem+swap Usage (mb): 859
Mem Usage (mb): 511

A new copy of foo is starting:

Mem+swap Usage (mb): 899
Mem Usage (mb): 511
Mem+swap Usage (mb): 932
Mem Usage (mb): 511
Mem+swap Usage (mb): 1010
Mem Usage (mb): 511

The OOM killer just said “Nope!”

Mem+swap Usage (mb): 814
Mem Usage (mb): 511
Mem+swap Usage (mb): 825
Mem Usage (mb): 511

At the point where the OOM killer fired up, I see this in my container:
[1] Killed ./foo

So the limits are set, and they’re working.


If you are using LXC or considering using LXC, you can use a memory limit to protect the host from a container run amok. You could also use it to test your code in an artificially restricted environment. In either case, try the tools above and let me know how it works for you.

Tagged , ,

13 thoughts on “Limiting LXC Memory Usage

  1. Jerry says:

    Nifty, and it looks like a great way to test software in the “out of memory” scenario. I’ll have to mess around with it too.

    My big question, though. Has this been ported to Cygwin? 🙂

    • Daniel says:

      Well, LXC is in essence a super version of chroot, it only works with the Linux kernel since it is a feature of it.

      So cygwin can’t do that since it’s simply a layer between unix programs and windows. So no, it isn’t and won’t be ported to cygwin

  2. Simon Daby says:

    Useful stuff, I will be playing with this, as I can sometimes have 40+ containers on one machine, so having some hard limits might be useful.

  3. ARFR says:

    Thanks and for stress testing you can you simple use the stress tool without writing own scripts (apt-get install stress) 😉

  4. Fainean says:

    How can one check if the cgroup flags are checked or not. My /boot/config does not have these flags and i am on Ubuntu 14.10. Also, i ran lxc-checkconfig and it shows that my memory controller is enabled. But yet i am unable to limit the mem+swap space. Container does not want to start with the config statements in this article. Thanks in advance

    • Simon says:

      Could you tell the version of LXC you’running? LXC v1.0.6 on my Debian Jessie starts correctly and the limitations are applied as well/

      • I see the same — specifically :
        Debian Jessie

        lxc 1:1.0.6-6

        /proc/cmdline contains : cgroup_enable=memory

        lxc-checkconfig shows everything is good, but I can’t see the ‘swap’ options —

        /sys/fs/cgroup/memory/lxc/ contains :

        • Hansal Shah says:

          @David I am using linux 3.16 and even I cannot find the swap options.
          Were you still able to limit the memory? If yes, could you please share!!

  5. Hansal Shah says:

    @David I am using linux 3.16 and even I cannot find the swap options.
    Were you still able to limit the memory? If yes, could you please share!!

  6. erkko says:

    it seems that ubuntu-compiled kernels have been failing to include that config option for some time…

    current 15.04 and 3.19.0-25 also has no memsw -features in /boot/config -file.

    have not tried myself yet, but someone has successfully worked around it using grub kernel -line addition swapaccount=1

  7. Richard says:

    Since a few days vivid seems to have enabled the options in to the kernel again (do a apt-get update && apt-get upgrade and then reboot). Then following now finally gives a working system again. Although the swap limit is not seen in htop (maybe related to

  8. Roger says:

    You mentioned that malloc fails when attempting to overcommit, that resembles strict overcommit with no need for the OOM, but your example shows the OOM working. In that case the malloc can’t be failing, if so, it would have failed to start and the OOM didn’t need to kill anything. Can you explain this?

Leave a Reply

Your email address will not be published. Required fields are marked *