Browsed by
Category: MariaDB

MariaDB Crashing Under Docker on Google F1 Micro Instance

MariaDB Crashing Under Docker on Google F1 Micro Instance

This website is being hosted on a Google F1 Micro Instance with 600MB of memory. A few days after enabling Jetpack I noticed the website had a DB connection error.

First I checked the running containers: docker ps

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                             PORTS                   NAMES
f1bcda68c62d        wordpress           "docker-entrypoint..."   6 hours ago         Up 6 hours                         10.128.0.2:80->80/tcp   dockerwordpress_wordpress_1
55bb57dfdc8d        mariadb             "docker-entrypoint..."   6 hours ago         Restarting (1) About an hour ago                           dockerwordpress_mariadb_1

Then I viewed the logs for the restarting container: docker logs dockerwordpress_mariadb_1

2017-04-04  8:28:46 139747895928768 [Note] mysqld (mysqld 10.1.21-MariaDB-1~jessie) starting as process 1 ...
2017-04-04  8:28:46 139747895928768 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2017-04-04  8:28:46 139747895928768 [Note] InnoDB: The InnoDB memory heap is disabled
2017-04-04  8:28:46 139747895928768 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-04-04  8:28:46 139747895928768 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2017-04-04  8:28:46 139747895928768 [Note] InnoDB: Compressed tables use zlib 1.2.8
2017-04-04  8:28:46 139747895928768 [Note] InnoDB: Using Linux native AIO
2017-04-04  8:28:46 139747895928768 [Note] InnoDB: Using SSE crc32 instructions
2017-04-04  8:28:46 139747895928768 [Note] InnoDB: Initializing buffer pool, size = 256.0M
InnoDB: mmap(281542656 bytes) failed; errno 12
2017-04-04  8:28:46 139747895928768 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2017-04-04  8:28:46 139747895928768 [ERROR] Plugin 'InnoDB' init function returned error.
2017-04-04  8:28:46 139747895928768 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2017-04-04  8:28:47 139747895928768 [ERROR] mysqld: Out of memory (Needed 128663552 bytes)
2017-04-04  8:28:47 139747895928768 [ERROR] mysqld: Out of memory (Needed 96485376 bytes)
2017-04-04  8:28:47 139747895928768 [ERROR] mysqld: Out of memory (Needed 72351744 bytes)
2017-04-04  8:28:47 139747895928768 [Note] Plugin 'FEEDBACK' is disabled.
2017-04-04  8:28:47 139747895928768 [ERROR] Unknown/unsupported storage engine: InnoDB
2017-04-04  8:28:47 139747895928768 [ERROR] Aborting

The MariaDB process was abruptly terminated before the failed restarts and there is no log output to show it shutting down. The guilty memory hog on the system seems to be the dockerwordpress container.

To stop this from happening again I made two changes:

  • First I modified the docker-compose.yml to constrain the memory used by dockerwordpress by adding the mem_limit directive
wordpress:
    image: wordpress
    restart: always
    mem_limit: 200MB
    links:
     - mariadb:mysql
    environment:
     - WORDPRESS_DB_PASSWORD=db_password
    ports:
     - "80:80"
    volumes:
     - /site_data/code:/code
     - /site_data/html:/var/www/html
mariadb:
    image: mariadb
    restart: always
    environment:
     - MYSQL_ROOT_PASSWORD=db_password
     - MYSQL_DATABASE=wordpress
    volumes:
     - /site_data/database:/var/lib/mysql

This seems to have had no major negative effects on Apache.

  • Next (just to be safe) I enabled 1024MB of disk swap. By default Docker will allow a container to swap up to twice the memory limit of the container, so in this case 400MB.
[email protected]:$ dd if=/dev/zero of=/swap bs=1M count=1024
[email protected]:$ mkswap /swap
[email protected]:$ swapon /swap

You can check swap is available and working: free -m:

             total       used       free     shared    buffers     cached
Mem:           588        543         44         43         13        170
-/+ buffers/cache:        359        228
Swap:         1023         32        991

Finally, after bringing up the WordPress and MariaDB containers you can check their memory utilization: docker stats:

CONTAINER           CPU %               MEM USAGE / LIMIT       MEM %               NET I/O             BLOCK I/O           PIDS
05cee6b27a54 0.00% 177.4 MiB / 200 MiB 88.72% 6.31 MB / 1.33 MB 43 MB / 24.6 kB 11
c6658a81bd3a 0.03% 119.1 MiB / 588.5 MiB 20.23% 799 kB / 5.9 MB 50.2 MB / 89.8 MB 29