Browsed by
Month: April 2017

Move a Ceph OSD Journal

Move a Ceph OSD Journal

Here are the steps to move a Ceph OSD journal. I previously described how to setup a journal partition here. In my case I am moving the journal on OSD 8. You can get lookup the correct UUID using: ls -l /dev/disk/by-partuuid/

Don’t forget to set (and unset) noout so Ceph doesn’t start rebalancing when the OSD in question temporarily dissappears.

[email protected]:$ ceph osd set noout
set noout
[email protected]:$ systemctl stop [email protected]
[email protected]:$ ceph-osd -i 8 --flush-journal
2017-04-25 23:16:38.445663 7f93bd7f1800 -1 flushed journal /var/lib/ceph/osd/ceph-8/journal for object store /var/lib/ceph/osd/ceph-8
[email protected]:$ rm -f /var/lib/ceph/osd/ceph-8/journal
[email protected]:$ ln -s /dev/disk/by-partuuid/5bc920ad-20ce-4e07-b879-f8d32556c65a /var/lib/ceph/osd/ceph-8/journal
[email protected]:$ echo "5bc920ad-20ce-4e07-b879-f8d32556c65a" > /var/lib/ceph/osd/ceph-8/journal_uuid
[email protected]:$ ceph-osd -i 8 --mkjournal
2017-04-25 23:21:06.353543 7f0eaf792800 -1 created new journal /var/lib/ceph/osd/ceph-8/journal for object store /var/lib/ceph/osd/ceph-8
[email protected]:$ systemctl start [email protected]
[email protected]:$ ceph osd unset noout
unset noout

Issue ceph osd tree to make sure everything is up and in and you are good to go.

Docker to solve SuperMicro IPMI iKVM – JavaWS Problems

Docker to solve SuperMicro IPMI iKVM – JavaWS Problems

icedtea-web 1.6.2 does not seem to work with SuperMicro’s IPMI Java iKVM viewer. SuperMicro’s helpful response is to only use Oracle’s Java.

net.sourceforge.jnlp.LaunchException: Fatal: Initialization Error: Could not initialize application. The application has not been initialized, for more information execute javaws from the command line.

Even when you have the right version of Java you often have to dance through security hoops or Java versions just to get it to work.

If you have Docker installed there is a great solution that avoids installing Oracle’s Java and/or tweaking any security settings. solarkennedy has created a very nice Docker container that encapsulates everything needed to access various Java based IPMI consoles.

 docker run -p 8080:8080 solarkennedy/ipmi-kvm-docker

Now point your browser to http://localhost:8080 and voila:

You are looking at a Java enabled Firefox (and OS) through a web VNC client accessed from the Docker host. Not bad!

MariaDB Crashing Under Docker on Google F1 Micro Instance

MariaDB Crashing Under Docker on Google F1 Micro Instance

This website is being hosted on a Google F1 Micro Instance with 600MB of memory. A few days after enabling Jetpack I noticed the website had a DB connection error.

First I checked the running containers: docker ps

CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                             PORTS                   NAMES
f1bcda68c62d        wordpress           "docker-entrypoint..."   6 hours ago         Up 6 hours               >80/tcp   dockerwordpress_wordpress_1
55bb57dfdc8d        mariadb             "docker-entrypoint..."   6 hours ago         Restarting (1) About an hour ago                           dockerwordpress_mariadb_1

Then I viewed the logs for the restarting container: docker logs dockerwordpress_mariadb_1

2017-04-04  8:28:46 139747895928768 [Note] mysqld (mysqld 10.1.21-MariaDB-1~jessie) starting as process 1 ...
2017-04-04  8:28:46 139747895928768 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2017-04-04  8:28:46 139747895928768 [Note] InnoDB: The InnoDB memory heap is disabled
2017-04-04  8:28:46 139747895928768 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-04-04  8:28:46 139747895928768 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2017-04-04  8:28:46 139747895928768 [Note] InnoDB: Compressed tables use zlib 1.2.8
2017-04-04  8:28:46 139747895928768 [Note] InnoDB: Using Linux native AIO
2017-04-04  8:28:46 139747895928768 [Note] InnoDB: Using SSE crc32 instructions
2017-04-04  8:28:46 139747895928768 [Note] InnoDB: Initializing buffer pool, size = 256.0M
InnoDB: mmap(281542656 bytes) failed; errno 12
2017-04-04  8:28:46 139747895928768 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2017-04-04  8:28:46 139747895928768 [ERROR] Plugin 'InnoDB' init function returned error.
2017-04-04  8:28:46 139747895928768 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2017-04-04  8:28:47 139747895928768 [ERROR] mysqld: Out of memory (Needed 128663552 bytes)
2017-04-04  8:28:47 139747895928768 [ERROR] mysqld: Out of memory (Needed 96485376 bytes)
2017-04-04  8:28:47 139747895928768 [ERROR] mysqld: Out of memory (Needed 72351744 bytes)
2017-04-04  8:28:47 139747895928768 [Note] Plugin 'FEEDBACK' is disabled.
2017-04-04  8:28:47 139747895928768 [ERROR] Unknown/unsupported storage engine: InnoDB
2017-04-04  8:28:47 139747895928768 [ERROR] Aborting

The MariaDB process was abruptly terminated before the failed restarts and there is no log output to show it shutting down. The guilty memory hog on the system seems to be the dockerwordpress container.

To stop this from happening again I made two changes:

  • First I modified the docker-compose.yml to constrain the memory used by dockerwordpress by adding the mem_limit directive
    image: wordpress
    restart: always
    mem_limit: 200MB
     - mariadb:mysql
     - WORDPRESS_DB_PASSWORD=db_password
     - "80:80"
     - /site_data/code:/code
     - /site_data/html:/var/www/html
    image: mariadb
    restart: always
     - MYSQL_ROOT_PASSWORD=db_password
     - MYSQL_DATABASE=wordpress
     - /site_data/database:/var/lib/mysql

This seems to have had no major negative effects on Apache.

  • Next (just to be safe) I enabled 1024MB of disk swap. By default Docker will allow a container to swap up to twice the memory limit of the container, so in this case 400MB.
[email protected]:$ dd if=/dev/zero of=/swap bs=1M count=1024
[email protected]:$ mkswap /swap
[email protected]:$ swapon /swap

You can check swap is available and working: free -m:

             total       used       free     shared    buffers     cached
Mem:           588        543         44         43         13        170
-/+ buffers/cache:        359        228
Swap:         1023         32        991

Finally, after bringing up the WordPress and MariaDB containers you can check their memory utilization: docker stats:

CONTAINER           CPU %               MEM USAGE / LIMIT       MEM %               NET I/O             BLOCK I/O           PIDS
05cee6b27a54 0.00% 177.4 MiB / 200 MiB 88.72% 6.31 MB / 1.33 MB 43 MB / 24.6 kB 11
c6658a81bd3a 0.03% 119.1 MiB / 588.5 MiB 20.23% 799 kB / 5.9 MB 50.2 MB / 89.8 MB 29
Benchmarking Ceph on a Two Node Proxmox Cluster

Benchmarking Ceph on a Two Node Proxmox Cluster

It is inadvisable to run Ceph on two nodes! That said I’ve been using a two node Ceph cluster as my primary data store for several weeks now.

In this post we look at the relative read and write performance of replicated and non-replicated Ceph pools using Rados Bench and from VM Guests using various backends. We’ll start with the results – the details of how we generated them are included after the break.

Read More Read More