John Bokma

Docker for Mac Named Volume Speed Penalty

September 21, 2016

Summary

Based on some test I did today; importing a 10MB gzip compressed MySQL database dump, it seems that using a named volume is over twelve times slower than using bind mount on Docker for Mac version 1.12.1. When a named volume is used the data is stored into a virtual harddisk Docker.qcow2. Besides the significant speed penalty, this has to me also the disadvantages of a higher risk of data loss; everything is in a single file, and making it more of a hassle to restore individual files from backup into the virtual harddisk.

Introduction

Yesterday I experimented a bit with a Docker MySQL container. I decided to check out a named volume first:

$ docker run --detach --name mysql-db --env MYSQL_ROOT_PASSWORD=S3CR3T \
         --volume mysql-data:/var/lib/mysql mysql

Somehow I was expecting a mysql-data directory to show up on my Mac mini, still running "El Capitan". So, I used find to hunt down this directory in ~/Library. Nothing found. Next, I inspected the mysql-db container:

$ docker inspect mysql-db
        ...
        "Mounts": [
            {
                "Name": "mysql-data",
                "Source": "/var/lib/docker/volumes/mysql-data/_data",
                "Destination": "/var/lib/mysql",
                "Driver": "local",
                "Mode": "z",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
	...

But sudo ls /var/lib/ didn't show a docker folder. Then I realised that since "Docker for Mac" uses a virtual machine the named volume must be created in a virtual drive. So I used screen to connect to the virtual machine as follows:

screen \ ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty

Note: if you don't see a prompt, press Enter.

I logged in as root, which requires no password. Running ls -al with the path reported by docker inspect did give the expected result:

moby:~# ls -al /var/lib/docker/volumes/mysql-data/_data/
total 188452
drwxr-xr-x    5 999      ping          4096 Sep 20 23:25 .
drwxr-xr-x    3 root     root          4096 Sep 20 23:25 ..
-rw-r-----    1 999      ping            56 Sep 20 23:25 auto.cnf
-rw-r-----    1 999      ping          1329 Sep 20 23:25 ib_buffer_pool
-rw-r-----    1 999      ping      50331648 Sep 20 23:25 ib_logfile0
-rw-r-----    1 999      ping      50331648 Sep 20 23:25 ib_logfile1
-rw-r-----    1 999      ping      79691776 Sep 20 23:25 ibdata1
-rw-r-----    1 999      ping      12582912 Sep 20 23:25 ibtmp1
drwxr-x---    2 999      ping          4096 Sep 20 23:25 mysql
drwxr-x---    2 999      ping          4096 Sep 20 23:25 performance_schema
drwxr-x---    2 999      ping         12288 Sep 20 23:25 sys
moby:~# 

Note: you can quit the screen session press Ctrl+A followed by Ctrl+\ and answer "Really quit and kill all your windows" with "y", see the last part of Getting Started with Docker for Mac.

The virtual harddisk is /dev/vda2 and the actual data is stored in the file Docker.qcow2 on the host computer, in my case a Mac mini:

$ cd ~
$ ls -hl Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/
total 2251328
-rw-r--r--   1 john  staff   1.1G Sep 20 18:13 Docker.qcow2
:
:

I don't consider it a safe idea, to store database related files all together into a virtual disk that will be backed up as a single large file. So I decided to check out bind mount; mounting a directory on OS X onto a path in the MySQL container. This way I would have access to all the database related files directly on OS X. This would make recovery from back up much easier in case of a mishap. But which method was faster? As I wanted to import a very large database dump, 1.2G compressed using gzip -9, speed was very important to me. Since I already had created a container with a named volume using:

$ docker run --detach --name mysql-db --env MYSQL_ROOT_PASSWORD=mypassword \
         --volume mysql-data:/var/lib/mysql mysql

I decided to test first the speed of storing data into a virtual harddisk. After a few attempts, which gave me the impression that this method would be slow, I decided to run the import using time to measure how long it would take.

$ time (gzip -dc big-data-dump.sql.gz  |\
        docker exec --interactive mysql-db \
        bash -c 'mysql -uroot -p$MYSQL_ROOT_PASSWORD')
mysql: [Warning] Using a password on the command line interface can be insecure.

Note that I use bash so I can use the environment variable $MYSQL_ROOT_PASSWORD, which is set to the actual password inside the container, instead of exposing the actual password. If there is an easier way to do this, let me know.

This morning, the process was still running. A quick check using du -hs showed that about 24% was done. I canceled the process, this was taking way too long; it had been running for close to 14 hours. As I had imported a smaller version, but still large, version of this database before in a Parallels virtual machine so I knew that this was excessive. So I decided to first do some tests with a much smaller database dump.

Edit: there seems to be an issue with the large database dump as I have problems importing it using bind-mount as well.

Named Volume versus Bind-mount

I run a local version of MediaWiki to keep notes and links. Its database dump compressed using gzip -9 is just 10M. I ran four tests using a named volume, the results are given below. Because I was not sure if VirtualBox might have something to do with the slow down I quit this program before I ran the last test. But no significant difference with the previous three runs.

Before each test I quit Docker and deleted Docker.qcow2 using:

$ rm Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/\
Docker.qcow2 

Then I restarted Docker and created the container and did the import as follows:

$ docker run --detach --name mysql-db --env MYSQL_ROOT_PASSWORD=S3CR3T \
         --volume mysql-data:/var/lib/mysql mysql
$ time (gzip -dc wikidb-20160921-100357.sql.gz | docker exec --interactive \
	    mysql-db bash -c 'mysql -uroot -p$MYSQL_ROOT_PASSWORD')

Results, formatted horizontally:

+-----------+----------+----------+
| real      | user     | sys      |
+-----------+----------+----------+
| 4m23.581s | 0m0.239s | 0m0.132s |
| 4m15.187s | 0m0.242s | 0m0.129s |
| 4m03.262s | 0m0.243s | 0m0.129s |
| 4m12.767s | 0m0.240s | 0m0.126s |
+-----------+----------+----------+

The average "real" over 4 runs is 4m13.634s.

Next I tested using a bind-mount. Again, before each test I quit Docker and deleted Docker.qcow2 using:

$ rm Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/\
Docker.qcow2 

Then I restarted Docker and created the container and did the import as follows:

$ mkdir -p ~/Docker/volumes/mysql-data/
$ docker run --detach --name mysql-db --env MYSQL_ROOT_PASSWORD=S3CR3T \
         --volume /Users/john/Docker/volumes/mysql-data/:/var/lib/mysql mysql
$ time (gzip -dc wikidb-20160921-100357.sql.gz | docker exec --interactive \
        mysql-db bash -c 'mysql -uroot -p$MYSQL_ROOT_PASSWORD')
$ rm -rf ~/Docker/volumes/mysql-data/

Results, formatted horizontally:

+-----------+----------+----------+
| real      | user     | sys      |
+-----------+----------+----------+
| 0m21.033s | 0m0.242s | 0m0.133s |
| 0m21.281s | 0m0.236s | 0m0.132s |
| 0m19.348s | 0m0.235s | 0m0.128s |
| 0m20.143s | 0m0.238s | 0m0.128s |
+-----------+----------+----------+

The average "real" over 4 runs is 20.451s. Or using bind-mount instead of a named volume is 12.4 times faster.

Out of curiousity I also ran four tests inside a Ubuntu 15.10 installation running under VirtualBox, repeating the following two lines:

$ mysql -uroot --password=S3CR3T -e 'DROP DATABASE wikidb'
$ time (gzip -dc wikidb-20160921-100357.sql.gz | mysql -uroot --password=S3CR3T)
+-----------+----------+----------+
| real      | user     | sys      |
+-----------+----------+----------+
| 0m48.117s | 0m1.196s | 0m0.220s |
| 0m40.243s | 0m1.228s | 0m0.160s |
| 0m39.176s | 0m1.208s | 0m0.136s |
| 0m42.999s | 0m1.172s | 0m0.328s |
+-----------+----------+----------+

As the configuration of MySQL might be different I can't conclude that VirtualBox is about twice as slow compared to Docker for Mac using bind-mount. Moreover, I had migrated this virtual machine from Parallels which might have resulted in a less than optimal virtual disk layout.

Conclusion

The current version of Docker for Mac; Version 1.12.1 (build: 12133), is over 12 times slower when using a named mount compared to bind-mount when importing a MySQL database dump.

Discussion

It's not clear to me why a virtual disk is being used for storing data. Not only named and unnamed mounts use this disk, but it's also used for storing images, etc. As this virtual disk has a limit of 60G, which can be much lower than the space available on the host operating system, this can lead to confusion of users; they might get an "out of space" error message while they have more than enough space left on the hosts' hard drive. As all data is stored in a monolithic file there is quite a risk of accidental data loss. And of course, this makes recovery of specific files from backup cumbersome.

Since it is already possible to access directories and files on the host operating system it's unclear to me why Docker choose this method. I would have preferred to have images somewhere stored under ~/Library. And maybe the same for named and unnamed volumes.

home - blog - email

I am a freelance Perl programmer for hire, download my up-to-date resume (PDF).