Docker and MySQL: the ups and downs

Standard

Docker and MySQL will play happily together. But there’s a few gotchas.

On a production system I manage, I run MySQL as a Docker container, in a private network shared between other containers that need to access it. As a result, we benefit from network and container isolation. However there are downsides.

The network isolation is great from a security perspective as MySQL isn’t exposed to the public at all.

Having it segregated in a Docker container gets all the benefits of container isolation, e.g. an exploit against another container would only give an attacker a connection to the container. It would not allow access to the data directly. Provided each container has a credential and database an attack on one container would at worst allow access at the privilege level of the container.

Separating the MySQL data into a volume lets you upgrade the version of MySQL fairly easily. You can just re-create the container provided you separated the data out as a volume. (and perhaps run mysql_upgrade afterwords)

However, Docker out of the box can completely destroy your database. The problem arises from how Docker kills a container. By default, Docker sends a SIGTERM and waits 10 seconds. If the process does not end by 10 seconds, Docker proceeds to effectively kill -9 (SIGKILL) the main process inside the container. I honestly fail to see why this timeout is so low by default. 10 seconds isn’t really enough time to even stop a small-sized database.

Anyone familiar with MySQL can see that this a problem. The SIGTERM sent by Docker causes MySQL to start cleaning up and writing to disk anything which was in memory. On a large database, this can take a few minutes. Interrupting this is BAD, which Docker proceeds to do 10 seconds after! If you’re lucky, and most likely most of the time, MySQL will recover.

This is a very large downside which by default isn’t exactly advertised. Oddly, the default MySQL and MariaDB images do not mention on their Docker Hub or GitHub pages that this should be a consideration. I understand MySQL has a great crash recovery, however I have personally been hit with this problem and can say it isn’t perfect.

This behavior will not only occur whenever you reboot the machine or stop the container, it will also happen without warning during an upgrade of Docker. (at least on CentOS 7)

Docker provides two ways of mitigating this. When creating the container with docker run, you can pass –stop-timeout=6000 which would give the container 100 minutes to gracefully shutdown. I recommend you set this very high as MySQL can be unpredictable in how long it takes to stop.

If you’ve already created the container, you can use docker stop -t 6000 to stop the container with a grace of 100 minutes. I highly recommend that you rebuild the container with an appropriate stop value however.

So as much as Docker provides advantages, there is a huge undocumented behavior that could cause significant damage to a database.

Leave a Reply

Your email address will not be published. Required fields are marked *