Docker Volumes for DevOps Engineers

▶What is Docker Volume?

Docker volumes are a widely used and useful tool for ensuring data persistence while working in containers. Docker volumes are file systems mounted on Docker containers to preserve data generated by the running container.

• The data does NOT persist when that container no longer exists, and it can be difficult to get the data out of the container if another process needs it.

• A container's writable layer is tightly coupled to the host machine where the container is running. The data cannot be easily moveable somewhere else.

• Writing into a container's writable layer requires a storage driver to manage the filesystem.

Docker has two options for containers to store files in the host machine so that the files are persisted even after the container stops:

1. Volumes are stored in a part of the host filesystem, which is managed by

2. Bind mounts may be stored anywhere on the host system.

Docker volumes are managed by Docker and a directory is created on /var/lib/docker/volumesthe container instance that contains the volume data.

Some common use cases for Docker volumes are as follows:

• To provide persistent data volumes for use with containers

• To share a data volume defined in different locations in different containers on the same container instance

• To define an empty, non-persistent data volume and mount it to multiple containers in the same job

• To provide a volume of data for your task managed by a third-party driver

The volumes are stored on the host, independent of the container life cycle. This allows users to back up data and share file systems between containers easily.

Types of Mount

In all types of mounts, the data looks the same from within the container. It is exposed as either a directory or an individual file in the container's filesystem.

1. Volumes: are stored in a part of the host filesystem, managed by Docker (/var/lib/docker/volumes/ on Linux). Non-Docker processes should not modify this part of the filesystem. Volumes are the best way to persist data in Docker. We can create a volume explicitly using the docker volume create command, or Docker can create a volume during container or service creation.

Volumes are isolated from the core functionality of the host machine.

A Volume can be mounted into multiple containers simultaneously. When no running container uses a volume, the volume is still available to Docker and not removed automatically. But we can remove unused volumes using Docker volume prune.

Volumes also support volume drivers, allowing storing the data on remote hosts or cloud providers, among other possibilities.

2. Bind mounts: may be stored anywhere on the host system. They may be important system files or directories. Non-Docker processes on the Docker host or a Docker container can modify them at any time. Bind mounts have limited functionality compared to volumes.

When we use a Bind Mount, a file or directory on the host machine is mounted into a container. The file or directory is referenced by its full path on the host machine. The file or directory does not need to exist on the Docker host already. It is created on-demand if it does not yet exist.

Bind mounts are very efficient, but they rely on the host machine's filesystem, having a specific directory structure available. If we want to develop new Docker applications, then consider using named volumes instead of bind mounts. We cannot use Docker CLI commands to directly manage bind mounts.

3. tmpfsmounts: are stored in the host system's memory only and are never written to the host system's filesystem. It is not persisted on disk, either on the Docker host or within a container.

tmpfs mount can be used during the container's lifetime to store non-persistent state or sensitive information.

4. Named Pipes: An named pipe mount can be used for communication between the Docker host and a container. The common use case is to run a third-party tool inside a container and connect to the Docker Engine API using a named pipe.

Bind mounts and volumes can both be mounted into containers using the -v or --volume flag, but the syntax for each is slightly different.

In Docker 17.06 and higher, we use the --mount flag for both containers and services, for bind mounts, volumes, or tmpfs mounts.

Some commands related to Docker Volumes:

To create a Volume

$ docker volume create <volume_name>

To list volumes

$ docker volume ls

To remove a volume:

$ docker volume rm <volume_name>

To start a container with a volume:

If you start a container with a volume that does not yet exist, Docker creates the volume for you.

You can choose to use them -v and --mount but I prefer to use --mount on the examples below to produce the same result. You cannot run them both unless you remove the devtest container and the testvol1 volume after running the first one.

 $ docker run -d \
  --name <container_name> \
  --mount source=<volume_name>,target=/<dir> \
  nginx:latest

To inspect a volume:

$ docker inspect <volume_name>

This shows that the mount is a volume, it shows the correct source and destination, and that the mount is read-write.

Stop the container and remove the volume. Note volume removal is a separate step.

To attach a Volume to a container:

$ docker run -v <volume_name>:/<dir_container> <image_name>

To detach a Volume from a Container:

To detach a volume from a running container, you need to stop and remove the container. The volume will still exist and can be attached to other containers if needed.

$ docker container stop <container_name>
$ docker container rm <container_name>

Here in this link, you can find more commands for Volumes from the Docker official documentation.

▶Task 1

• Create a multi-container docker-compose file which will bring UP and bring DOWN containers in a single shot ( Example - Create application and database container )

hints:

• Use the docker-compose up command with the -d flag to start a multi-container application in detached mode.

• Use the docker-compose scale command to increase or decrease the number of replicas for a specific service. You can also add replicas in deployment file for auto-scaling.

• Use the docker-compose ps command to view the status of all containers, and docker-compose logs to view the logs of a specific service.

• Use the docker-compose down command to stop and remove all containers, networks, and volumes associated with the application

Step 1 - Create a docker-compose YAML file for a Web Server and MySQL DB.

Step 2 - Run the docker-compose up command with the -d flag to start a multi-container application in detached mode.

Step 3 - Use the docker-compose scale command to increase or decrease the replicas of a service. You can also add replicas to the deployment file for auto-scaling.

$ docker-compose up --scale <container_name>=<no number of replicas> -d

Step 4 - Checking the number of container replicas running and their state using docker-compose ps. Also, you can check the logs of a specific service using the docker-compose logs by making a grep (i.e. nginx)

$ docker-compose ps
$ docker-compose logs | grep -i nginx

Step 5 - Use the docker-compose down command to stop and remove all containers, networks, and volumes associated with the application.

$ docker-compose down

▶Task 2

• Learn how to use Docker Volumes and Named Volumes to share files and directories between multiple containers.

• Create two or more containers that read and write data to the same volume using the docker run --mount command.

• Verify that the data is the same in all containers by using the docker exec command to run commands inside each container.

• Use the docker volume ls command to list all volumes and the docker volume rm command to remove the volume when you're done.

Step 1 - To mount a shared volume to a container

Listing the current Volumes available.

We will create a Docker Container with the Nginx base Image and mount the testvol Volume to that Container using the -v flag.

$ docker run -ti -d -v <vol_name>:/<dir_name> --name <container_name> \ <image_name>

Step 2 - To create a file inside the Docker Volume

Inside the bash of the Container, create a new file and add some content.

$ docker exec -it container1 /bin/bash

Step 3 - To create another Container and Mount the same Volume

$ docker run -ti -d -v <vol_name>:/<dir_name> --name <container_name> \ <image_name>

If you go to the shared-volume directory and list the file, you will find the testsharedvolume.txt file that you had created in the same volume but mounted in container1 earlier and it also has the same content inside it. This is so because the volume is shared among the two Containers.

Thank you for reading this article and hope it is helpful to you.