Problem Scenario
You have been tasked with finding ways of securing Docker containers, a Docker host, and your network that has Docker containers. How do you harden a Docker container and its related infrastructure (i.e., the Docker network and Docker host)?
Solution
Overview
From a pragmatic perspective we understand that there are exceptions to the recommended practices. (Not the least of which is the rapid rate that technology changes.) The assertions below are recommended practices. There can be ways to mitigate risks when you implement Docker contrary to these recommendations below. The following are 50 possible tips -- none by themselves is sufficient for securing a container. They are in random order.
(In this site's articles, we use some practices that are specifically contrary to the recommendations below. For proofs-of-concept experimentation in secure environments without sensitive data, we find it is acceptable to not use recommended security practices for containers. If you are hardening your containers for a company, it is best to follow the practices below rather than the articles that illustrate methods that are contrary to these recommendations.)
1. If your Docker host is a Linux server, ensure you keep the kernel up-to-date. Docker containers emulate resources by having direct access to the kernel. You need your kernel to be patched because the containers share this precious and potentially vulnerable resource. Also, #5 below is related to hardening the kernel.
2. OWASP and other authorities emphasize the origin of Docker images being a huge potential security issue. Slide 7 of this presentation says that a typical image on Docker Hub contains over 170 vulnerabilities. You may want to build the image yourself, host your own registry, or be very selective if you use a Docker image from Docker Hub.
3. If your Docker host is a Linux server, check the configuration of /etc/sysctl.conf. This important file sets kernel parameters. To ensure it is secure, see this posting (https://www.cyberciti.biz/faq/linux-kernel-etcsysctl-conf-security-hardening/). If the Docker host is a Linux server, you should harden it as much as possible; to do this see this posting.
4. Start the Docker services with a regular user. Do not start Docker services with root. Try to avoid starting a Docker container with the "sudo " command. If you need assistance with this, see How Do You Run Docker commands without sudo?.
5. Use AppArmor, Grsecurity, or SELinux. Some people recommend AppArmor because it has fewer limitations than SELinux (page 170 of Docker: Up and Running).
"You can add an extra layer of safety by enabling AppArmor, SELinux, GRSEC, or another appropriate hardening system." taken from Docker's website.
Beyond using it at the host level, AppArmor can be used at container level. To use it at the container level, you will configure a profile; to learn more about this, see this posting.
"The default SELinux policy for Docker is designed to protect the host from containers, as well as containers from other containers... Be aware that you will be unable to run SELinux commands from inside containers. Inside the container, SELinux appears to be turned off, which prevents applications and users from trying to run commands such as setting SELinux policies that will get blocked by SELinux on the host." The previous quote was taken from this PDF. (1)
"The modules normally used with Docker are SELinux (typically with Red Hat-based distributions) and AppArmor (typically with Ubuntu and Debian distributions). " This was also taken from this PDF. (1)
Outside of the context of Docker, some people prefer SELinux or Grsecurity (rather than AppArmor); for more information about their differences see this posting.
If you install the grsecurity kernel patch or the related PaX component, be aware that you have to pay for either of them.
"You can run a kernel with GRSEC and PAX" according to Docker's website.
Grsecurity's website (on 12/23/18) said "[o]ur unmatched defenses add critical hardening to the Linux kernel, a ripe source of vulnerabilities and involved in most container escapes." You can download Grsecurity from this site if you are a Grsecurity customer. PaX is a major component of Grsecurity. You can read more about it here.
6. Slide 22 of this presentation says that you should not use docker run; it says you should use docker compose instead. Many of the recommendations below pertain to "docker run". To avoid using "docker run" some of the recommendations can be adapted to .yaml files as the input of Docker compose commands. You may want to consult with this page for using Docker compose (also known as the docker-compose command).
7. Use scanning tools on the Docker images and Dockerfiles. Go to this link and see "Rule #9".
This next part is not specific to Docker, but relevant to security from a pragmatic perspective. If you have custom development running in the Docker instance, analyze the code for potential security flaws. SonarQube combined with SonarScanner may be able to help you with this. If you need directions for installing SonarQube, see these postings:
- How Do You Install SonarQube Scanner on a RHEL Server?
- How Do You Install SonarQube on a Red Hat Distribution of Linux?
8. Is the Docker version up-to-date? Run this command from the Docker host: docker version
In a web browser, go to this site. A newer version of Docker may be more secure.
9. Implement centralized logging for the benefit of being able to keep track of events of failed Docker hosts and potential security breaches. Audit the logs periodically.
Slide 9 of this presentation says that attackers can remove local logs. Having a remote, centralized logging server can be your only hope for understanding what happened as irretrievable logs can be associated with Docker hosts that need to be reformatted.
You may want to configure Splunk, Graylog, the Elastic Search logging tools, or see other articles related to logging, click here. This helps you configure a remote syslog on the Docker host to write to a remote server.
10. Do not share Linux user namespaces with containers because greater isolation leads to greater security. There is an option with starting the Docker daemon --userns=host that you should try to avoid using. To read about this option see this page.
11. Minify the Docker image. If your container is not minimal, rebuild the Docker image and container so that it has the minimum packages, tools and files to function properly. Having fewer tools makes the image smaller and faster. It also gives hackers fewer devices should they gain control of the container. You may want to see this GitHub repository to help.
12. If your Docker container provides a web service, ensure it uses HTTPS. Apache web server and Nginx have configuration files on the back-end to facilitate HTTPS. HTTPS uses TLS. If you want to know more about TLS and Docker containers, see this posting. If you need assistance on using TLS with Nginx, see this posting.
To learn how to create a certificate that can be installed on the web clients, see this posting. Web services by their very nature are accessible; you want the underlying components to be secure. If you need assistance with Nginx in a Docker container, see this posting.
13. Do not use the --insecure-registry flag (e.g., when starting the Docker daemon service). This would use a Docker registry that is not secure. See this article for a way to avoid using it.
14. Do not use the --privileged flag (e.g., when executing docker run). It can give the Docker container access to kernel processes that it does not need; this could be a vulnerability.
For more information see these postings:
https://www.eweek.com/security/researchers-reveal-play-with-docker-security-vulnerability
https://stackoverflow.com/questions/36425230/privileged-containers-and-capabilities
15. Make sure you can run Docker commands (other than starting the Docker service or starting a Docker container like #4 above) without sudo. If you need assistance, see this posting. Remember that the quick-and-easy way to not need sudo is for the Linux user to be a member of the docker group on the Linux server.
16. Do not run SSH inside a Docker container. If you want more information, see this article.
17. Do not map host ports that are numbered 1 through 1023 to container ports. Mapping can be done at runtime (with docker run) or with a dockerfile. The World Wide Web Consortium states that privileged ports are below 1024 here. Privileged ports tend to carry sensitive data; thus these ports should not be used by Docker because if exploited, the security problem could be worse than it would have been. Use ports 1024 to 65535.
18. Do not use the --net=host flag with the docker run command. This bypasses the containerization of networking processes (according to a StackOverflow posting). With isolation removed a number of security problems become possible.
19. Do not use the --pid=host flag with the docker run command. This bypasses the containerization of process namespace of the host. Processes in a container would be able to see other processes on the host. With isolation removed a compromised container becomes a vulnerability to the entire host. For more information, see this page on Docker's website.
20. Do not use the --ipc=host flag with the docker run command. This bypasses the containerization of the interprocess-communication space of the host for the benefit of using shared memory (according to StackOverflow and Docker's website). With isolation removed a compromised container becomes a vulnerability to the entire host.
21. Limit the amount of memory that a Docker container can use. This way if one container is compromised, it will not make the Docker host unavailable. When you use docker run use the -m flag to restrict the amount of RAM that can be consumed. For details see this.
22. Do not use the --uts=host flag with the docker run command. This bypasses the containerization of the Unix Timesharing System namespace on the host. The UTS namespace governs hostnames (according to https://windsock.io/uts-namespace/). Without containerization of this feature, hostnames of containers are liable to change. A hacker could manipulate hostnames and spoof them in ways that are undesirable. This could lead to a denial-of-service attack.
"You may wish to share the UTS namespace with the host if you would like the hostname of the container to change as the hostname of the host changes." The preceding quote was taken from Docker's website.
23. Limit the amount of CPU shares that a Docker container can use. This way if one container is compromised, it will not make the Docker host unavailable. When you use docker run use the -c flag to restrict the amount of CPU capacity that can be consumed. For details see this page.
24. Limit the number of child processes that a given Docker container can generate. This way a compromised container will less likely create a denial-of-service for the Docker host. To do this, when you issue a docker run command, use a flag and parameter combination like this --user jdoe --ulimit nproc=7. Substitute "jdoe" for the user you want associated with the process limitation and substitute "7" for the maximum number of processes you want the Docker container to be able to create. (Not that you would likely use "root" as the user in this type of flag, the nproc setting would not work for the "root" user. This is according to page 34 of this PDF.) (1)
25. Limit the number of automatic restarts that a Docker container will have. A continually stopping and restarting container will consume resources. It may contribute to a denial-of-service. The way to do this is to use the "docker run" command like this (but substitute "contint-image" with the name of the Docker image you want to run a container from):
docker run -d --restart=on-failure:10 contint-image
26. Prohibit the ability to write to a Docker container. This way if the container is compromised, no data will be manipulated and no backdoor will be created that needs to write to the file system. When you use docker run use the --read-only flag to restrict the ability to write to the container. For details see this page on Docker's website.
27. If you use the --device flag with docker run, remember to use the principle of least privilege. The --device flag gives the container access to a device on the Docker host; all things constant this is less secure because if the container was compromised, the device on the host could be compromised too. Therefore if you give a container access to a device, you can limit the access to read, write, or mknod.
Here is an example of how to use the --device flag with the docker run command while passing read permissions (without write or mknod permissions):docker run --device=/dev/contint1:/dev/contint2 --r -it ubuntu fdisk /dev/contint2
You should avoid using it like this (free of --r or something along those lines):
docker run --device=/dev/contint1:/dev/contint2
This page shows an example of how to use docker run with the --device flag
28. Do not use mount docker.sock in a container. This voted-up StackOverflow posting mentions that docker.sock increases the attack surface. (If you are not using docker.sock, then you will not have a problem.) To read about this potential exploit if you want to learn more about it, see this page.
29. Set a limit to the number of file descriptors for the Docker daemon. This limit will apply to each Docker container started after the limit takes effect. This limit will make a compromised container less likely for a denial-of-service attack.
"It may not happen all that often, but a single user who starts too many processes can make a system unusable for everyone else. A fork bomb -- a denial-of-service attack in which a process continually replicates itself until available resources are depleted -- is a worst case of this. " Taken from https://www.networkworld.com/article/2693414/operating-systems/setting-limits-with-ulimit.html.
The Linux built-in command ulimit can "get or set" (as its man page says) a maximum number of file descriptors that a user or application can create. Under normal circumstances you can run these following commands to find the ulimit value for Docker daemon if you run them when logged into the Docker host as that user:
ulimit -Hn # this will show you the hard limitulimit -Sn # this will show you the soft limit
It may be advisable to keep the maximum number for the soft limit at 1024 or at least some definite number. You may want to view .service files in /etc/systemd/system/ as they have can stanzas that will override the override the setting in the .conf files in /etc/security/limits.d/.
"A file descriptor is a pointer into a table recording information on the open files on the system." This quote was taken from this PDF. (1)
The "nofile" setting refers to the number of file descriptors that the OS can handle for a given process. If you need a Docker container to have many file descriptors, it can be run above the maximum when you issue the docker run command. This ability should help facilitate actually using the OS limitation throughout the Docker host (because there should be no human objection to its enforcement). Here is an example of a docker run command that will override the ulimit maximum:
docker run --ulimit nofile=1024:1024 --rm debian sh -c "ulimit -n"
This command was taken from Docker's website.
To learn more about ulimit outside the context of Docker with its ability to govern the number of file descriptors, see this.
30. Use Atomicorp.com's downloads to harden your containers.
31. Use Twistlock to harden your containers.
32. Create VMs for Docker hosts and separate the containers by the Linux user; this idea was taken from this PDF. (1)
This assumes that you have different Linux users running containers. Having different VMs can isolate a compromised Docker host from other Docker hosts. If a user's credentials are compromised, containers run by other users will remain secure. In production you would be most inclined to have separate VMs as Docker hosts because of the enhanced need for security.
33. Remove "setuid" and "setgid" permissions from every file in a Docker container. One way to do this is make sure each image is created from a Dockerfile with a stanza such as this:
RUN find / -perm +6000 -type f -exec chmod a-s {} \; || true
The line above was taken from this PDF. (1)
34. Restrict capabilities of Docker containers. To eliminate capabilities when executing a docker run command, use this flag: --cap-drop all
Further capabilities can be added granularly (immediately after this flag and parameter). The subsequent --cap-add flags can enable capabilities and thereby override the "--cap-drop all" explicitly and selectively. These below lines were taken from page 31 of this PDF (1) and illustrate how capabilities can be removed and granted:
$ docker run --cap-drop all debian chown 100 /tmp
chown: changing ownership of '/tmp': Operation not permitted
$ docker run --cap-drop all --cap-add CHOWN debian \
chown 100 /tmp
35. Use a firewall to protect your network. Have a separate device for your network-based Intrusion Detection System. If your firewall or your IDS is compromised, one device can still provide protection. Some IDSes have automatic intrusion prevention capabilities too. For small networks, you may want to buy something like this.
36. Install a host-based IDS on your Docker host. If you need technical assistance with different IDSes, see this link.
37. Monitored containers are more secure. Monitor the Docker containers' CPU, memory, network utilization, and disk I/O consumption. Monitoring these features can alert you to an impending denial-of-service attack. Here are some options to help you out with container monitoring:
- Prometheus
- Here is an external link that compares a dozen options.
38. If your containers have the ability to write data, make sure you back up this data. Container volumes are not backed up. If there is a denial-of-service or a hacker destroys your containers and logs, you may be in a compromising situation. Some hackers install ransomware or demand a ransom for service restoration. Having backups empowers you in this contingency.
39. Keep the number of your images and the number of containers to a minimum. While in development it may be necessary to create many different images and containers. From a pragmatic perspective it is best to keep these to a minimum. If you can clean up these experimental images and containers, you will better able to manage what you have.
40. Create a user-defined network rather than use the default networking connectivity for containers on a Docker host. The default networking is susceptible to denial-of-service attacks.
"We note here that the default connectivity model of Docker is vulnerable to ARP spoofing and Mac flooding attacks since the bridge forwards all of its incoming packets without any filtering." taken from page 4 of this PDF.
If you need assistance with creating a user-defined network, follow the solution here.
41. If you want to harden Kubernetes and its containers, see this posting to help you secure it.
42. If you want to harden OpenShift containers, see this posting.
43. Do penetration testing of your network via a trusted third party. Professionals who specialize in this can help find flaws and give you confidence your containers are secure.
44. Put server and application monitoring into place. This way you can be alerted to potential problems (especially denial-of-service attacks). To learn more about how to implement Nagios, Zabbix, or use other tools, see these postings.
45. Put networking monitoring into place. While network IDSes can monitor for intrusions, some hackers can intrude in an undetected way. Cacti is one networking monitoring tool can help you identify the source of packet floods. In a secure Docker network, network monitoring is one tool to help you avoid or minimize denial-of-service attacks.
46. While not a hardening mechanism, after a potential security breach, you may want to run docker logs (with no quotes). It can help you learn more about what has happened. To learn more about this command, see this page.
47. Do security audits of the Docker host and its containers from time-to-time. This can work to audit Docker containers. To learn more about assessing your Docker containers for vulnerabilities, see this posting. To learn more about general auditing techniques, see this posting.
48. Use the docker diff command (with no quotes) as one technique for determining the changes the container has had since it was created from an image. To learn more, see this posting.
49. The Center for Internet Security has a number of PDFs that are over 150 pages long on identifying and remediating Docker security issues. To download one for your Docker version, go here.
50. Read about the industry's concerns with container security. The company StackRox released a report on the modern state of security in containers in November 2018. You can view it here.
51. Scan the containers with AquaSecurity's Trivy.
Footnote
(1) (The old PDF was https://www.oreilly.com/webops-perf/free/files/docker-security.pdf.)