Previously, I looked at the current threats to Kubernetes and container-based virtualized environments, the methods attackers are now utilizing, and the need for real-time automated security strategies to safeguard dynamic container deployments. This follow-up will cover how to think through your Kubernetes security in production.
First, prepare Kubernetes worker nodes ahead of production.
It’s critically important to lock down the host systems for Kubernetes worker nodes before deploying application containers. To do so, enact these pre-deployment security measures:
- Use namespaces
- Restrict Linux capabilities
- Enable SELinux
- Use Seccomp
- Configure Cgroups
- Use R/O Mounts
- Use a minimal Host OS
- Run CIS Benchmark security to test your environment for best practices
With containers running in production, security measures must be in place to cover three key areas: network filtering, container inspection, and host security. Let’s look at each.
Utilizing a container firewall to inspect and secure network traffic is similar to using a traditional firewall, instead with tools designed for the cloud-native Kubernetes environment. Several different options are available for introducing a firewall to safeguard a container network, including the following three techniques.
Layer 3-4 filtering of IP addresses and ports. This network filtering technique uses Kubernetes network policy to dynamically update rules as deployments change and scale. While these simple network segmentation rules are insufficient for achieving the full monitoring, logging, and threat detection capabilities required to safeguard business-critical container deployments, this method offers some ability to block unauthorized connections.
Web application firewall (WAF). This firewall can safeguard web-facing containers in the same manner as it protects web applications — by detecting external attacks over HTTP. However, it is unable to provide the multi-protocol filtering that can be necessary to secure internal traffic.
Layer-7 container firewall. This approach uses network application protocol whitelists to secure containers with Layer 7 filtering and deep packet inspection (DPI) of traffic between pods. It also features built-in capabilities to detect DDoS, DNS, and SQL injection, as well as other common network-based application attacks. This firewall can also offer container process monitoring and host security. Because most attacks arrive from predictable vectors — HTTP requests with malicious code hidden in the header, executable shell commands in XML objects, etc. — Layer 7 DPI is important for detecting the hallmarks of these attacks and determining if each pod connection is safe and should be allowed or denied.
With traditional tools unable to keep pace with the realities of containerized environments, implementations used to secure these environments must be both container and Kubernetes aware, and designed for their dynamic nature and complexity.
The ability to inspect container processes and activity in the file system (and to recognize suspicious processes) is a key component of container security. Attacks often rely on privilege escalations, malicious processes such as port scanning and reverse shells, or vulnerabilities in the Linux kernel, packages, libraries, or applications. The suspicious activity these methods produce within a container must be detected. At the same time, it’s valuable to develop a baseline of container activity through behavioral learning in order to recognize processes that deviate from normal activity.
Another best practice here is to utilize containerized applications that are built on microservices principles and feature reduced attack surfaces. By using applications with streamlined functions and containers that use only those packages and libraries that are required, it becomes much simpler to detect suspicious behavior.
Securing the host, or Kubernetes worker node, is absolutely crucial. If you allow the host that containers run on to become compromised, it can lead to an entire list of negative consequences that enable an attacker to:
- Escalate to root privileges,
- Access application or infrastructure controls,
- Change cluster admin privileges,
- Damage or hijack host resources,
- Stop the API Server, Docker daemon, or other critical orchestration tool infrastructure,
- Or start malicious processes.
To avoid these threats, the host system must be protected by the same monitoring and security functions used to secure containers. Implementing effective network filtering, container inspection, and host security offers the best capabilities to detect and thwart attacks and kill chains at multiple points.
The Kubernetes system and resources, as well as management platforms built on top of it, present an attack surface as well – one which can and will be exploited if not secured. It’s important to properly configure and review role-based access controls (RBAC) for system resources, in order to protect them against attacks. This requires:
- Safeguarding the API Server by configuring RBAC (or creating manual firewall rules to block unauthorized access).
- Securing Kubelets by configuring RBAC and managing certificate rotation.
- Securing all external ports. Review these ports to remove any that are unnecessary, and require authentication for those required; where non-authenticated services are used, restrict access to whitelisted sources.
- Limiting console access and ensuring that access requires a user login with strong passwords or two-factor authentication.
With host security in place to also protect worker nodes, these methods are effective in securing Kubernetes deployment infrastructure. As an additional measure, access to infrastructure services should be monitored for unauthorized connection attempts.
Kubernetes Auditing and Compliance
Kubernetes and containers remain very rapidly growing and changing technologies as their popularity continues to rise. This means that enterprises will constantly update their container environments as these technologies evolve. In order to maintain security throughout these changes, enterprises should get in the habit of performing regular security tests, as well as recertification in specific compliance standards as necessary.
Both the CIS Benchmarks for Kubernetes and Docker Bench for Security tests offer comprehensive checks of Kubernetes and Docker security, respectively, and should be run as a regular and automated process. The tests cover host security, Kubernetes security, Docker daemon security, container security, RBAC configurations, and data security (both data at-rest and in-transit).
Further, vulnerability scanning should be executed to ensure images and containers in registries are compliant and free of known exploits.
Open Source Tools Available for Kubernetes Security
While more robust protections are recommended for business-critical deployments, several open source projects are available to help enterprises secure their Kubernetes environments. These include:
- Kubernetes Network Policies offer automated segmentation by IP address.
- Istio offers a service mesh for service to service communication. While not a purpose-built security tool, it offers routing, authentication, and encryption capabilities.
- Grafeas provides an artifact metadata API to audit and govern the software supply chain.
- Clair enables vulnerability scanning and analysis for container images.
By implementing a layered strategy that employs run-time Kubernetes security automation and making use of the latest strategies and best practices, enterprises should feel confident in deploying Kubernetes with a higher degree of safety and confidence.