Production quality container deployments require several components besides the execution sandbox that renders workloads. While there isn’t a canonical standard set of categories, a representative example can be fashioned by using container-based platform-as-a-service (PaaS) reference architectures, product documentation, and research papers. For example, a paper by IBM researchers offers a logical set of container-building blocks that groups components into two classes: container lifecycle management and cluster management, with 10 total elements.
The container image consists of an archive of a filesystem tree along with all installed libraries and user changes on top of the very basic OS kernel.
Runtime engines like Docker or rkt provide the execution environment for container images. Note that the Open Container Initiative has projects to define standards for both the container image and runtime.
The image repository is akin to a source code version control system that holds image archives ready to be extracted and run.
The OS driver is a set of software drivers that allow the containers to run on different platforms (e.g. Azure, AWS, DigitalOcean) and that includes the appropriate code for the base operating system as well.
The orchestrator manages container deployment, optimizes workload allocation, and sets environmental constraints across a cluster of servers.
A discovery service exposes metadata and API endpoints for containerized microservices and acts as a searchable directory to allow other containers to find and access container images matching particular criteria.
An overlay network is needed — namely, a virtual network akin to those created by hypervisors for VMs, which enables communications between containers on the same host, to outside networks via the physical server NICs, and to containers on other hosts in the same cluster. Networking specifics can vary and are a function of the orchestrator and runtime engine. For example, Kubernetes isn’t prescriptive in how cluster-level virtual overlays are implemented, and supports several techniques including OpenVSwitch with Open Virtual Network (OVN) and a Kubernetes plug-in. Conceptually, nodes within a Kubernetes cluster communicate through VXLAN tunnels, while containerized applications are exposed as standard L3 (VIP and port) services.
State storage is necessary to provide continuity between execution environments and application state for a particular container. Containers are inherently stateless and typically ephemeral; however, many applications need to resume processing using previously calculated results or execution state, or share results across multiple instances running on the same cluster. State storage, usually provided by a database service, provides a mechanism for saving data between execution session and sharing state across a distributed system.
A scheduler is similar to an OS process scheduler and allocates and scales cluster resources for each container instance. The scheduler can also enforce resource constraints or Quality-of-service (QoS) requirements for particular workloads. Container scheduling is typically incorporated into orchestration software such as Kubernetes, Swarm, or ECS.
The resource monitor collects and distributes information about resource usage on a container host or cluster, including parameters like CPU, memory, filesystem, and network usage statistics. Popular cluster-aware monitors include cAdvisor and Heapster.
These components are built on a foundation of physical host hardware and operating systems. The container runtime can be run in a Virtual Machine (VM), which is how AWS ECS on EC2 works, and it’s also possible to run directly on bare metal hardware, which research presented at a 2016 conference shows offers somewhat better performance by eliminating an intermediate abstraction layer.
Container Infrastructure Use Cases
Using the above taxonomy as a guide, the typical container deployment uses three sets of servers: worker nodes to run container images, control nodes to manage clusters and workloads, and ecosystem nodes to provide supporting services like an image repository/registry/discovery, storage, and monitoring.
Like VMs, containers can be used in myriad ways that span core infrastructure to application components. Red Hat offers a useful categorization:
Application containers include core components of an application stack like MongoDB, Redis, and Node js, along with custom microservices of a modular, multi-tier application.
System containers are essentially containerized VMs that combine the versatility of customized OS environments with the convenience and resource efficiency of containers. Since these are containers, they still share the host kernel, but run a full init system (systemd) to allow for the spawning of multiple processes. The LXC and LXD projects from Linux Containers are the most common system containers, and its LXCFS solution provides a corresponding user-space filesystem.
“Pet” containers are fully-packaged monolithic applications or multi-application environments that might traditionally be run on bare metal hardware. Containerizing these legacy systems provides the flexibility of workload orchestration and automation, while meeting the need for a fully customized runtime environment.
Super Privileged Containers (SPC) have elevated system privileges that are required for core infrastructure services like backup, monitoring and security services.
A 2017 Docker user survey by Sysdig shows that the most common container applications include:
- Web proxies and ADCs (NGINX and HAProxy)
- Logging software (fluentd)
- Search engines (Elastic)
- Databases, caches and key value stores (MongoDB, PostgreSQL, Redis, etcd)
- Application platforms and services (JVM, PHP form, Consul)
Containers are also useful for batch data processing and ETL, such as using Hadoop YARN Carriers. Service providers also use containers to host infrastructure services like virtual network services (VNFs).
This article was adapted from SDxCentral’s “Container Infrastructure — What You Need to Know in 2018” report