To that end, Mesosphere today announced partnerships with three software vendors that have a hand in real-time applications: Confluent, DataStax, and Lightbend. The three companies‘ applications are now supported on DC/OS, Mesosphere’s data center operating system.
As containers mature, they’re being used for more complex tasks — distributed applications, in particular. Originally, a container was meant to hold one application and its dependencies, such as libraries. Now, developers are interested in developing distributed applications that will run on multiple containers spread across multiple machines, says Tobi Knaup, Mesosphere’s CTO.
If you want to pin the trend on one type of application, you could pick big data.
Big data has become a real-time endeavor, and that means its applications need to be stateful. “It’s no longer the traditional big data where we collect the data and once a week, or once a month, we analyze the data to make business decisions,” Knaup says.
This is where the new partnerships come in. Confluent, whose founders created the Kafka open source messaging system, helps operators make real-time decisions about data streams. DataStax is a data management platform built on the Cassandra open source database. And Lightbend, which was already a Mesosphere partner, offers the Reactive development platform for building stateful applications.
Containers are stateless. That is, they lack knowledge of real-time events; when you replace a container, the new model doesn’t “remember” the old model’s network connections or the status of its data flows.
There’s nothing inherently wrong with that, but as containers have caught on, they’re being eyed for applications that need to keep track of state — databases, for instance.
That’s why you’re seeing more efforts to create stateful containers. The 1.3 version of container management platform Kubernetes includes stateful-container support. Startups tackling container storage are mindful of state. And ClusterHQ was on this idea in 2014 with its Flocker volume manager.
Mesosphere would like to be at the forefront of this movement, because DC/OS has an inherent way to support statefulness. Mesos, the distributed kernel at the heart of the operating system, uses a two-level scheduler. That is, it has an ordinary scheduler, atop which developers add application-specific schedulers; the latter can include the logic for handling stateful recovery of an application.
Mesosphere’s to-do list includes the creation of a software development kit (SDK) so that software vendors can create these schedulers for their own applications, Knaup says.