Google is layering Kubernetes on top of its fully managed Cloud Dataproc service to ease the management of big data workloads. The move continues a growing focus on using the container orchestration platform to drive more efficiency out of big data.

The Cloud Dataproc for Kubernetes service will see Kubernetes become the de facto management plane for Apache Spark jobs running on YARN and Kubernetes within Google Kubernetes Engine (GKE) clusters. YARN, which amusingly stands for “yet another resource negotiator,” provides open source resource management for Hadoop. It also schedules computing resources in Cloud Dataproc clusters that are based on Google’s Compute Engine (GCE).

Hadoop is a distributed database system designed for interpreting large amounts of data. Spark is focused on processing data in parallel across a cluster that is in-memory, which makes it more efficient.

Christopher Crosbie, product manager for Google Cloud, and Patrick Clay, a software engineer at Google Cloud, explained in a blog post that Cloud Dataproc for Kubernetes provides for greater flexibility to support more predictable workload cycles; uses Kubernetes to provide auto-healing of Spark jobs; and offers a more unified approach to data management.

It can also allow data professionals to move faster and remove the version and library dependencies associated with traditional big data technologies by containerizing and isolating their open source software jobs.

“You can move models and new ETL [extract, transform, load] pipelines from dev to production without having to worry about compatibility,” Crosbie and Clay wrote. “Building on a new agile infrastructure like Kubernetes will make [open source software] easier and faster to upgrade.”

They explained that Cloud Dataproc on Kubernetes eliminates the need for multiple types of clusters that have various sets of software. Those dependencies can be packaged into a single Docker container that allows a user to integrate Spark jobs into software development pipelines.

Matt Aslett, research VP at 451 Research, noted in the blog post that the move “provides customers with a single control plane for deploying and managing Apache Spark jobs on Google Kubernetes Engine in both public cloud and on-premises environments.”

Crosbie and Clay also explained that the Apache Spark support was the “first step in a larger journey to a container-first world,” but “it won’t be the last.” One of those next steps is to integrate the Apache Flink open source community. Flink is a framework and distributed processing engine for stateful computations over unbounded data, which is a stream of data without a programmed end, and bounded data streams, which have a defined start and end.

The Google move is similar to Kubernetes plug-ins developed by other vendors like Robin Systems and Portworx.

Robin Systems last year added a Kubernetes layer to its platform to deal with stateful container applications running on premises or in a public cloud. This allows management using standard Kubernetes, which was originally focused on dealing with stateless container management.

Portworx made a similar move late last year to its PX-Enterprise platform. Michael Ferranti, VP of product marketing at Portworx, explained that its platform was complementary to Dataproc and “customers would use both, both in the cloud and on-prem.”

“Portworx has a lot of customers who run Spark and Hadoop in containers, but managing a Spark or Hadoop cluster can be complex and Dataproc simplifies this,” he noted. “For instance, if you are running a Spark job based off of data in Hadoop with Dataproc, you still need to provide Hadoop with a persistent volume that needs to be managed, encrypted, backed-up, migrated, etc. Dataproc doesn’t provide this storage management, that is what Portworx does.”

Robin Systems and Portworx are also working with Google on its Kubernetes-based Anthos platform.