Netflix originally used the Apache Mesos container resource manager because of its fine-grained resource allocation when working with Amazon’s Elastic Compute Cloud (EC2) instances. An EC2 instance is a virtual server for running applications on Amazon Web Services (AWS).
In a corporate blog, Netflix gave an overview of three of its projects using Apache Mesos.
1. Mantis. Netflix created its project — Mantis — for its engineering teams to get access to real-time events and build applications on top of them. Mantis uses a master and an agent cluster. “Currently, Mantis is processing event streams of up to 8 million events per second and running hundreds of stream-processing jobs around the clock,” says the blog.
The reactive stream processing platform covers varied use cases including real-time dashboarding, alerting, anomaly detection, and metric generation.
2. Titus. Netflix created the Docker container job management and execution platform Titus. It’s recently been supporting Netflix microservices, especially those that need a consistent local development experience. Titus uses a master to assign resources from Mesos agents. Titus provides integration into the Netflix microservices and AWS ecosystem, including integrations for service discovery, software-based load balancing, monitoring, and Netflix’s CI/CD pipeline, Spinnaker, according to the Netflix blog.
3. Meson. Meson is a general-purpose workflow orchestration and scheduling framework that Netflix built to manage machine learning pipelines. It can handle a variety of jobs with widely different resource requirements for CPU, memory, and disk space. Tasks are resource isolated, and the Meson scheduler evaluates readiness of tasks based on a graph and launches the ready tasks using resource offers from Mesos.
The Netflix blog says the company will continue to engage with the Mesos community through its Netflix’s open source contribution, Fenzo.