Security and monitoring startup Corelight announced a new integration with big data processing company Databricks. Databricks brings its analytics capabilities and intelligence to enhance Corelight’s ability to detect and remediate threats.

Both of the companies’ technologies are based on open source frameworks. Corelight is built on the open source network monitoring framework Bro — it was also founded by the creators of Bro. Similarly, Databricks was founded by the creators of Apache Spark, and its technology is based on Spark’s open source cluster computing framework. Databricks had been promoting combining Apache and Bro for a few months, but this is the first time the two companies' commercial products will combine.

“Developing software using open source principles has been proven to be a powerful way to harness the knowledge of a wide range of developers around the world, to develop more robust software more quickly than doing it in a proprietary one-company effort,” said Corelight CMO Alan Saldich.

Corelight Sensor is the company’s main product and is the enterprise version of Bro. It investigates and prevents security threats. Enterprises can more easily deploy and scale this software than they can on the open source counterpart. Also compared to Bro, Corelight Sensors can offer 10 times the throughput of monitored traffic and the ability to monitor higher bandwidth links. Bro produces BroLogs that contain data that describes network traffic specifically for security operations teams, incident responders, and threat hunters.

It already integrates with monitoring tools from Splunk, Amazon, and Kafta and has performance optimizations that provide four times higher data processing throughput than standard servers.

However, it was missing an analytics capability wrote Saldich and Brian Dirking, Databricks senior director partner of marketing, in a blog post announcing the integration. Saldich and Dirking wrote that “The increasing volume and complexity of threats require security teams to capture and mine mountains of data in order to avoid a breach.”

And while Corelight has integrations with other companies, Saldich said that “Databricks is different because of its underlying technology (Apache Spark). Spark is not strictly a security solution, but its powerful real-time analytics capabilities are a great match for the needs of security teams.”

Databricks Unified Analytics Platform assists developers who want to build and deploy analytics and machine learning (ML) models in real-time. It processes streaming and historic data that can then be used by security teams to analyze, review, and profile threats. It integrates with numerous types of enterprise data including SIEM logs, cloud logs, system security logs, threat feeds, and Bro logs.

The product integration enables Databricks to incorporate data from the Corelight Sensors into its Spark analytics stack. Combining the two commercial versions of the open source technology makes it easier for customers to deploy, use, and maintain the systems, said Saldich.

With this new integration, Corelight’s appliances, Corelight Sensors, can now be configured to send their data to their Databricks cluster for analysis.