Cloudflare has published a post-mortem on this week’s global outage that saw the likes of ChatGPT, X, and even SDxCentral go offline for several hours – with a database issue identified as the cause of the downtime.

In a detailed blog, co-founder and CEO of Cloudflare Matthew Prince explained the failover – the company’s worst since 2019 – was the result of a change to permissions in ClickHouse, one of Cloudflare’s database systems. This caused the database in question to output multiple entries into a ‘feature file’ used by the firm’s Bot Management system, causing it to double in size, with Cloudflare’s servers “panicking” in response.

Amongst features of the Bot system is a machine learning (ML) model that generates bot scores for every request traversing Cloudflare’s network, which customers use to control the bots’ permission access settings.

A feature configuration file is input into the ML model, representing an individual trait used by the model to predict whether or not a request was automated or not. The file is then refreshed every few minutes and published to Cloudflare’s entire network to help react to traffic flow variations.

“It allows us to react to new types of bots and new bot attacks. So it’s critical that it is rolled out frequently and rapidly as bad actors change their tactics quickly,” explained Prince. “A change in our underlying ClickHouse query behaviour … that generates this file caused it to have a large number of duplicate ‘feature’ rows. This changed the size of the previously fixed-size feature configuration file, causing the bots module to trigger an error.

“As a result, HTTP 5xx error codes were returned by the core proxy system that handles traffic processing for our customers, for any traffic that depended on the bots module.”

The CEO explained that ClickHouse database clusters store data on underlying distributed tables, with queries to the tables run through a shared system account. On Monday (November 17), though, Cloudflare allowed explicit rather than implicit access to help users see the metadata of these tables.

“The change explained above resulted in all users accessing accurate metadata about tables they have access to … [the query then] started returning ‘duplicates’ of columns because those were for underlying tables.”

Prince explained this query was one performed by the Bot Management feature file generation logic to enable each input feature for the file, causing it to run amok and generate every five minutes.

“The Bot Management system has a limit on the number of machine learning features that can be used at runtime. Currently, that limit is set to 200 … When the bad file with more than 200 features was propagated to our servers, this limit was hit – resulting in the system panicking.”

Hence the crash, which likely affected some 20% of websites globally which are thought to use Cloudflare services in some form.

Initially, Cloudflare engineers thought an attack was to blame, specifically the Aisuru botnet behind a massive 15.72 Tb/s attack revealed by Microsoft this week, which the hyperscaler dubbed the largest DDoS attack ever observed in the cloud.

Prince wrote that to prevent similar mistakes, his firm was hardening the ingestion of Cloudflare-generated configuration files, as it does for user-generated input, while enabling extra global kill switches for features. Additionally, Cloudflare is removing the potential for core dumps or other error reports to overwhelm system resources while reviewing failure modes for error conditions across all core proxy modules.

The Cloudflare outage came one day after the firm acquired AI deployment startup Replicate, which focuses on the streamlined creation of AI agents and models.

In a recent SDxCentral deep dive on agentic AI, experts agreed that if AI agents take actions on the network similar to the Cloudflare ML snafu, then safety guards will be needed to provide any accidental outages should agents take the wrong action.

When contacted for comment on this issue by SDxCentral, Cloudflare referred back to Prince's blog in response to questions.