Cloudflare accidentally took down its own tenant service offering following a botched software update.

The incident, first spotted by Daniel Lockyer on X, was caused by a bug in the provider's dashboard, which led to continuous, redundant calls to the firm's tenant service API, which automates the provisioning and management of Cloudflare services – resulting in the firm effectively DDoS’ing itself.

“The API calls were managed by a React useEffect hook, but we mistakenly included a problematic object in its dependency array,” company engineers explained in a blog post. “Because this object was recreated on every state or prop change, React treated it as ‘always new,’ causing the useEffect to re-run each time.

“As a result, the API call executed many times during a single dashboard render instead of just once. This behavior coincided with a service update to the tenant service API, compounding instability and ultimately overwhelming the service, which then failed to recover.”

Once Cloudflare’s tenant service became overloaded, it had a cascading effect, impacting other APIs and the dashboard.

How the outage unfolded

At 4:32 pm on September 12, Cloudflare’s latest dashboard update went live, which contained the bug. The vendor then launched the newest iteration of its tenant API service at 5:50 pm, and within just seven minutes, became overwhelmed.

By 6:20 pm, the tenant API service availability rose back up to 98% after more resources were provided, but the dashboard offerings failed to recover.

Just before 7 pm, error codepaths were removed and a new version of the tenant service was released – in what Cloudflare would describe as “ultimately a bad change” as it further impacted its APIs again.

The problematic changes were then reverted by 7:12 pm. While the whole incident lasted just three hours and didn’t affect the firm’s network, it showed just how a botched software update can quickly turn sour.

Cloudflare apologized for the outage, with engineering VPs Tom Lianza and Joaquin Madruga writing in a company blog post that the firm’s automatic alerting service helped to kickstart remediation at speed.

“We believe it’s important to learn from our mistakes, and this incident is an opportunity to make some improvements,” the pair wrote. “Those improvements can be categorized as either ways to reduce/eliminate the impact of a similar change or as improvements to our observability tooling to better inform the team during future events.”

Cloudflare’s effective self-DDoS comes as genuine attacks are becoming larger and increasingly more complex. A Nexusguard report suggests that DDoS attacks are soaring in size, surging by 69%, with peak sizes reaching 962.2 gigabits per second (Gb/s).

Cloudflare itself has noticed an ever-growing number of larger attacks, with the firm reporting an attack earlier this month that reached peaks of 51 billion packets per second (PPS) and 11.5 Tb/s.

That mammoth attack surpassed Cloudflare's previous record of 7.3 Tb/s, set only a few months earlier. The vendor had also reported attacks reaching 6.5 Tb/s and 4.8 billion packets per second in April, in a worrying trend of ever-increasing DDoS attacks.