Cloudflare Declares ‘Code Orange’ So It Stops Breaking The Internet

It's all about change management.

Outages at content delivery network Cloudflare took down massive parts of the worldwide internet this year not once, but twice. Now, the company is rolling out a plan to make sure it doesn’t happen again.

The first outage, in November, was caused by a config error whereby the company effectively DDOSed itself.

The second error, in December, was related to “issues with Cloudflare Dashboard and related APIs.”

In both cases Cloudflare fixed the problem in a matter of hours, but the effect on customers’ websites dragged on for most of the day.

The company has now shared details of its plans to ensure neither problem reoccurs next year.

Known as “Code Orange: Fail Small,” in line with the company’s brand colour, the project is prioritised over everything else, “allowing teams to work cross-functionally as necessary to get the job done while pausing any other work.”

Cloudflare has only declared a Code Orange once before, in April last year.

The work now going on is organised into three areas: controlled rollouts; improved failure modes for all systems handling network traffic; and changing the internal “break glass” procedures and removing circular dependencies.

‘Serious Gap’ In Change Management

Despite the differences in the November and December incidents, both had a similar root cause: a configuration change deployed to Cloudflare’s datacenters globally, moments before the outage.

“This pattern exposed a serious gap in how we deploy configuration changes at Cloudflare, versus how we release software updates,” the company said.

While Cloudflare releases software updates in a controlled, monitored fashion, configuration changes are deployed instantly, with obvious tradeoffs between speed and risk.

Configuration changes propagate through Cloudflare’s network very quickly thanks to a software component called Quicksilver. While useful, that speed is “rarely necessary,” so the company is introducing controlled deployments to the component now, in a similar fashion to how it rolls out software updates.

Cloudflare releases software updates through the Health Mediated Deployment (HMD) system. Under this, every service-owning team at the company defines the metrics for the deployment’s success or failure, the rollout plan, and the steps to take if it doesn’t succeed.

“Once deployed, our HMD toolkit begins to carefully progress against that plan while monitoring each step before proceeding,” says Cloudflare. “If any step fails, the rollback will automatically begin and the team can be paged if needed.

“By the end of Code Orange, configuration updates will follow this same process. We expect this to allow us to quickly catch the kinds of issues that occurred in these past two incidents long before they become widespread problems.”

Addressing (Inevitable) Service Failures

Even with a better change management process, service outages are still possible. With that in mind, Cloudflare is adopting a policy of assuming failure will occur and handling that failure in “the absolute most reasonable way possible.”

Taking the example of the Bot Management service failure in November, that would have meant adding failover policies at two key interfaces, which would have kept traffic moving – albeit with some limitations – rather than dropping it entirely.

Break Glass In Case Of Emergency

Cloudflare’s “break glass” procedures allow certain individuals to elevate their privilege under specific circumstances, meaning they can perform urgent actions to resolve high severity scenarios.

In the November and December incidents, this failed because security systems stopped employees from accessing the tools they needed to fix the problems. Circular dependencies also slowed work down, as some internal systems became unavailable.

The company says it is reviewing its break glass procedures and technology to make sure staff can access the right tools at the right time, which includes being able to bypass circular dependencies in the event of an incident.

Work is ongoing now and will continue through Q1 and beyond. By the end of March 2026, Cloudflare intends to cover all productions systems with Health Mediated Deployments; update systems to “adhere to proper failure modes”; and put new break glass processes in place.

This article originally appeared on MES Computing’s sister site Computing.