How We Dealt with the Cloudflare Outage

Or: That Time Half the Internet Died and We Didn't

So yesterday Cloudflare decided to take an unscheduled three-hour nap, and took half the internet with it. Fun times.

For context: we use Cloudflare at Jitbit to serve all our static assets - JavaScript, CSS, images, the whole shebang - through cdn.jitbit.com. When CF went down, our assets went down wtih it. And when your assets go down, nothing works, and your app looks like a GeoCities website from 1997.

Attempt #1: The "Just Turn It Off" Approach

Our first brilliant idea was simple: just disable Cloudflare proxying and serve assets directly. Turns out you can't log into the Cloudflare control panel when, you know, Cloudflare is down. Right.

Attempt #2: The API Rescue Mission

Then some absolute legend on Hacker News posted a comment explaining how to use the Cloudflare API to disable proxying if you still had an API key. Thank you, random internet hero.

Tried it. The DNS change wouldn't save. And even if it had worked, we'd still be stuck waiting for DNS propagation, which can take hours for some customeers. Meanwhile, our customers are basically staring at broken websites. Not ideal.

The Nuclear Option (that actually worked)

So we said "screw it" and decided to just move everything. All static assets. To a completely different, non-proxied domain.

Here's where past-us deserves a pat on the back: we had all our asset URLs wrapped in one nice, tidy URL resolver class. One place to change them all. This turned what could have been a nightmare into something merely annoying.

We updated the resolver, configured Nginx for another domain, hugged each other over the upcoming AWS traffic bills and pointed all 1,490 asset links (yes, I counted) to the new non-proxied domain, pushed the changes, recycled the service, and boom - we were back online.

Our customers could use our SaaS Helpdesk while half the internet continued burning for another hour and a half. I'm not saying we're heroes. But I'm also not not saying it.

Lessons Learned

1. Always have a backup plan that doesn't rely on the thing you're trying to fix being working.
2. Encapsulation isn't just a fancy programming concept - it's a "save your butt during an outage" tool.
3. The internet is held together by duct tape and prayers, and occasionally both fail at the same time.

Anyway, if you were affected by yesterday's outage and somehow found this post: we feel your pain. And if you're a Jitbit customer: you're welcome. 😎