Founder's blog

We had Downtime Today

Feb 27 2010 :: by Alex
The short version: Jitbit's email-sending server was down for more than 5 hours. This was due to a technical error. We screwed up. I apologize to all the affected users, sorry for all the inconvenience caused. All unsent order confirmations and license information were resent manually. Most of the support cases you sent us - have reached us, but some may have been lost.

The long version: .NET framework has a bug that they have been promising to fix for 3 years now. The bug is - the .NET Framework's SmtpClient class never closes its connection to the SMTP-server. After hours of email-intensive work your server has thousands of open TCP-connections to the email server. That leads to another problem called "too many open connections" causing your server to response slowly. The workaround is to lower the idle timeout for the connection so it closes automatically. But the workaround sometimes does not work if the timeout is configured BEFORE you set up a host for the SmtpObject...

Anyway. Our server has stopped sending out email notifications. We have not tested the "workaround" before deploying it to our server.

Now Patrick McKenzie has a great list of standard industry responses to outages:
  1. "Outage? What outage?"

  2. "Please see our status page, which we’ve conveniently located in electronic Siberia."

  3. "ATTENTION ALL USERS! 0.7% of you were affected by some very serious sounding things yesterday! Please be worried unnecessarily even if you weren’t affected, and swamp our support line, who we will provide no effective tools to tell you whether you’ve been affected or not!"

We admit the error and apologize to you. Sorry. We screwed up. But we're still here to help in case you need us.

'We had Downtime Today' was written by Alex by Alex. CEO, founder


comments