Spammer attack post-mortem

During the night hours today, approximately between 1 AM and 7 AM UTC, our SaaS Helpdesk service was under a heavy spammer attack. It caused some issues with email deliverability for our Helpdesk customers who use our built-in SMTP server.

How the attack was executed

The attacker has registered a number of trial accounts for our Saas Helpdesk and identified a bug that allowed them to send tens of thousands of emails in a short time. While we do have various limitations applied to "trial" accounts (some functionality is disabled, number of users is limited to 150 etc.) they still managed to forge HTTP requests that kept changing a user account's email over and over again, and then used the "resend welcome email" functionality. Sending almost half a million emails.

Most of those emails were blocked by our outgoing spam filter or deferred (yes, we have an outgoing spam filter just for that). However, the side effect was our email server logs grew up in size tremendously, which caused one of the servers to run out of space.

Attack consequences

Most of the incoming and outgoing emails were delayed, but should start coming through now. Unfortunately, some of the messages sent and received between 4 AM and 7 AM UTC are probably going to be lost. It is hard to say how many due to the nature of the email protocols (no delivery confirmations, etc.).

Remediation

We missed the first half of the attack since it was executed so late in the night. This is not our first nighttime attack - we have multiple alerts set up for everything, but this issue was a little out of the ordinary so we are currently setting up new alerts. Next time we will wake up as the attack happens. The bug that the attacker used was quickly identified and fixed. We are really sorry about this.