Last month during a random Friday morning the Keystroke team started noticing email services behaving erratically. It wasn't long before we noticed that all our websites were timing out, as well. Our Internet was working fine, but our email and production websites were inaccessible. It took about 30-minutes before we discovered our data center was under attack.
A DDOS attack.
In the 25-years Keystroke has been around, this is only the second time we've been hit like this, and the first time we received an actual ransom demand. Thankfully our team is much bigger and more talented this time around, and we have a few more resources at our disposal than we did 8-years ago during the last attack, but ti was still a learning opportunity. Since most of the people reading this won't have some fo the technical talent at their disposal as I did, I thought I'd share some lessons learned:
- it costs the hackers approximately $30/hr to run DDOS attacks, so they often cannot afford to run them continuously or indefinitely
- you will get a ransom demand email (we got several). Do NOT pay or respond to them. See my first point. If they don't get a response from you, they will move on to another target. If you reply, you'll become a mark, and motivate them to invest more in the attack
- Services like Azure already use Checkpoint.com, so they have built-in defenses against DDOS attacks and blunt or negate their impact. Migrating to them provides you scalabaility and security over a Co-Lo setup in a data centre.
- Have your DNS servers spread across 2-3 locations, and replicate accordingly. It goes without saying that if your DNS servers get overwhelmed, it won't matter where your site are located, they'll go down. Having DNS replication setup across several servers in several locations provide you DNS fault tolerance at relatively low costs.
- We also found out Tucows sorts DNS requests alphabteically, so even if you register a particular server as first, it may be bumped by the secondary DNS server if it starts with a higher letter (that one amazed us!)
- Be sure to have a firewall that can be supported by services like Checkpoint so they can intervene faster.
Some of the people reading this were effected by this outage, and for that we apologise. We've migrated all our production services to Azure to reduce our future vulnerability, which isn't the most economical hosting method, but certainly amongst the safest.
