I’m the CTO for a technology consulting company and this is the call I got this week: “Our entire AWS account is gone. The call center is down, we can’t log in – it’s like it never existed! How do we get it back?”
One of our former clients, a multimillion dollar services provider, called us in a panic. They had terminated an employee, and in retaliation, that employee shut down their call center capabilities (hosted on Amazon Web Services via AWS Connect). The client was completely locked out and looking for the “undo” button.
After some digging, and a favor from some friends at AWS, we discovered that the former employee had turned everyone off, then changed the email address and password associated with the root AWS account. This locked our client completely out of the account, and since everything was done with the right credentials, AWS couldn’t reverse the damage.
Everything hit at once: they were frantically attempting to log in, and contact AWS, and deal with their entire operation being offline, and figure out exactly what had happened and why.
Their only option was to get the login from the former employee. They tried the nice way first, but by the end of the day the FBI was at his door. Once the account was back in our clients’ hands, they were able to turn the call center back on pretty quickly, but it still cost a full day.
The legal costs, user panic, and productivity loss could have been avoided by following a few best practices.
Here are three precautions you can take to safeguard your company against a security issue like this one:
1. Practice Least Privileges
The idea here is simple – everyone should have exactly the permissions they need and nothing more. Most cloud computing systems allow very fine-grained control of privileges. The Admin or Root account on any system shouldn’t be used for daily work – write the password on a piece of paper, print out the backup MFA codes (more on that below) and stick it in a fireproof safe.
For the truly paranoid: put two safes in two locations.
After that, ensure that two people have enough access to create users and fix permissions – that way, someone can be out sick without grinding the company to a halt.
In this case, 5 people shared an email “group” address and they all knew the password. That user had global access to everything, and when he was burned he decided to burn back.
Create an admin or two, then set up other accounts for your employees with very specific limitations on what they can do.
2. Multi-Factor Authentication
Multi-Factor Authentication (MFA) attaches a secondary authentication to your account (the email and password being the primary). You have likely experienced this when you were texted a code while signing up for something. Turn it on everywhere that you can.
In the book “Tribe of Hackers”, Marcus Carey sent 12 questions to 70 cyber security professionals.
When asked “What is the most important thing your organization can do to improve its security posture?” nearly all of them included requiring MFA wherever possible.
There are many forms of MFA, including text messages, apps on your phone, physical keyfobs, and encrypted thumb drives.
It’s very important to have a backup as well. Most systems will give you a set of “backup codes” which will each work 1 time. You can print them or put them in an encrypted note – but make sure you get them.
The importance of using multi-factor authentication cannot be overstated. Had the company used multi-factor authentication, this ex-employee would have never been able to log into the account and shut it down without them knowing about it.
Turn on Multi-Factor Authentication
3. Offboarding Process
Finally, ensure your company has a secure offboarding process. We encourage our clients to write up an “86 procedure” and review it quarterly.
The goal should be to strip all privileges in 5 minutes or less. When an employee is terminated, they should walk out of the termination meeting with no access and not be allowed back on their laptop.
Today, so many services exist that can become critical to a business’s operation. If you can afford to use something like Okta to manage these services you will have an easy off-button, but if not at least consider using your email provider (Google Apps and Outlook both provide this service).
Create and review an offboarding process.
Ultimately you have to protect your data. A few small steps can go a long way to ensuring one bad actor won’t negatively impact your business.
As exciting as that phone call was, I don’t want to take another one like that again!