when you get pwned

If your site’s been whacked, I hope this post is helpful. Taking your own action while waiting for your provider or someone else to help can offer some peace-of-mind.

Get Rid of the Malware

NOTE: This guide presumes the attack (that’s what it is) took place on a Linux-based server and you have just enough fluency at the command line to be dangerous.

If you’ve been attacked on a Windows Server system, the damage is likely more widespread than we are going to have time in this short and sweet post. There are, however, some great guides on cleaning up extensive Windows Server attacks.

However the criminal(s) got into your server, once we discover the attack, the first thing we want to do is get rid of the tool they most likely used to get in. We want to make sure it isn’t as easy for them to use it to get in again. And again.

To do this, we’ll go through and verify every file and folder (sigh) associated with the compromised user account and get rid of everything that doesn’t belong there.

From the command line (CLI) in the root of your webserver’s home directory, do:

ls -a

This will show all files, including hidden ones. Criminals often hide the files they use as tools using a . in front to hide the file from less experienced users. It is far easier to detect this via the CLI than the desktop interface (GUI).

The first thing we’ll look for is files with timestamps that show changes have been made recently or around the time the attack took place. Once we identify a file or files modified in the attack (such as a messed up home or index page), we can identify files used to modify the hacked file by searching the HTTP logs using those timestamps. Timestamps are key to helping us understand exactly when the attack took place.

Here’s one way to do this:

[yourservername]$ find /yourhomedirectory/yourpublicwebsitefolder/ ! -name "log" -mtime -3

The “-3” equals the number of days back to look for files modified within this time frame. This command lists all files under /yourhomedirectory/yourpublicwebsitefolder that have been modified in the past 3 days. We can certainly change this to 4 or 5 or 6 or 7 but be aware that this will greatly increase the amount of files and folders that show up in our search. Getting as close to the time as possible will make our work easier!

Website access logs (HTTP logs) are awesome for helping us isolate this information. They are typically located outside the public website directory, in a place like this:

/yourhomedirectory/logs/websitename/http

Is the Database borked?

SQL injection attacks are especially effective against vulnerable servers, as much or more than XSS. These attacks often result in altering databases with malicious code. Such modifications allow criminals back in even after identifying and removing affected files in the steps above.

This is why it’s essential after an attack to verify the database like we checked our files above to see if anything has changed that shouldn’t have.

Figuring out when the attack occurred is important for this reason. When we know when it occurred, we can even quickly revert the database back to a prior time using backups. That’s ideal but not always viable because maybe a backup is corrupted, etc.

It bears repeating: keep comprehensive backups in good, working order.

Restoring Order to the Universe

If important files were modified or deleted in the attack, this can be much more involved. Most hosting providers provide tools we can use to restore from their dashboard panels but some are sneaky and charge extra or require you to set it up yourself. Best to know the answers to these questions before you really need them. Check your hosting provider for specific details on available tools and steps.

Once we’ve restored working files, folders and domains, we can begin again, directory-by-directory, from the top of this guide to make sure everything is in order.

There are no shortcuts so be prepared to do some work but also be prepared to celebrate that you understand your systems better and therefore can then defend them better and respond faster next time something unplanned (and unwanted) happens.

Oh, no you don’t!

Criminals hack to gain and maintain access to systems for their use today and/or tomorrow based on their needs and intent. We can always count on at least one back doors having been installed and hidden so that the attacker can bend the system to their will later on when needed. As long as these back doors remain on our system, lying in wait, our site or service will remain vulnerable.

Where do we look? What can we do about it?

  • rogue users – Check and make sure there are no unauthorized users. These could be web access, shell or database users. Verify all users and delete any that aren’t authorized or required. Keep in mind some hosting providers create users with weird usernames for backups, etc. Don’t delete one that does something important that you may not be aware of!
  • authenticated cookies – Websites remember users by storing a small text file (cookie) on the user’s computer. Criminals often create and/or modify a valid user account and then force an authenticated cookie to their browser. That cookie can be reused until it expires even if the password for that user is changed. This is another reason why it’s important to modify HTTP headers to include expiration values for cookies. These cookies can be reset, forcing every user to log in again. There are many ways to do this, depending on your hosting provider and your web server’s configuration.
  • installing remote shell – It’s trivial for a criminal to install a file that gives them complete access anytime they want. Files like these can be written in any object-oriented programming language, such as PHP or HTML. They are often given legitimate names to look like necessary files and sometimes contain obscured code to appear authentic to less-discriminating eyes. They can also be appended to legitimate files. The extra code enables functionality that’s only available when an extra parameter or two is passed to the file during the request that only the attacker who put it there knows about.
  • changing permissions on stuff – This is the oldest trick in the book but still works quite well. Criminals will often set permissions on a publicly-addressable directory or folder so that they can read and write to it. This means they can copy files to the folder, delete files, whatever they want. Not awesome. We will want to be sure and do a permissions audit to make sure no other folders are still accessible to unauthorized users.
  • only using SSH – Sure it’s true that many people are still using FTP to transfer files back and forth. FTP sends usernames and passwords “in plain text” which means the connection strings (handshakes between computers) include this information and it’s really, really easy for others to obtain it. Using SSH/SFTP is more secure. Most up-to-date FTP clients support these protocols and in most cases doing so doesn’t change the workflow in any way outside of a configuration change. Most of us won’t even know the difference, honestly. Using a secure and encrypted method to move files back and forth is recommended.

Wrap Up – CELEBRATE (then document!)

Obviously, there is no single-click, easy peasy tool or method for tending to all this schwack, from isolating timestamps and indicators-of-compromise (IoCs), removing back doors, and cleaning up after a site has been whacked. It definitely takes time, experience and knowledge to investigate and isolate how attackers got in, what they added, deleted, or changed and the what exactly occurred.

The single best thing we can do after an attack has been remediated is to document it in as much detail and accuracy as we are able and share a post-mortem if it’s more than just you. Share this internally so everyone has a chance to better understand what happened and what can be done to help prevent it from happening again. Add any important insights to your organization’s incident response plan so responding to it next time is as painless as possible. Because it’s not likely it’ll never happen again.

Last but not least: C E L E B R A T E . This victory may seem small but it’s no small thing. While it sucks to get pwned, the one upside is how much stronger our overall resilience engineering skills grow from each one. Through exercises like these we grow our overall understanding of our operations, how the technology works, more visibility into the hidden machinery, while amplifying resilience and response methods.

I often think of this stuff like playing hide and seek. Criminals find great hiding spots once but, once we discover it, they have to work hard to find another one.

So go celebrate with your team. You’re all a whole lot smarter than you were before and have taken the first step into learning to be prepared for a crisis. Good luck out there!