This is a copy of Thomas Plunkett's memo to Gawker Staff regarding the security breach that happened to its databases containing commenter's passwords. I am posting it to my blog because it reveals a lot about the current security implementation in Web 2.0 sites. I have highlighted some parts which I think are important. Except for the highlights, I have not modified it.
First, we never planned for such an event, and therefore had no systems, or processes in place to adequately respond. Our focus as a team (and company) has been on moving forward. This put up blinders on several fronts. As a result, numerous wrong decisions were made by me this past weekend in responding to the security breach.
Further, attention to completed work is every bit as important as attention to upcoming work. Our development efforts have been focused on new product while committing relatively little time to reviewing past work. This is often a fatal mistake in software development and was central to this vulnerability.
Finally, we have not only seen tremendous growth as a company, we have never been afraid to take an unpopular or controversial stance with regard to individuals or organizations. Let’s face it: we draw the ire of many. This creates a unique set of demands to meet rapid growth as well as threats that often specifically target us. We did not establish standards and practices to handle growth and the fact that we have a target on our back.
On several fronts — technically, as well as customer support and communication — we found ourselves unprepared to handle this eventuality. The tech team should have been better prepared, committed more time to perform thorough audits, and grown our team’s technical expertise to meet our specific business needs. As a result of not having done these things, we have not adhered to standards expected of us, and our response was inadequate. The remedy to this situation will not be immediate, but it will be swift as possible.
Current Activity: Regaining Control
The tech team have moved our operation to the third floor of the Gawker Media office in order to focus on the work that needs to be done. We are currently in the process of performing a complete review of what happened with an independent security firm.
Here’s what we’ve done so far to regain control:
We have been able to establish a fairly complete timeline of intrusion activity, and have identified compromised assets within Gawker. We have re-established control of compromised systems including ourGoogle Apps accounts. As a result, you will have to reconfigure your Google Apps access (more on this below).
In addition, we have addressed all known vulnerabilities and will continue auditing our system for security flaws, and we have made appropriate changes to administrative accounts to our web and application infrastructure. There are many people reviewing our code base, and because of this, we will also reach out to members of the technical community to harness their expertise. This process will continue as we move to an entirely new, hardened web infrastructure.
We have introduced a help desk to address commenter concerns related to the breach. This will continue to exist as long as it is needed. Scott, Greg, Jeremy, Nick and a host of interns, and many of you, have been active in the threads, and communicating as much as possible as we work through this event.
Moving Forward
We’ve learned many lessons from this experience, both as a tech team, as a company, and as individuals. If there’s one lesson nearly all of us learned, it’s that we can and must be smarter with passwords. Lifehacker is a great resource for password advice (and there are many others). I suggest you start here: http://lifehacker.com/184773/geek-to-live–choose-and-remember-great-passwords.
Efffective immediately, we have enabled SSL, a more secure method of communicating over the internet, for all users with Gawker Media accounts on Google Apps (this does not affect your personal Gmail). Those of you not using web-based Gmail will have to reconfigure your clients (this includes any desktop mail client as well as other devices). The attached document provides instructions to make this easier, and includes information to configure different devices including iPhone, Android and Blackberry phones.
Also effective immediately: If you require access to sensitive materials (legal, financial, or accounting documents) on Google Docs, you must have two-factor authentication setup on your account. No documents will be shared with personal Gmail accounts. We are also strongly encouraging all staff to setup two-factor authorization even if you do not require access to sensitive material.
We will enforce a policy that sensitive information not be posted to the editor wiki. This policy will also apply to chat communications (e.g., Campfire, AIM).
On all of our sites, we will be introducing several new features to our commenting system to acknowledge the reality that we have lost the commenters’ trust and don’t deserve it back. We should not be in the business of collecting and storing personal information, and our objective is to migrate our platform away from any personal data dependencies (like email & password). We will push further integration of external account verification sources using OAuth (like Facebook, Twitter, and Google) for those that want to use them, and we’ll also be introducing disposable accounts. Disposable accounts are similar to the service a pre-paid phone offers to drug dealers (a disposable, untraceable communication device). Commenters seeking anonymity will be able to do so confident that when necessary they can simply toss out the account and there will be no connection to the individual. They will work like this:
- no password will be stored
- no email will be stored
- account can be used as long as you have the key code; lose or delete it, the account is abandoned.
In addition, we are establishing a public Gawker Tech & Product blog (a long time coming) from which we will communicate product information as well as product plans to our readers. You can expect to see it by early next week.
This has been a very unfortunate event in Gawker Media history, and we have learned much from it. Above all, this has been an enormous inconvenience for everyone affected, and for this I apologize. You can expect a much more responsive and proactive technology and product team for 2011. You can also expect a much more public me — if there is one critical thing that has been missing, it is a lack of consistent communication from me. That will change.
Regards,
Tom Plunkett
From: Thomas PlunkettIt is clear that the Gawker tech team did not adequately secure our platform from an attack of this nature. We were also not prepared to respond when it was necessary. These things can be attributed to several factors.
Subject: The Gawker Media security breach — status and moving forward
To: [Gawker staff]
Date: Friday, December 17, 2010, 4:43 PM
Everyone -
As you know, this has been the Gawker tech team’s most difficult week ever. This note has been too long coming, but the following is meant to communicate several things: what happened, our current activities, and our plans for moving forward. I suggest you read all of this as I am making several recommendations below, and we are implementing some changes that will affect all of you.
What Happened
Gawker Media servers and some company email accounts were compromised by hackers at some time during the last few weeks; the compromise was made public to us (and everyone else) this past weekend. In recent weeks, intruders were able to gain access to our web servers by exploiting a vulnerability in our source code, allowing them to gain access to user data and passwords. With this information, they were able to gain access to the editor wiki, some Gawker Media email accounts, and other external resources.
First, we never planned for such an event, and therefore had no systems, or processes in place to adequately respond. Our focus as a team (and company) has been on moving forward. This put up blinders on several fronts. As a result, numerous wrong decisions were made by me this past weekend in responding to the security breach.
Further, attention to completed work is every bit as important as attention to upcoming work. Our development efforts have been focused on new product while committing relatively little time to reviewing past work. This is often a fatal mistake in software development and was central to this vulnerability.
Finally, we have not only seen tremendous growth as a company, we have never been afraid to take an unpopular or controversial stance with regard to individuals or organizations. Let’s face it: we draw the ire of many. This creates a unique set of demands to meet rapid growth as well as threats that often specifically target us. We did not establish standards and practices to handle growth and the fact that we have a target on our back.
On several fronts — technically, as well as customer support and communication — we found ourselves unprepared to handle this eventuality. The tech team should have been better prepared, committed more time to perform thorough audits, and grown our team’s technical expertise to meet our specific business needs. As a result of not having done these things, we have not adhered to standards expected of us, and our response was inadequate. The remedy to this situation will not be immediate, but it will be swift as possible.
Current Activity: Regaining Control
The tech team have moved our operation to the third floor of the Gawker Media office in order to focus on the work that needs to be done. We are currently in the process of performing a complete review of what happened with an independent security firm.
Here’s what we’ve done so far to regain control:
We have been able to establish a fairly complete timeline of intrusion activity, and have identified compromised assets within Gawker. We have re-established control of compromised systems including ourGoogle Apps accounts. As a result, you will have to reconfigure your Google Apps access (more on this below).
In addition, we have addressed all known vulnerabilities and will continue auditing our system for security flaws, and we have made appropriate changes to administrative accounts to our web and application infrastructure. There are many people reviewing our code base, and because of this, we will also reach out to members of the technical community to harness their expertise. This process will continue as we move to an entirely new, hardened web infrastructure.
We have introduced a help desk to address commenter concerns related to the breach. This will continue to exist as long as it is needed. Scott, Greg, Jeremy, Nick and a host of interns, and many of you, have been active in the threads, and communicating as much as possible as we work through this event.
Moving Forward
We’ve learned many lessons from this experience, both as a tech team, as a company, and as individuals. If there’s one lesson nearly all of us learned, it’s that we can and must be smarter with passwords. Lifehacker is a great resource for password advice (and there are many others). I suggest you start here: http://lifehacker.com/184773/geek-to-live–choose-and-remember-great-passwords.
Efffective immediately, we have enabled SSL, a more secure method of communicating over the internet, for all users with Gawker Media accounts on Google Apps (this does not affect your personal Gmail). Those of you not using web-based Gmail will have to reconfigure your clients (this includes any desktop mail client as well as other devices). The attached document provides instructions to make this easier, and includes information to configure different devices including iPhone, Android and Blackberry phones.
Also effective immediately: If you require access to sensitive materials (legal, financial, or accounting documents) on Google Docs, you must have two-factor authentication setup on your account. No documents will be shared with personal Gmail accounts. We are also strongly encouraging all staff to setup two-factor authorization even if you do not require access to sensitive material.
We will enforce a policy that sensitive information not be posted to the editor wiki. This policy will also apply to chat communications (e.g., Campfire, AIM).
On all of our sites, we will be introducing several new features to our commenting system to acknowledge the reality that we have lost the commenters’ trust and don’t deserve it back. We should not be in the business of collecting and storing personal information, and our objective is to migrate our platform away from any personal data dependencies (like email & password). We will push further integration of external account verification sources using OAuth (like Facebook, Twitter, and Google) for those that want to use them, and we’ll also be introducing disposable accounts. Disposable accounts are similar to the service a pre-paid phone offers to drug dealers (a disposable, untraceable communication device). Commenters seeking anonymity will be able to do so confident that when necessary they can simply toss out the account and there will be no connection to the individual. They will work like this:
- no password will be stored
- no email will be stored
- account can be used as long as you have the key code; lose or delete it, the account is abandoned.
In addition, we are establishing a public Gawker Tech & Product blog (a long time coming) from which we will communicate product information as well as product plans to our readers. You can expect to see it by early next week.
This has been a very unfortunate event in Gawker Media history, and we have learned much from it. Above all, this has been an enormous inconvenience for everyone affected, and for this I apologize. You can expect a much more responsive and proactive technology and product team for 2011. You can also expect a much more public me — if there is one critical thing that has been missing, it is a lack of consistent communication from me. That will change.
Regards,
Tom Plunkett
Comments