Facebook has categorically mishandled some extremely sensitive information, our passwords, by allowing them to be written to a plain TEXT FILE on a drive on their internal network. This was reported by Brian Krebs yesterday and subsequently acknowledged by Facebook. In this post, I’ll discuss the magnitude of the fact that they were unaware of this for so long (since 2012), what this says about their internal processes, why the little that Facebook has said about it doesn’t add up, and potential consequences for their users. I go on to explain how an engagement with ActZero may have made this a lot less likely for them.
How could they not know?
I cannot understate the shock I experienced to learn that Facebook didn’t know about this problem for so long. To me, this indicates a clear absence of checks, balances, and audits that would’ve made such an occurrence obvious to anybody looking for it. Facebook claims to have discovered this in January of this year in a “routine security review” – which begs the question, how many such “routine” reviews have occurred since 2012 (the earliest instance of the problem according to Krebs’ senior Facebook source), and why didn’t they uncover this grievous error?
Process Issues Are Requisite for This to Happen
As some of you are aware, I developed the ActZero Platform, so I know a thing or two about software development. The burning question for me is, how do developers have access to production data at all, let alone passwords, in the first place?
The fact that this was able to be written to an open text file means this information could be anywhere – Facebook’s assurances that “these passwords were never visible to anybody outside of Facebook” or that they have found “no evidence that anybody internally abused or improperly accessed them” don’t add up. The very admission that such information left a production environment makes such explanations questionable because the passwords could easily have been written elsewhere, before arriving at their final “secure” location. Any logs derived from that internal location would not indicate that something had happened to the passwords before they were written to that location.
Not only that, but the fact that production data was able to flow from a secure production environment to a random internal disk somewhere, shows that the chain of custody has not been preserved.
Furthermore, there had to have been a series of (at least three) approvals, because this password mishandling happened within an application. That means that there would have been a feature request, where somebody would’ve reviewed and approved such functionality. And then, it would’ve been tested in a development environment, and then tested in a test environment. We need to invoke so many rogue, negligent, or complicit actors (development, QA, management) in order to explain how this wasn’t systematic.
Why the Explanation is Unlikely to be True
There are several reasons that I am having a hard time swallowing the explanation pill Facebook has offered:
No Notification – If Facebook did indeed discover, and fix, this issue back in January (it’s somewhat ambiguous in their post as to when exactly the fix occurred), why didn’t they notify users then? Why is this “precautionary” notification still to come?
Zuckerberg’s Testimony – Facebook CEO just defended something very similar to this in Congress as being unethical. His exact words (you can watch them spill from his mouth here) were:
Congressman: Do you have the ability [to put a name on my data, and share it with somebody]?
Zuckerberg: “Technically, I think someone [at Facebook] could do that, but that would be a massive breach, so we would never do that.”
Facebook’s Explanation Itself – as I mentioned above, Facebook’s claim that a “routine security check” uncovered this is highly questionable, but we have more information from Krebs’ interview with Facebook engineer Scott Renfro. He stipulates that Facebook is “reviewing any logs [they] have to see if there has been abuse or other access to [the passwords].” The problem is that the place they are looking (logs) is very unlikely to have any information about access. In fairness, I can’t know exactly what technology Facebook has at its disposal – but I do know (having developed technology that needed to determine when files are moved/created/accessed) that neither Windows nor Linux can do that out of box. It’s difficult to do that sort of thing, let alone at the scale of an environment like Facebook’s. I also know that such logs on the server device (not those that Facebook’s app is generating) would only be generated once the passwords were already on the disk.
Consequences for Users
I’m going to bet that for most of you (OK, at least a few of you), your Facebook password is the same as most of your other passwords. Which means, at the very least, Facebook may have enabled the single-greatest brute-force attack ever, if the wrong people get their hands on those passwords. And even if they weren’t the same – Facebook authenticates your accounts for a pile of other services! Finally, you probably don’t have two-factor authentication enabled, so the risk of your password being compromised is that much more likely to you losing data, or losing access, or enabling a breach.
Based on what we have heard through Krebs’ investigation, and from their response, Facebook is not structured in a way that it can take privacy seriously (no matter what Mr. Zuckerberg testifies).
But, hey, we’re about solutions, not problems – right? So here are three ways Facebook could’ve benefited from ActZero Networks’ services in this specific context. We’re focused on the small to medium-sized business, so they’re not our target market – but organizations that are still stand to benefit:
- Facebook’s software development life-cycle clearly had no input from a CISO. So, to me, it is obvious that they stand to benefit from our Virtual CISO service. We could have advised them on best practices concerning storage or analysis of passwords, separation of development from production environments such that security of production (read, sensitive) data is preserved. Just to walk the talk, the way our systems are set up, it is not possible for our developers to code straight to prod. Period.
- With the installation of our Endpoint Detection and Response Sensor, ActZero would have been able to tell Facebook which files were created, accessed, and when/by whom. This sensor is included as part of our MDR Service.
- The fact that none of us have received a notification of this mishandling of our personal data tells me that Facebook is not compliant with several privacy/security regulations. Which, had they engaged one of our Virtual CISOs, they could’ve understood, and had policies, processes, and frameworks for enforcement, created, documented and implemented.
You’ll forgive me if I’m not leaping to accommodate software giants (who are supposed to have this sort of thing under control). Instead, my focus on the businesses who need our help the most. I’d love to say more, but I’ve got to go delete my Facebook account.