"If you were affected by this issue, it’s important to note that email sent to you between 6:00 PM PST on February 27 and 2:00 PM PST on February 28 was likely not delivered to your mailbox, and the senders would have received a notification that their messages weren’t delivered. "
That's pretty scary to me. I would've hated to have gotten any e-mail alerts or payment due date reminders during that time.
Since they got a warning that the email address was temporarily unavailable (at least that's what they should have received) any well configured mailserver should have resend the email.
If you look at http://www.oracle.com/us/products/servers-storage/storage/ta... it talks about the reduced cost of ownership for a 20PB tape library, so there are tape products available for large volumes of data. The SL8500 supports 500PB of raw storage.
If everybody filled their mailbox, 500 PB would back up 66.1 million Gmail accounts. Since they don't, we're probably looking at everybody on Earth: 73.8 MB per mailbox, which sounds like a pretty good shot at an average.
Looking at the SL8500, I think I'd intentionally lose data just so I could play with it to restore it.
There's probably a lot of people that don't "get" Gmail and delete all of their messages. There's probably also a lot of people who sign up for an account and don't use it, ever, and they contribute to the numbers. Keep in mind ~80 MB is for everyone on the planet (all nearly 7 billion) in 500 PB.
I'm betting the average is a lot lower than we'd think.
74MB seems little to me too (I am thinking of how often people have sent me ppt of cute cats, word documents of a todo list and tiff scans of some thing they drew).
It would be nice if grandparent has some reference for such a number to share
To quote from The Ringworld Engineers: "The Kzinti Patriarchy is not normally terrified by sheer magnitude." I can safely say that the same statement applies to Google.
Total conjecture, but 3 ways come to mind: De-duplication, compression, and hierarchical storage management. Keep less-frequently accessed parts of the data 'Nearline', and snapshot the lower-demand, older data slightly less often.
Continuous data protection appliances were probably designed and tuned specifically for this problem set, and one could assume it was built into the original architecture so that an appropriate amount of the infrastructure already had backup space in mind and accounted for.
150,000 accounts out of SEVEN HUNDRED AND FIFTY FUCKING MILLION had trouble accessing email for LESS THAN A SINGLE DAY on A WEEKEND. No accounts or emails were deleted.
Why? Because I'd need to wait 20 hours to get that critical life-changing email? If it was critical, someone could call my home phone, my cell phone, text me, actually have the local police stop by my apartment, etc.
This winter was pretty bad here in the US. More people (and much higher percentage of some company's user bases) lost power for longer periods of time in freezing temeratures.
More people die in a single day than the people who temporarily lost email access.
Sure, It'd suck in that period where I thought I lost all my emails, but all's well that ends well. Lets keep things in perspective. This was more of a "SHIT THE CABLE'S OUT" issue than a serious life changing problem for anyone affected.
It turns out that I was hit by this issue. And guess what - it was a big deal. For starters, it is a MONDAY here in Australia. There was no support information on this, and no help from Google.
Secondly, the login message was "Your account has been disabled". If I could log in, and there was a message that my email was being restored from backup because they had a failure - fine. That wasn't the case. They had disabled my Google Apps Administrator account (same account as my email), email was bouncing, and I couldn't log in to my email OR the administration interface. The "reset your password" and "unlock your account" links failed to unlock it.
Comparing an email outage to people's deaths is poor form. Of course they aren't the same. That doesn't make it an issue when email is a super important communication tool.
People who were affected had their entire Google accounts disabled, and upon trying to log in, they got exactly the same messages they would have gotten if Google had decided to delete the account for ToS abuse. Additionally, since the entire account (not just GMail) had to be disabled in order to repair things, stuff like shared google calendars went offline, so users whose accounts were not directly affected were getting misleading error messages, too.
And the bounces didn't stop until the end of the working day on Monday on the east coast.
You're both right. It IS an insignificant number, but it would suck, out loud, if you happened to be in that number. (I was not.)
I have 2 opposing feelings on the matter; 1 is that email is not backup. If you're using it as such, well, you run this risk. Email is a communication medium; if ideas or information presented therein are important to you, save them in appropriate places. Email is only a way to get the idea/info TO you, not where to keep it. And yes, I realize reasonable people disagree here.
The other is that at least with Google, the account is intimately tied to any Android devices you have. I've heard reports of people not being able to use large swaths of functionality on their phones or tablets due to them being in that insignificant number. That would make the suck possibly an order of magnitude or more worse.
Bringing the total number of affected users down to 38,660.
Not nothing, but not a lot either. My company has an exchange server for 200 or so people, and at any one point there are probably one or two people who are waiting for a password reset, errors with Calendar, public folder permissions errors, or other human error on the part of the helpdesk interns. That's a 100x increase from Google's error rate in this scenario, but it's perfectly acceptable within the context of our organisation.
That's why we have tape backups (which can take some time / effort to restore from). I think the difference is that if a user at my company has a problem they can call a human being and be told that someone is working on it, but with Google you can't do that.
This is not true. During the affected time period, those accounts could not receive email, and sent out permanent failure bounce messages. If you were one of the affected and you were sent an important email during that time... You didn't get it, and you might never get it.
I really hope that I'm not using anything you created on the web. I would hate to be a customer of a service whose owner think 150,000 users are insignificant.
The original commenter pointed out that 0.02% is a more significant figure than the Gmail team makes it out to be, and that 150,000 is very significant. I'm saying the 150,000 figure isn't as significant as it looks, either.
"We released a storage software update that introduced the unexpected bug, which caused 0.02% of Gmail users to temporarily lose access to their email. When we discovered the problem, we immediately stopped the deployment of the new software and reverted to the old version."
It is worth noting that this is the first significant data loss issue that I am aware of involving gmail. As much as the incident sucks for those involved, that is a pretty good record overall.
This is great, last night I went to bed thinking "How could Google just walk away from this problem without fixing it, don't they have backups, what about corporate users?"
And this morning "we have backups, we're restoring, oh and we're trying to fix what caused it in the first place"
Would it be safe to assume then, that Google tests it's software upgrades with 0.02% of it's gmail account? (However many thousands clients are affected, it's probably better than affecting ALL their clients).
I wonder how they choose their 'test' client base.
Yahoo is far worse. When they lose your mail, it's gone.
Yahoo Mail lost two of my accounts over the years. The first time was early in the service, and they seem to have lost all their accounts at once. Suddenly you couldn't log in, and there was an apology message saying that you'd need to create a new account. And that you couldn't get the same email address back, which is why my @yahoo.com address had a 2 at the end of it for several years...
Until they locked me out of it one day. As in, I checked my mail at night, then in the morning my password didn't work anymore. Since they have no human beings working for them, an entire month's worth of attempts to get access back resulted in nothing but a big pile of form letters and links to the Forgot Password page.
So no, you're mistaken. Yahoo does have this problem. If you have an account with them I'd strongly advise you to back everything up. Then switch to Gmail.
Yahoo also charges a monthly fee for access to pop3. Their web interface is slower and buggier than gmail's. There is free unlimited storage, but a smaller limit on file sizes. The virus scan on outgoing messages is usually slow; if gmail is doing virus scans, it's on the backend, it does not delay the user.
Oh, I'm on Gmail already, trust me. But thats because I like the faster interface, search capabilities, being able to check multiple mail accounts, gtalk etc. Yahoo is a great repository for my non-friends and family email (semi-spam).
I guess I've been very lucky that never happened to my Yahoo account.
That's pretty scary to me. I would've hated to have gotten any e-mail alerts or payment due date reminders during that time.