Page 1 of 2

Downtime and Data Loss

PostPosted: Wed Nov 14, 2012 5:51 pm
by Stacy
A harddrive in our server failed and it corrupted the database. Unfortunately, backups were corrupted, so we weren't able to recover all the lost data. As a result, we lost roughly 6,800 posts and 385 topics. I apologize for this. I should've done a better job ensuring we had frequent, working backups.

As a result of this, and the poor state of the VPS, we've moved to a new host. Now, we're being hosted by WebFaction (affiliate link). Hopefully things will work better now.


Edit: Also if you cannot log in please contact Larry at lwf58 at hotmail dot com

Re: Downtime and Data Loss

PostPosted: Wed Nov 14, 2012 5:58 pm
by Sunshine Temple
You also have my apologies for not taking Stacy's advice and moving to a more stable host.

This is the price of complacency. Of particular note is that the C&C section lost 21 topics and nearly 400 posts. Which, regrettably, has turned many of the threads into Swiss cheese.

Re: Downtime and Data Loss

PostPosted: Wed Nov 14, 2012 6:03 pm
by Crescent Pulsar
<-- Her two accounts were swiss-cheesed. XD;;

Oh, well.

Re: Downtime and Data Loss

PostPosted: Wed Nov 14, 2012 6:07 pm
by Dumbledork
That's too bad.

Re: Downtime and Data Loss

PostPosted: Wed Nov 14, 2012 6:20 pm
by frice2000
I lost over a thousand posts on this...Guys really. Hit the reset button and start over from scratch. This swiss cheesed stuff isn't better then just offering a link to what you had and restarting from zero again on a newer and hopefully less corruptible database. Hope you didn't lose user databases as well because that's almost inexcusable. And the loss of seeing a thousand posts off that post count does rather irk me. Would rather just see everyone starting over from zero when it's that bad.

Re: Downtime and Data Loss

PostPosted: Wed Nov 14, 2012 6:22 pm
by Sunshine Temple
That's not a bad idea. Or at least lock down all the old posts.

And a reset for the post counts is a good idea as the number of posts on your profile does not match the number of posts you have actually posted.

But this is in flux and can be sorted out going forward.

Re: Downtime and Data Loss

PostPosted: Wed Nov 14, 2012 6:43 pm
by Crescent Pulsar
I like the idea of having the old stuff available but otherwise starting anew. I was already thinking of having my two accounts deleted and starting a new one, as I'd rather do that than see what's still here and what's not. I was already thinking of starting anew with my account anyway, upon using my fiftieth avatar in the "series" on a permanent basis.

Wait, how's the list of fukufic stories and such?

Re: Downtime and Data Loss

PostPosted: Wed Nov 14, 2012 7:06 pm
by Sunshine Temple
Yes, Stacy is currently doing a migration where all the old posts will be moved to an archive section and locked.

Also what's the name on the two accounts of yours that got eaten? So far no one else has commented with a bad username, but that could be time.

Also if you cannot log in please contact me at sunshine AT

Edit: The fukfics page is also being worked on.

Re: Downtime and Data Loss

PostPosted: Wed Nov 14, 2012 7:21 pm
by Crescent Pulsar
This one and Crescent Pulsar R. Not so much eaten as chewed; I can't even begin to place exactly how far back my accounts went, seeing as R went back about five-thousand posts and this one went back to 9999, at some point while I was using the other account and not this one. If I had to guess, I'd say somewhere around a year and a half to two years.

It's not a big deal, but it kind of feels... not mine, anymore? I dunno how to describe it, exactly. And I was already feeling like making a change, anyway. But if we're starting a new forum, I'll just wait and take care of whatever myself when everything set up and running.

Re: Downtime and Data Loss

PostPosted: Wed Nov 14, 2012 7:43 pm
If I'd saved the text of certain missing C&C topics, I'd suggest maybe a temporary rule change allowing commentators with such to restart a missing C&C thread, if the original poster hadn't done it.

Or perhaps a sticky thread where we can mention threads which have been lost, or which have lost material, that we'd like the original posters to have available for comment again.

Maybe I'm full of it.

On my side:
a) I really need a better system for handling C&C on my end.
b) I need to be more productive about writing C&C, and posting it shortly after the new material is posted.

Re: Downtime and Data Loss

PostPosted: Thu Nov 15, 2012 4:58 am
by Shadou
I'll go ahead and post my observations on the crash here, since I was trying to log in when the crash happened. Due to the circumstances of what I was doing, I was more inclined to think the server was hacked. I'm still not convinced it wasn't, you'll see why when I explain.

I was able to browse to a few pages on the forum before the crash, and I was looking for a particular topic I had posted in, and couldn't find it. So I decided to log in and use the 'my posts' link on the front page to find it. The first time I tried, the login failed, it told me that my username was wrong. I double checked the username, and figured I had just typed the password wrong; sometimes boards have really obscure ways of telling you that you typed your password wrong that don't reveal that the username was a hit in case someone's trying to brute force usernames. But when I retyped my password and tried to log in again, the board completely crashed. So sometime in those 5 or so seconds between when I tried to log in the first time and when I tried to log in the second, that's when the board crashed.

Apparently, either the database was corrupt before the crash, including my profile having already been wiped from the board before the crash happened, or there was someone messing around with the database.

Re: Downtime and Data Loss

PostPosted: Thu Nov 15, 2012 8:34 am
by CRBWildcat
Amazing... spent a day in recovery from a hospital appointment, and I find out that things went kaboom.



Re: Downtime and Data Loss

PostPosted: Thu Nov 15, 2012 9:40 am
by lwf58
A correction to what Stacy posted: You should contact me, rather than Sunshine, to reset your account or create a new one. That's still my department.

Send an email to lwf58 at hotmail dot com stating the username and email account to be used, and I'll (re)create the account. Note that you'll get a randomly generated password that you can change in your profile controls once you log in. That way, I don't know your password, protecting your privacy.

This has been a distressing experience all around. I had been making periodic backups, but got so tied up in my personal life that I neglected to get even one from this year. Stacy had made a script that was intended to make regular backups, but it failed and she never got a notification. While her last backup was more recent than mine, it was still rather old. The poor girl had to work her tail off to get what she managed to out of the mess left by the server breakdown.

Since I was the one in contact with the web host we had, I'll put it in my own words. When the site went down, I contacted the host, and was told that a hard drive had failed in the RAID array used to hold our VPS. They'd replaced it routinely, but while failing, it corrupted the data on the hard drive that was still good. They attempted to replace the site with a copy from their own backup, but that copy seems to have been taken after the bad drive had already hashed things up. When they were done, we were able to access the server administrative controls, but the site database tables were damaged severely.

Coupled with the history of outages we've had, Sunshine and I agreed that the server was jinxed, and it was time to move on and find another. Stacy had been recommending the one we are now on, as she has had a good history with them. After much labor on her part, here we are, and we hope that the move will improve the site uptime to what it should have been all along. While I deeply regret that the site has lost so much, it was the coming together of every single thing that could go wrong, going wrong. We had know way of knowing it would happen since RAID arrays are designed to prevent this sort of thing, and Stacy thought she was making regular backups.

That said, here's to the future, and the hope that the Curse of Fukufics will finally be lifted. :roll:

Re: Downtime and Data Loss

PostPosted: Thu Nov 15, 2012 2:29 pm
by Noy TelinĂº
This just sucks.

Perhaps if I just did that post 666, nothing would have happened...


Anyway, do you want to see my fic posted here again? The one you guys helped me with but have no record of now?

Re: Downtime and Data Loss

PostPosted: Thu Nov 15, 2012 3:58 pm
by lwf58
This is the wrong place to ask that question. But I'd say just post the stories, and never mind asking if people want you to. Those who want to read and comment on them again will, and vice-verse.