Sign in to follow this  

Data loss, server migration

Recommended Posts

These forums have just been migrated to a new, less congested server. This should improve the slowness and occasional database errors.


However, because of how our web host handled the move, we're stuck bringing it back online today with a database backup from May 20. Any posts, PMs, etc. made between then and when the old server was taken down on the 24th, have been lost. We're sorry for any trouble this has caused.

Lyra: human female, ~17

Evan: boy, ~14, was an Eevee

Anera: anime-style girl, ~12; Lyra made her

My blog :: Time expectations are bad (forcing time targets are good though)

Share this post

Link to post
Share on other sites

I understand quite well that a lot can happen in 4 days, and I worked for hours yesterday to try and salvage any further backups past the 20th. The later one we had appeared to have been corrupted. While chupi gave a good overview, I'll try to go in a little more detail.


Every day for the past month, I've been getting emails that we've exceed our maximum concurrent sql connections. I would get about 4-5 of these emails per day... On May 17th, I got an email from my host; they said that they plan on migrating the server to new hardware, and that they'd do so on the 22nd. I made sure there was a backup in place (because I figured something might go wrong)... Then the 22nd came and went without any signs of migration. I contacted my host and awaited a response that never came.


On the 24th, the host apparently decided to do the migration. They shut off the current webservers and brought the new ones online; after realizing this, I changed the DNS settings to point to the IP of their new servers to find that not only were the latest files on the server from May 8th (which would hint that they migrated the server at a time earlier than even their earlist email) but the mySQL databases -- all of them -- were empty. 0MB in size. While was still online due to caching,,,, it was all gone. I opened a support ticket with them.


Later that day, they got back to me apologizing and assuring us that the databases were restored. Things were at least functional, so we attempted to restore the latest backup of the forums using BigDump. Sadly, we kept getting an error, and when looking at the database, we noticed several tables have been corrupted with gibberish. The latest backup that was not corrupted was an auto-backup from May 20th, 2014.


We tried to get any later backups working, notably one from May 21st that Chupi had acquired manually. We still didn't have any luck.


I'm sorry about all of this. As I've mentioned before, we've essentially outgrown our shared hosting plan, having several million visitors a day, things are getting less manageable for us with this hosting company. Very soon, I'll be releasing an announcement about what the next step is for us; we can't stay where we are, we're running into too many problems. We're aiming for a dedicated server and I've been looking into a really nice datacenter for it; we'd have real access to these systems and easier methods of recovery. I'm also going to start paying for a better recovery system; Rasznir recommended Amazon Glacier for backups, we might go with them.


Once again, I'm really sorry about the 4 days of loss. I'm well aware of how much could have -- and has -- happened in the last 4 days. We're going to everything we can to prevent that from happening in the future, and I'll release another announcement about that soon (I was just hoping I could get through finals first at uni, which doesn't seem like it'll be the case).

Share this post

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.