Hello There, Guest! Register

Data loss, server migration
Chupi Offline
Goopi on IRC
Forum Administrators

Posts: 1,491
Threads: 39
Joined: Apr 2012
#1
 
Data loss, server migration
These forums have just been migrated to a new, less congested server. This should improve the slowness and occasional database errors.

However, because of how our web host handled the move, we're stuck bringing it back online today with a database backup from May 20. Any posts, PMs, etc. made between then and when the old server was taken down on the 24th, have been lost. We're sorry for any trouble this has caused.
05-25-2014, 05:15 AM
Website Find Reply
Pleeb Offline
Owner
Registered

Posts: 759
Threads: 60
Joined: May 2012
#2
 
RE: Data loss, server migration
I understand quite well that a lot can happen in 4 days, and I worked for hours yesterday to try and salvage any further backups past the 20th. The later one we had appeared to have been corrupted. While chupi gave a good overview, I'll try to go in a little more detail.

Every day for the past month, I've been getting emails that we've exceed our maximum concurrent sql connections. I would get about 4-5 of these emails per day... On May 17th, I got an email from my host; they said that they plan on migrating the server to new hardware, and that they'd do so on the 22nd. I made sure there was a backup in place (because I figured something might go wrong)... Then the 22nd came and went without any signs of migration. I contacted my host and awaited a response that never came.

On the 24th, the host apparently decided to do the migration. They shut off the current webservers and brought the new ones online; after realizing this, I changed the DNS settings to point to the IP of their new servers to find that not only were the latest files on the server from May 8th (which would hint that they migrated the server at a time earlier than even their earlist email) but the mySQL databases -- all of them -- were empty. 0MB in size. While http://www.tulpa.info was still online due to caching, community.tulpa.info, articles.tulpa.info, links.tulpa.info, it was all gone. I opened a support ticket with them.

Later that day, they got back to me apologizing and assuring us that the databases were restored. Things were at least functional, so we attempted to restore the latest backup of the forums using BigDump. Sadly, we kept getting an error, and when looking at the database, we noticed several tables have been corrupted with gibberish. The latest backup that was not corrupted was an auto-backup from May 20th, 2014.

We tried to get any later backups working, notably one from May 21st that Chupi had acquired manually. We still didn't have any luck.

I'm sorry about all of this. As I've mentioned before, we've essentially outgrown our shared hosting plan, having several million visitors a day, things are getting less manageable for us with this hosting company. Very soon, I'll be releasing an announcement about what the next step is for us; we can't stay where we are, we're running into too many problems. We're aiming for a dedicated server and I've been looking into a really nice datacenter for it; we'd have real access to these systems and easier methods of recovery. I'm also going to start paying for a better recovery system; Rasznir recommended Amazon Glacier for backups, we might go with them.

Once again, I'm really sorry about the 4 days of loss. I'm well aware of how much could have -- and has -- happened in the last 4 days. We're going to everything we can to prevent that from happening in the future, and I'll release another announcement about that soon (I was just hoping I could get through finals first at uni, which doesn't seem like it'll be the case).
(This post was last modified: 05-25-2014, 11:20 AM by Pleeb.)
05-25-2014, 11:16 AM
Website Find Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)

Contact Us | Tulpa.Info | Return to Top | Return to Content | Mobile Version | RSS Syndication