Gandi Has Hardware Issues, Loses Some Customer Data and Backups

[cheat_master30]

Well, this is a bit of a disaster in the making. Domain registrar and web hosting company Gandi had some issues with their hardware today, and has seemingly lost a fair bit of customer data in the process. They're currently suggesting their users restore their data from backups (at another data centre no less!), and saying that their regular snapshots were also apparently hosted on the same machines that experienced errors.

Here's the email they sent customers about it:

Dear customer,
This mail is a follow-up to the previous email we sent (on January 8th, 2020) on this topic. As a reminder, yesterday, we experienced an incident on a storage unit at our LU-BI1 datacenter, located in Luxembourg.
Despite the replication systems in place, and the combined efforts of our technical teams throughout the night, we were unable to reover the data that was lost on the impacted storage unit.
We sincerely apologize for the inconvenience that this situation has caused. This type of incident is extremely rare in the web hosting industry.
In the event that you have a backup of your data, we suggest that you to use it to recreate your server at a different datacenter.
To help you in this, we have provided you with a promo code that will give you one free month for an instance, so that you can create a new Simple Hosting instance in a different datacenter:

Along with their status page:

Gandi.net Status Status

Welcome to Gandi.net Status's home for real-time and historical data on system performance.

status.gandi.net

Plus a rather... interesting Twitter thread showing some less than professional responsives from the PR team:



So yeah, if you're using them, you may want to consider going with another company now. The way they're dealing with the issue is not very good at all, and it seems their backups weren't up to scratch here.

 

[MagicalAzareal]

Ouch, that's bad. I agree with Tracy Perry here, you always need to be keeping regular backups. It might be interesting, if a software could perhaps run a backup every week and push it to something like S3 as a somewhat resilient third party backup?

 

[cheat_master30]

Never "depend" on an ISP's backups (or promises of backups). You should always do a manual backup of your system at least 1 time a week. That way at the worst you will only have lost 7 days of data.

Yeah, I agree 100% with this one. You definitely shouldn't rely on an ISP/host/software company to provide backups for you.

But you also shouldn't be like the hosting company here, and seemingly store your own 'backups' on the same hardware as the customer sites in question.

 

[Paul M]

It might be interesting, if a software could perhaps run a backup every week and push it to something like S3 as a somewhat resilient third party backup?

You can do that yourself, Ive been copying mine to S3 for years.

 

[zappaDPJ]

This type of incident is extremely rare in the web hosting industry.

Actually no it isn't, it's a surprisingly common occurrence so the advice never to rely on a service provider's backups is essential no matter what the provider promises.

 

[overcast]

I remember similar and even worst happened with Hostwinds. And they didn't even refunded or recovered the data for users.

 

[Chokkol]

Never "depend" on an ISP's backups (or promises of backups). You should always do a manual backup of your system at least 1 time a week. That way at the worst you will only have lost 7 days of data.

I learned this lesson a long time ago!

 

[mysiteguy]

Ouch, that's bad. I agree with Tracy Perry here, you always need to be keeping regular backups. It might be interesting, if a software could perhaps run a backup every week and push it to something like S3 as a somewhat resilient third party backup?

I setup my own servers to copy local backups to a Gsuite Google Drive and Microsoft OneDrive account, and it can as easily do this with almost any cloud storage system.

 

[Kyrie]

I've also learned to not buy cloud-based software long-term... company shut down and yet to receive a refund