Dear customers.
On 30 November 2011, after a successful install and configuration of a new router in our Melbourne datacenter, we had to restart a few servers in order to update new network configuration and reload new kernel settings. One of servers (my.redyhost.com.au) was unable to reboot and was down since 05:20AM until 6:50AM AEST, while our support team was working to resolve the issue.
Servers affected
Only my.redyhost.com.au server was affected. Clients accessing their services via different URL were not affected.
What happened
Beyond advised maintenance window (5:00AM-5:20AM AEST), at around 5:20AM one of our cPanel servers was unable to restart. This happened due to kernel packages installed previously failed to load.
Our support team was working to resolve the issue with no success. After 6:00AM AEST we began rolling back server files to previously created daily snapshots. Unfortunately, only the snapshot from 11:59PM 25 November 2011 AEST was successful, while all other snapshots taken earlier failed on the same reason (kernel modules panic).
Current situation
All services are fully operation.
Possible data loss
It is possible that some web site data and emails were lost between 05:00AM - 06:50AM on 30 November 2011 AEST.
If you have updated your web site between 11:59PM 25 Nov - 6:50AM 30 Nov 2011 AEST - it is possible that those changes were lost during the rollback process.
Resolution plan
Our support team continue to work on restoring the lost data. The backups require uncompressing and decrypting of files from the second backup source. Please note that this process may take between 1 and 5 days to complete (this is due to high level of compression and 256bit encryption). Once the data decompression is complete, we will be able to update any missing files and configuration options for our cPanel clients. Please contact RedyHost as soon as possible if you believe your web site data is missing or incomplete.
Customer compensation
RedyHost management decided to provide compensation to affected customers. The amount of compensation will be equal to the value of one month cost of the service (for affected service/s only). Customers are advised to contact RedyHost by email or submit a support ticket and claim the compensation. Please include service name affected by the disaster. Only claims received by 30 December 2011 will be compensated.
RedyHost apologises for any inconvenience caused. We are currently reviewing our backup strategy to avoid similar problem in the future.
Ivan Grynenko
Director
RedyHost (trueCMS Pty. Ltd.)
Wednesday, November 30, 2011