R. Bemrose (powerlord) wrote in vgmusic_status,
R. Bemrose
powerlord
vgmusic_status

  • Mood:

Filesystem problem

We're currently dealing with a file system problem. During the meantime, the site will be down.


Friday



1:00pm EST (approx): I reboot the server after seeing strange behavior from the system. When the system comes back up, it does so with the main hard drive in read-only mode because of a file system (journal playback) error on boot.

1:25pm EST: I'm currently downloading site backups, as FTP seems to be working OK. This will take some time.

2:20pm EST: I ran a file system check and fixed some errors. Now the file system's REALLY screwed up. Trying to shut the system down tried to run one of the site's text files.

2:30pm EST: I have posted a trouble ticket with our host... most likely the first in a chain of them in order to get things restored. Having done this before, I know I can't just ask for a server restore or hardware replacement (Hard Drive) or they'll want me to pay for it.

3:16pm EST: EV1 changes my check system request to a Reboot request. I have no idea why, but they say they'll investigate if it doesn't come back up.

3:31pm EST: Server has been sent to Investigations.

4:56pm EST: Investigations says the kernel isn't booting. No surprise there. They recommend restoring the system. Unfortunately, this takes 3-24 hours unless you're willing to pay more.

5:25pm EST: Ordered a System Restore (Sorry, I was taking a short nap during the above time).

6:11pm EST: Customer Service has stated "We will begin imaging a new drive to be used for the restore and will be taking your server offline shortly. Please standby for further updates." If it's the drive that was the problem, that should fix it. If it was the IDE controller...

7:51pm EST: The server is finally up, although I didn't know it yet.

8:30pm EST: Reinstalling OS.

11:30pm EST: Forgot to install the network driver when switching kernels, leaving networking disabled.

11:45pm EST: Asked for a KVM to be connected to the system so I can fix the previous problem.

2:00am EST: Still having no response from EV1Servers about the KVM deal, I call it a night

Saturday



10:45am EST: I'm back up and waiting on data center again.

Noon EST: Nothing yet, taking a nap.

1:00pm EST: Change trouble ticket type in hopes they'll at least acknowledge it.

Played Sam & Max: Hit the Road for a bit while waiting for EV1.

3:15pm EST: After no acknowledgement after ticket was updated, I've submitted a new trouble ticket. We'll see if that gets us anywhere. Tonight is the night my friends usually stop over, so it'd be nice if the server is finished by the time they get here.

5:20pm EST: Investigations has reported that the system wanted a manual file system check. Needless to say, the system requesting a file system check the first time a partition is booted from is abnormal and may indicate a hardware problem. If so, the site's return may be further delayed.

10:20pm EST: The server is now operational, but backups have not yet been transferred to the server. I will begin transferring those as soon as I get the initial system up and running... and booting from the correct drive every time.

11:00pm EST: Beginning to upload backups... estimated transfer time on the main backup is 2 hours. However, I'm likely to be asleep by then.

8:30am EST: The files finished transferring while I was asleep. Now bringing the site online.

9:10am EST: The site is now online.

ETA: Site now online
Subscribe
  • Post a new comment

    Error

    default userpic

    Your reply will be screened

    Your IP address will be recorded 

    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.
  • 7 comments