There’s an old list of how different programming languages let you shoot yourself in the foot. I managed to do something similar on our virtual servers last Wednesday. I just got it fixed Saturday afternoon.

I learned something. Never touch the disks of a virtual machine that has snapshots. “The parent virtual disk has been modified since the child was created.” Fear these words, for you will lose sleep over them. Below is a time line of what got me into the mess and how I got myself out. I tried a few more things, but this is what actually worked.

Goal: Install 3 topic CDs into the Read180 software.

Failure: Server only has 1.5 GB free. It ran out of space last week because Read180 doesn’t clean up after itself (1.5 GB+ of log files). There’s no chance of installing 3 CDs worth of data and keeping the server from taking itself out from too many logs.

New Goal: Increase HD size from 10 GB to 30 GB.

Procedure:

1. Shutdown Read180 virtual machine
2. SSH into ESX server
3. Use vmkfstools to extend server VMDK to 30 GB
4. Attach VMDK to utility server to extend NTFS partition
5. Boot utility server
6. Extend NTFS partition using DiskPart
7. Shutdown utility virtual machine
8. Boot Read180 server

Failure: “The parent virtual disk has been modified since the child was created.” Uh oh. I felt a bit nauseated at this point.

New Goal: Restore server from a backup.

Procedure:

1. Create new virtual machine from template
2. Install Backup Exec agent
3. Restore last full backup and latest differential backup
4. Reboot server

Mixed Results: The server looked just like it did when it was actually working. A sigh of relief. But wait, Backup Exec couldn’t backup Read180’s embedded MySQL database because the database files were always in use! Therefore Read180’s database was missing, and the restore was useless! Got some experience restoring a server entirely from backup, which was good. Still nauseated.

New Goal: Manage to restore the database somehow, or prep a new installation of Read180 and break the bad news to some VERY unhappy teachers (and boss) Monday morning.

Next Action: Make a post on the VMTN Discussion Forums about my problem. I didn’t realize at the time, my own theory in the second post would be the solution that worked.

Wait: Take a breather for the holiday. Still nauseated.

Next Steps: Try my crazy idea that seemed far too simple for such a complex problem. If it doesn’t work, begin rebuilding the server from scratch. This time with a bigger drive, and figure out somehow to backup the database nightly.

Procedure:

1. Download GParted LiveCD
2. Create new virtual machine that boots from CD
3. Attach broken 30 GB VMDK to GParted virtual machine
4. Boot GParted ISO, using manual video card settings (VESA at 800×600)
5. Use GParted to shrink NTFS partition from 30 GB to 10 GB
6. Create blank 10 GB VMDK
• vmkfstools -c 10G -d thick wwcs-read180-10g.vmdk
7. Copy contents of 30 GB disk to new 10 GB disk using DD
• dd if=wwcs-read180-flat.vmdk of=wwcs-read180-10g-flat.vmdk bs=1048576 count=10240
• Note: bs and count are only for 10 GB of data
8. Move old broken VMDK out of the way, rename new 10 GB drive old name
• vmkfstools -E wwcs-read180.vmdk wwcs-read180-30gbad.vmdk
• vmkfstools -E wwcs-read180-10g.vmdk wwcs-read180.vmdk
9. Open a snapshot depends on the broken VMDK (Read about CID chains for more info about this)
• nano wwcs-read180-000001.vmdk
• Note: Record parentCID variable
• Example: parentCID=b7c9e128
10. Open fixed VMDK, set CID to value from parentCID in previous step
• nano wwcs-read180.vmdk
• Example: CID=b7c9e128
11. Boot up virtual machine
12. Virtual Infrastructure Client wanted to know if I wanted to convert the SCSI type. I told it no.
13. Read180 virtual machine booted
14. Verified Read180 database as best I could
15. Copied database to file server
16. Shutdown services, kicked off Full Backup Exec backup

Success: The server booted. I had some pie.

Next Steps: Figure out how to get a good daily backup from Read180. I hope there’s no secret password on the MySQL database, or getting a backup will be very hard. It’s interesting that tools like ESXRanger or VMware Consolidated Backup would not have helped with this problem.

This entry was posted on Sunday, November 26th, 2006 at 1:22 am and is filed under Best Of, Information Technology, System Administration, Virtualization. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.