660350 - vm builds fail on kvm.build.mtv1.mozilla.com

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Amy Rich [:arr] [:arich]]

I was trying to build two vms on the kvm infrastructure in mtv1, and got a fatal error on both:  


[[e-mail]1 ~]# gnt-instance add -t drbd -I hail -o image+centos-55 -B memory=786M --no-start --net=0 -s 20G  ns2.build.mtv1.mozilla.com
Fri May 27 14:10:36 2011  - INFO: Selected nodes for instance ns2.build.mtv1.mozilla.com via iallocator hail: kvm1.build.mtv1.mozilla.com, kvm2.build.mtv1.mozilla.com
Fri May 27 14:10:36 2011 * creating instance disks...
Fri May 27 14:10:52 2011 adding instance ns2.build.mtv1.mozilla.com to cluster config
Fri May 27 14:10:53 2011  - INFO: Waiting for instance ns2.build.mtv1.mozilla.com to sync disks.
.
.
.
Fri May 27 14:42:43 2011  - INFO: Instance ns2.build.mtv1.mozilla.com's disks are in sync.
Fri May 27 14:42:43 2011 * running the instance OS create scripts...
Failure: command execution error:
Could not add os for instance ns2.build.mtv1.mozilla.com on node kvm1.build.mtv1.mozilla.com: OS create script failed (exited with exit code 1), last lines in the log file:
Units = mebibytes of 1048576 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start   End    MiB    #blocks   Id  System
/dev/drbd4p1   *     0+   101-   102-    104422   83  Linux
/dev/drbd4p2       101+   894-   793-    811282+  82  Linux swap / Solaris
/dev/drbd4p3       894+ 20473- 19580-  20049120   83  Linux
/dev/drbd4p4         0      -      0          0    0  Empty
Successfully wrote the new partition table

Re-reading the partition table ...
BLKRRPART: Invalid argument

Setting up swapspace version 1, size = 811276 KiB
no label, UUID=8519e02a-7fd4-4cf1-86a0-75c072f7ed32
Dump tape is compressed.
decompression error, block 7451: data error
File decompression error while restoring <directory file - name unknown>
decompression error, block 7461: data error
File decompression error while trying to resynchronize
/usr/lib/ganeti/os/image/create: line 112: 11794 Segmentation fault      (core dumped) restore -r -y -f ${root_dump}


And the second:

Fri May 27 14:46:51 2011 * running the instance OS create scripts...
Failure: command execution error:
Could not add os for instance ns1.build.mtv1.mozilla.com on node kvm2.build.mtv1.mozilla.com: OS create script failed (exited with exit code 1), last lines in the log file:
Units = mebibytes of 1048576 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start   End    MiB    #blocks   Id  System
/dev/drbd3p1   *     0+   101-   102-    104422   83  Linux
/dev/drbd3p2       101+   894-   793-    811282+  82  Linux swap / Solaris
/dev/drbd3p3       894+ 20473- 19580-  20049120   83  Linux
/dev/drbd3p4         0      -      0          0    0  Empty
Successfully wrote the new partition table

Re-reading the partition table ...
BLKRRPART: Invalid argument

Setting up swapspace version 1, size = 811276 KiB
no label, UUID=29a0c01a-98e8-4298-a23a-f6e19861ac91
Dump tape is compressed.
decompression error, block 7451: data error
File decompression error while restoring <directory file - name unknown>
decompression error, block 7461: data error
File decompression error while trying to resynchronize
/usr/lib/ganeti/os/image/create: line 112: 15585 Segmentation fault      (core dumped) restore -r -y -f ${root_dump}


These are the first vms I've tried to build since you copied the new image over, Ben... Could that have something to do with it?

Comment 1

[Amy Rich [:arr] [:arich]]

Two issues resolved:

1) corrupted centos-55 image replaced with the one on kvm.infra.scl1.
2) patch for restore leaving files in /tmp causes creation/import to fail: 
http://code.osuosl.org/issues/5967

reinstalled vms for ns1 and ns2 without issue.