649972 - Random failures with "hdiutil eject" during partner repacks on macosx64

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, defect, P2)

Description

[Rail Aliiev [:rail]]

During Firefox 4.0.1 build we hit random failures on different slaves and different partner configs. Log snippet:

##################################################
### Starting repack process for partner: yahoo-it_browser1
### Found /builds/slave/rel-2.0-osx64-partner-rpk/partner-repacks/scripts/original_builds/4.0.1/build1/mac/it/Firefox 4.0.1.dmg on disk, not downloading
### Repacking mac build "Firefox 4.0.1.dmg"
expected   CRC32 $75645842
/dev/disk1          	Apple_partition_scheme         	
/dev/disk1s1        	Apple_partition_map            	
/dev/disk1s2        	Apple_HFS                      	/private/tmp/FirefoxInstaller
hdiutil: couldn't unmount "disk1" - Resource busy
Error: shellCommand had non-zero exit status: 1
command was: hdiutil eject  /tmp/FirefoxInstaller
program finished with exit code 1


Maybe we need to sleep or call sync after rsync here: http://hg.mozilla.org/build/partner-repacks/file/bcae25fd0a92/scripts/partner-repacks.py#l434 ?

Comment 1

[Rail Aliiev [:rail]]

Attachment: sync before ejecting

r=coop (IRC)

Comment 2

[Rail Aliiev [:rail]]

Comment on attachment 525984 [details] [diff] [review]
sync before ejecting

Checked in: http://hg.mozilla.org/build/partner-repacks/rev/79217934f156

Comment 3

[Rail Aliiev [:rail]]

For Firefox 4.0.1 it was fixed by a manual build with added "sleep 15" after mount and before eject.

Probably it would be better to use http://mxr.mozilla.org/build/source/tools/buildfarm/utils/installdmg.sh here.

Comment 4

[Rail Aliiev [:rail]]

backed out http://hg.mozilla.org/build/partner-repacks/rev/c7c7f8241fbc

Comment 5

[Rail Aliiev [:rail]]

Attachment: use tools/release/common/unpack-diskimage.sh

release/common/unpack-diskimage.sh could be used to extract dmg contents. Not tested yet.

Comment 6

[Rail Aliiev [:rail]]

Attachment: buildbotcustom part

Comment 7

[Chris Cooper [:coop] (he/him)]

cc-ing Kev so that he knows he'll need to pull the build/tools repo to get the installdmg.sh script.

Comment 8

[Rail Aliiev [:rail]]

Attachment: use tools/release/common/unpack-diskimage.sh

Comment 9

[Rail Aliiev [:rail]]

Comment on attachment 528099 [details] [diff] [review]
use tools/release/common/unpack-diskimage.sh

>     def cleanup(self):
>         super(RepackMac, self).cleanup()
>         rmdirRecursive("stage")
>-        rmdirRecursive(self.mountpoint)

unpack-diskimage.sh handles this already.

Comment 10

[Rail Aliiev [:rail]]

Attachment: tools

Comment 11

[Rail Aliiev [:rail]]

Comment on attachment 528100 [details] [diff] [review]
tools

FTR.

>-rsync -av $MOUNTPOINT/* $MOUNTPOINT/.DS_Store $MOUNTPOINT/.background $MOUNTPOINT/.VolumeIcon.icns $TARGETPATH/. > $LOGFILE
>+rsync -av $MOUNTPOINT/* $MOUNTPOINT/.DS_Store $MOUNTPOINT/.background $MOUNTPOINT/.VolumeIcon.icns $TARGETPATH/ > $LOGFILE

I removed the trailing dot to make rsync happy. It was complaining "no such file or directory"

> hdiutil detach $MOUNTPOINT
>-rm -rdf $MOUNTPOINT
> # Sleep for a bit to let messages from diskimage-helper go away
> sleep 5
>+rm -rdf $MOUNTPOINT

Sometimes rm doesn't work as expected and complains about ro file system. It's
not critical because the next call removes the directory (after final sleep 5).
However it doesn't apply for the last repacks.

Comment 12

[Rail Aliiev [:rail]]

Test scenario:

1) downloaded production Firefox 4.0.  build1 partner repacks to "orig"
2) downloaded new Firefox 4.0.1 build1 partner repacks to "mine"

set -x
UNDMG=$PWD/tools/release/common/unpack-diskimage.sh

find orig -type f -name '*.dmg' | while read orig; do
 mine=$(echo $orig | sed 's|orig/|mine/|')
 $UNDMG "$orig" /tmp/mnt orig_dir
 $UNDMG "$mine" /tmp/mnt my_dir
 diff -Nr orig_dir my_dir >> diff.diff
 rm -rf orig_dir my_dir
done

$ wc  diff.diff 
       0       0       0 diff.diff

Comment 13

[Rail Aliiev [:rail]]

Need to be landed during the next reconfig.

Comment 14

[Rail Aliiev [:rail]]

Attachment: tools

I think this patch should work much more better than previous.

* Instead of dummy "sleep 5" at the end, I added a loop, which handles the situation with diskimage-helper safer and may finish faster (5 sec! :))

* Under some conditions hdiutil command exits non-zero complaining about "Resource is busy". In this case the loop I mentioned above doesn't help. I added "sleep 10" and forced eject for this case.

I compared the previous and the current implementations. The previous one was failing in ~30-40% cases, while the latest one finished green for 12 runs in a row.

I also tested l10n verify factory, which uses unpack-diskimage.sh and it worked fine. The output is the same as before.

Update verify results will be ready later today.

Comment 15

[Chris Cooper [:coop] (he/him)]

Comment on attachment 528289 [details] [diff] [review]
tools

Yay for more resiliency.

Comment 16

[Rail Aliiev [:rail]]

Comment on attachment 526690 [details] [diff] [review]
buildbotcustom part

http://hg.mozilla.org/build/buildbotcustom/rev/5985ac76d496

Comment 17

[Rail Aliiev [:rail]]

Comment on attachment 528099 [details] [diff] [review]
use tools/release/common/unpack-diskimage.sh

http://hg.mozilla.org/build/partner-repacks/rev/dd0383f01845

Comment 18

[Rail Aliiev [:rail]]

Comment on attachment 528289 [details] [diff] [review]
tools

http://hg.mozilla.org/build/tools/rev/7002fa1094be

Comment 19

[Rail Aliiev [:rail]]

All done here.