276523 - add verification checks to make-diskimage

Status: RESOLVED INVALID

(Firefox Build System :: General, enhancement)

Description

[Krishna Sethuraman]

It would be handy if the nightly builds always pointed to a valid disk image,
even if the current nightly is invalid due to a build or packaging error.  I
have a script that pulls and runs the latest build when I restart Firefox (or it
crashes, which is very, very infrequently these days).  My script handles the
case of bad/empty images most of the time, but sometimes it poops out.

It would be nice if the Mac build process could 
* check if the disk image was at least 8M in size
* maybe also run some kind of mac utility (disk image verifier or whatever) to
verify that it is a valid disk image
* If invalid/too small, run the same tests against the previous nightly
* If the tests pass for the previous nightly, copy or move the previous nightly
into place instead
* Use that as the 'current' nightly

The 'current' nightly may then be a couple days old, but that's just fine for
me, considering the time scale of this project and the benefit of always having
a recent nightly image to download and test with.

See also bug 276123 , bug 273264 , and bug 267209 .

Thanks for a great product.

Krishna Sethuraman

Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8a6) Gecko/20041230
Firefox/1.0+

Comment 1

[Jo Hermans]

[e-mail] checked in a fix on the 5th, for the problem that was prompted
this bug-report (bug 276259)

see
<http://bonsai.mozilla.org/cvsview2.cgi?diff_mode=context&whitespace_mode=show&subdir=mozilla/build/package/mac_osx&command=DIFF_FRAMESET&file=make-diskimage&rev1=1.9&rev2=1.10&root=/cvsroot>


Is this enought to close this bug-report ?

Comment 2

[Krishna Sethuraman]

I'd prefer to have the script written in such a way that the last night's build
is retained if the current nightly build is screwed in any of multiple ways (not
just the ditto failing due to disk space) so I'd like to leave this open until a
more general solution is available.

I am, however, willing to have this close for now and reopen it only if the
problem (in particular, a corrupted build) pops up again.

Comment 3

[Chase Phillips]

The problem is that one or more commands in make-diskimage failed unexpectedly.
 In those cases (where there's an unknown problem that's causing our process to
fail) the accepted course of action is that the build should halt and generate a
failure report.  The build machine is unexpectedly broken and it's not worth
trying to predict how in scripts, just halt the build with a failure.

In your case, you're a very early adopter helping us to test the latest and
greatest changes in our code.  (Thanks!)  Why don't you add redundancy on your
end that caches a build you've marked as good in case the build it just
downloaded is bad?  If you're bitten by this kind of build bug, you can use the
good one to file a bug report on the broken one and we can hunt down what in the
build process is neglecting to verify the result of a particular command.  Not
only will that help us but it will also help you: if/when that problem occurs in
the future, the build will fail and you will not be presented with such an
unusable build.

As for your suggestion of strengthening make-diskimage, absolutely, I'll be
happy to review any patch you attach here that calls a disk image verifier.  If
that command were to fail, we should halt the build and let the Tinderbox
scripts send an error report.

Please resummarize this bug to reflect the need for more sanity checks in
make-diskimage.

Comment 4

[Gervase Markham [:gerv]]

This is an automated message, with ID "auto-resolve01".

This bug has had no comments for a long time. Statistically, we have found that
bug reports that have not been confirmed by a second user after three months are
highly unlikely to be the source of a fix to the code.

While your input is very important to us, our resources are limited and so we
are asking for your help in focussing our efforts. If you can still reproduce
this problem in the latest version of the product (see below for how to obtain a
copy) or, for feature requests, if it's not present in the latest version and
you still believe we should implement it, please visit the URL of this bug
(given at the top of this mail) and add a comment to that effect, giving more
reproduction information if you have it.

If it is not a problem any longer, you need take no action. If this bug is not
changed in any way in the next two weeks, it will be automatically resolved.
Thank you for your help in this matter.

The latest beta releases can be obtained from:
Firefox:     http://www.mozilla.org/projects/firefox/
Thunderbird: http://www.mozilla.org/products/thunderbird/releases/1.5beta1.html
Seamonkey:   http://www.mozilla.org/projects/seamonkey/

Comment 5

[Gervase Markham [:gerv]]

This bug has been automatically resolved after a period of inactivity (see above
comment). If anyone thinks this is incorrect, they should feel free to reopen it.

Comment 6

[Krishna Sethuraman]

> As for your suggestion of strengthening make-diskimage, absolutely, I'll be
> happy to review any patch you attach here that calls a disk image verifier. 

I don't have a full build environment available to me to test this, but I ran a
couple tests on the command line and found:

/usr/bin/hdiutil verify "$DMG_PATH" > /dev/null

should do the trick -- it returns 0 if the checksum is valid, or 1 if the
checksum is bad or another error occurs.

Inserting the command in:
----------------------------------------------------------------------
`dirname $0`/pkg-dmg --target "$DMG_PATH" --source "$SRC_FOLDER" \
--volname "$VOLUME_NAME" $EXTRA_ARGS
 
/usr/bin/hdiutil verify "$DMG_PATH" > /dev/null

exit $?
----------------------------------------------------------------------

Should produce an exit code reflecting the soundness of the generated .dmg.

Comment 7

[Reed Loden [:reed]]

While I moved it to mozilla.org :: Build & Release earlier, a second thought makes me wonder if it should not be in Core :: Build Config since build/package/mac_osx/make-diskimage is a core file. However, this only deals with the nightly build process, so I'm not sure. :)

Comment 8

[Chase Phillips]

(In reply to comment #7)
> While I moved it to mozilla.org :: Build & Release earlier, a second thought
> makes me wonder if it should not be in Core :: Build Config since
> build/package/mac_osx/make-diskimage is a core file. However, this only deals
> with the nightly build process, so I'm not sure. :)

If you're going to move it at all, you should be sure first.  Krishna did not resummarize this bug as I asked.  Doing so now.

Mark, is this bug still valid given the changes you've made to make-diskimage over the past 8 months?

Comment 9

[Mark Mentovai]

This bug is invalid now that the Mac packaging system has been entirely rewritten.  The new packager performs full error checking.  I don't think we've had a broken dmg attributable to mispackaging in many months.

Comment 10

[Chase Phillips]

(In reply to comment #9)
> This bug is invalid now that the Mac packaging system has been entirely
> rewritten.  The new packager performs full error checking.  I don't think we've
> had a broken dmg attributable to mispackaging in many months.

And thanks for that! :)

Comment 11

[Krishna Sethuraman]

Sorry, I missed the request to resummarize.  And thank you greatly for the fixes!