729931 - Missing command did not trigger a build failure

Status: RESOLVED WONTFIX

(Firefox Build System :: General, defect)

Description

[Joey Armstrong [:joey]]

Submitting one file to try: config/NOP will report the error

http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-6e0b8936ecb1/try-macosx-debug/try-macosx-debug-build7733.txt.gz
http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-6e0b8936ecb1/try-macosx64-debug/try-macosx64-debug-build6525.txt.gz

Two mac64 slaves reported this problem:
    /bin/sh: objdump: command not found

bug 729926 was also open to determine if the command was not installed on a slave, cannot be found via $PATH or if configure may be horked and we should be using otool from xcode.


This bug is for tracking why exit status/failure was not propagated { configure, make, through shells, whatever }.  The try submission did not report an error status either in email or the web interface even though a failure should have been reported.

=======================================================================================
Thanks for your try submission (http://hg.mozilla.org/try/pushloghtml?changeset=6e0b8936ecb1).  It's the best!

Watch https://tbpl.mozilla.org/?tree=Try&rev=6e0b8936ecb1 for your results to come in.

Builds and logs will be available at http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-6e0b8936ecb1.

This directory won't be created until the first builds are uploaded, so please be patient.
========================================================================================

Comment 1

[Joey Armstrong [:joey]]

probably should also setup a unit test for this case for an easy sanity check later on.
Add config/flags/environment variables for configure use that will force configuring a non-existent shell command.

Unit test would create a scratch area beneath /tmp.
Define the magic value.
Run configure
Then verify a non-zero exit status is returned.

Comment 2

[Mike Hommey [:glandium]]

We don't need to care that it doesn't stop the build from happening because the test is wrong on mac anyways.

Comment 3

[Joey Armstrong [:joey]]

Independent of the command being valid or not for mac I would think not paying attention to and reporting exit status will be a great source for errors silently working their way into the tree.

At the least there will be a false sense about job success.  Checking tinderbox, green everywhere and assume everything is fine.  Worse if build failures {even transient ones} are ignored people have to waste time researching backward from the point where a problem manifested rather than going right to the source.

Even if the failures are ignored ala make -k to allow accumulating more results any failures reported during build/configure should be reported as and handled as a failure in the logs.  For individual failures that are spurious fix exit status for that one instance so errors can be reported in general for a build.

Comment 4

[Mike Hommey [:glandium]]

This particular command is a check. When what is checked is wrong, the build fails. In that, the check does its job. On mac, however, the command fails, but this doesn't indicate anything about the status of the check. It's just fine to not fail hard *for this particular command*.
Bug 729926 will either remove the check on mac, or add the correct corresponding check for mac, if there is one (it's not even guaranteed there's a possible corresponding check). We don't need two bugs for the same problem.

Comment 5

[Joey Armstrong [:joey]]

(In reply to Mike Hommey [:glandium] from comment #4)
> This particular command is a check. When what is checked is wrong, the build
> fails. In that, the check does its job.

This is probably the crux of the issue I have with this bug.
For the mac & mac64 builds:
  1) configure - objdir command check failed
  2) The mac build logs are littered with "objdump: command not found", 106 errors.

The build logs for mac reported a status of green/success *not* failure.
Try email did not report a failure for the build log.
At a casual glance mac* was able to build the job successfully.
==========================
rm -f libmozglue.dylib
/tools/buildbot/bin/python2.6 build/config/expandlibs_exec.py --uselist
/bin/sh: objdump: command not found
chmod +x libmozglue.dylib
==========================

If shell commands fail and/or configure and success is reported that is a pretty strong hint that exit status is being overlooked or lost somewhere along the way.


> On mac, however, the command fails,
> but this doesn't indicate anything about the status of the check. It's just
> fine to not fail hard *for this particular command*.

Hmmm, rather than loosey-goosey error handling I would actually like to see configure harden to the point that 'set -e' could be embedded within the scripts.  Yes I know this is a tall order, having seen how badly configure will upchuck when early exit on error is enabled.

But having configure not report failure for all error conditions will mean people have to manually review logs for problems when the configure script is changed.  This will also allow configure to have variant behavior when transient errors occur.

Since multiple configure scripts are run during the build:
If configure is unable to find a path/command because of a transient network/filesystem problem this would mimic the command not found reported by the mac build.  Then when configure[2] is run, if the problem no longer exists different command paths can be found.  This is a formula for building different parts of the tree with a distinct set of shell commands.

Say a default of 'python' v2.6 -vs- /usr/local/bin/python v2.7

Comment 6

[Mike Hommey [:glandium]]

(In reply to Joey Armstrong [:joey] from comment #5)
> (In reply to Mike Hommey [:glandium] from comment #4)
> > This particular command is a check. When what is checked is wrong, the build
> > fails. In that, the check does its job.
> 
> This is probably the crux of the issue I have with this bug.
> For the mac & mac64 builds:
>   1) configure - objdir command check failed
>   2) The mac build logs are littered with "objdump: command not found", 106
> errors.

Past tense. The mac build logs WERE littered with...

> The build logs for mac reported a status of green/success *not* failure.

And they rightfully did so, since the check wasn't meant to be used on mac, and can't be used on mac.

> Try email did not report a failure for the build log.

Because it was not a failure.

> At a casual glance mac* was able to build the job successfully.

And it was successful.
> > On mac, however, the command fails,
> > but this doesn't indicate anything about the status of the check. It's just
> > fine to not fail hard *for this particular command*.
> 
> Hmmm, rather than loosey-goosey error handling I would actually like to see
> configure harden to the point that 'set -e' could be embedded within the
> scripts.  Yes I know this is a tall order, having seen how badly configure
> will upchuck when early exit on error is enabled.
> 
> But having configure not report failure for all error conditions will mean
> people have to manually review logs for problems when the configure script
> is changed.  This will also allow configure to have variant behavior when
> transient errors occur.

As far as I can tell, there is no check for objdump in configure. Would there be one, and would it be running on mac, it would be WRONG. objdump isn't expected to be there on mac. It probably doesn't even exist for mac, or if it does, it probably doesn't make sense. objdump is for Linux, and is pretty much expected to always be there, since it comes with GNU ld.

So, running objdump on mac was a bug, and it's a good thing that it only "littered" logs. That bug was fixed. No need to go further down this road.