Closed Bug 468447 Opened 13 years ago Closed 12 years ago

Nonfatal "NMAKE : fatal error U1052: file 'makefile.sub' not found" in every single log masks real errors in failure logs

Categories

(Firefox Build System :: General, defect)

x86
Windows NT
defect
Not set
normal

Tracking

(status1.9.2 beta1-fixed, status1.9.1 ?)

RESOLVED FIXED
Tracking Status
status1.9.2 --- beta1-fixed
status1.9.1 --- ?

People

(Reporter: mats, Assigned: ted)

References

()

Details

Attachments

(1 file)

This is a recurring build error on Tinderbox
"WINNT 5.2 mozilla-central nightly %"

NMAKE : fatal error U1052: file 'makefile.sub' not found

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1228730968.1228746087.18971.gz
This is from the Visual C++ CRT build process which we build to get jemalloc. It's possible we could patch the nmake Makefiles to avoid this, but I have not been motivated to fix it. It's not fatal to the actual build.
Yeah, I recommend WONTFIX
Would a bug to not make them show up as errors in the (brief) build log also be wontfix? Training people to ignore that because it's always full of all that expected crap doesn't seem like a perfect situation.
Hiding the error from from the build log would really be helpful, when a build fails for other reasons you get a log like:

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1230175606.1230176938.8666.gz

Build Error Summary

NMAKE : fatal error U1052: file 'makefile.sub' not found
NMAKE : fatal error U1052: file 'makefile.sub' not found
NMAKE : fatal error U1052: file 'makefile.sub' not found
NMAKE : fatal error U1052: file 'makefile.sub' not found
NMAKE : fatal error U1052: file 'makefile.sub' not found
NMAKE : fatal error U1052: file 'makefile.sub' not found
NMAKE : fatal error U1052: file 'makefile.sub' not found
make[6]: *** [install] Error 3
make[6]: *** Waiting for unfinished jobs....
make[5]: *** [export] Error 2
make[4]: *** [export_tier_js] Error 2
make[3]: *** [tier_js] Error 2
make[2]: *** [default] Error 2
make[1]: *** [build] Error 2
make: *** [profiledbuild] Error 2

The first pile of "errors" are misleading; the real problem is afterwards. (In this case, apparently a stuck process locking a DLL)
The error parser being used by the tinderbox server is 
  http://mxr.mozilla.org/mozilla/source/webtools/tinderbox/ep_unix.pl
(from the tinderbox: lines at the top of the full log).
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1240135320.1240141918.14560.gz
WINNT 5.2 mozilla-central nightly on 2009/04/19 03:02:00
And I presume that dholbert and ehsan are doing what they can to illustrate that this is in fact a problem, that people believe that when they see it in the log they've found the problem and when they mention an instance in this bug they can then star the build and move on.
Summary: NMAKE : fatal error U1052: file 'makefile.sub' not found → Nonfatal "NMAKE : fatal error U1052: file 'makefile.sub' not found" in every single log masks real errors in failure logs
(In reply to comment #8)
Yup, that's exactly what happened with me. :) Phil, thanks for clarifying the actual situation behind this issue by updating the bug summary.

I think some fix is needed here, whether it's hiding the error from the parser, or fixing the parser to ignore the error.

(In the meantime, we should educate people, perhaps with a note on the tinderbox header, that this error isn't the actual cause of bustage, but instead masks other problems...  Maybe this updated bug summary is enough in that department)
I have looked at the CRT makefile and I have no idea where these errors are coming from. I guess we could just hack build-crt.py (which executes nmake to build the CRT) to hide them, or something.
This failure happened today on the try server. There was significant load at the time.

http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1240263699.1240274665.3265.gz

I have seen it before on the TM tree, along with a similar looking problem where xpcom.lib can't be deleted or created. My wild guess would be a NAS issue (locking?).
Maybe this is related. I have seen a lot of forced and periodic clobbers on the TM tree.

17:31:08 < nagios> [27] moz2-linux-slave12.build:disk - /builds is WARNING: DISK WARNING - free space: /builds 1553 MB (5% inode=70%):
gal: these are not the error messages you are looking for. Every single Windows build log, including green ones, shows these exact same error messages. Whatever your actual error is, it is not this bug, which is about getting rid of these bogus errors so they don't confuse you into thinking they were real errors and the cause of your problem.
Terrible, but effective. Easier than trying to hack Microsoft's nmake makefile. r?Waldo because bsmedberg is out and a) it's just Python, b) I borrowed this code from automation.py anwyay.
Assignee: nobody → ted.mielczarek
Status: NEW → ASSIGNED
Attachment #373825 - Flags: review?(jwalden+bmo)
Attachment #373825 - Flags: review?(jwalden+bmo) → review?(benjamin)
Attachment #373825 - Flags: review?(benjamin) → review+
Pushed to m-c:
http://hg.mozilla.org/mozilla-central/rev/2e1cacdcdd16
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Comment on attachment 373825 [details] [diff] [review]
hack build-crt.py to elide these errors from the log

This shows up on Tinderbox for branches.  It would be great to take this fix on branches as well, so that people don't see misleading oranges on branches.  According to Ted, this is really low-risk.
Attachment #373825 - Flags: approval1.9.2.9?
Attachment #373825 - Flags: approval1.9.1.12?
Comment on attachment 373825 [details] [diff] [review]
hack build-crt.py to elide these errors from the log

a=LegNeato for 1.9.2.9 and 1.9.1.12
Attachment #373825 - Flags: approval1.9.2.9?
Attachment #373825 - Flags: approval1.9.2.9+
Attachment #373825 - Flags: approval1.9.1.12?
Attachment #373825 - Flags: approval1.9.1.12+
I noticed that 1.9.2 has been branched before this changeset.  I landed it on 1.9.1 only:

http://hg.mozilla.org/releases/mozilla-1.9.1/rev/54ff947391f9
Apparently 1.9.1 needs more bitrotting work than I realized.  The optimized build failed with:

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox3.5/1280958092.1280959119.493.gz&fulltext=1#err0

Traceback (most recent call last):
  File "e:/builds/moz2_slave/mozilla-1.9.1-win32/build/memory/jemalloc/build-crt.py", line 10, in <module>
    cwd=sys.argv[1])
  File "d:\mozilla-build\python25\lib\subprocess.py", line 593, in __init__
    errread, errwrite)
  File "d:\mozilla-build\python25\lib\subprocess.py", line 793, in _execute_child
    startupinfo)
WindowsError: [Error 22] The directory name is invalid
make[5]: *** [src/build/intel/mozcrt19.dll] Error 1
make[4]: *** [libs_tier_base] Error 2
make[3]: *** [tier_base] Error 2
make[2]: *** [default] Error 2
make[1]: *** [build] Error 2
make: *** [profiledbuild] Error 2

I backed the patch out:

http://hg.mozilla.org/releases/mozilla-1.9.1/rev/fa0601d54205
Comment on attachment 373825 [details] [diff] [review]
hack build-crt.py to elide these errors from the log

Removing .12 approval as this didn't land before freeze. Feel free to nominate again, though the bar for approval will be higher.
Attachment #373825 - Flags: approval1.9.2.9+
Attachment #373825 - Flags: approval1.9.1.12-
Attachment #373825 - Flags: approval1.9.1.12+
Duplicate of this bug: 416125
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.