BUILDSTATUS messages may be interleaved with other output, confusing mach

RESOLVED FIXED in mozilla31

Status

()

Core
Build Config
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: glandium, Assigned: glandium)

Tracking

Trunk
mozilla31
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments, 2 obsolete attachments)

(Assignee)

Description

4 years ago
Filing in build config, but this is a joint bug with mach. Currently, it's not very likely to happen, because there aren't too many directories for which we print TIERDIR_START. With my wip for bug 907365, I made it such that every directory (or almost every) prints it, because, really, in a world where everything can be built in parallel, there's no reason not to.

The problem is that some of those TIERDIR_START/TIERDIR_STOP messages are interleaved with build logs from other make processes, most usually with compiler warnings. And mach doesn't catch them.

So it ends up failing with a backtrace like:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 808, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 761, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/mh/mozilla-central/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 698, in _processOutput
    self.processOutputLine(line.rstrip())
  File "/home/mh/mozilla-central/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 663, in processOutputLine
    handler(line)
  File "/home/mh/mozilla-central/python/mach/mach/mixin/process.py", line 86, in handleLine
    line_handler(line)
  File "/home/mh/mozilla-central/python/mozbuild/mozbuild/mach_commands.py", line 261, in on_line
    warning, state_changed, relevant = self.monitor.on_line(line)
  File "/home/mh/mozilla-central/python/mozbuild/mozbuild/controller/building.py", line 350, in on_line
    self.tiers.finish_dir(tier, subtier, d)
  File "/home/mh/mozilla-central/python/mozbuild/mozbuild/controller/building.py", line 152, in finish_dir
    self.active_dirs[subtier].remove(d)
KeyError: u'netwerk/streamconv/converters'

I think the best thing to do is to change BUILDSTATUS to do echo >&$(MACH_FD) and have mach set up a pipe on that MACH_FD and pass its number. That would, however make pymake unhappy, so we may want a native command for pymake.
(Assignee)

Comment 1

4 years ago
Using a separate fd, while the most definitive way to avoid the issue, requires fiddling deep inside mozprocess.processhandler :(
(Assignee)

Comment 2

4 years ago
So, in fact, it looks like redirecting BUILDSTATUS messages to stderr works around the issue, without using a third file handle.
(Assignee)

Comment 3

4 years ago
(In reply to Mike Hommey [:glandium] from comment #2)
> So, in fact, it looks like redirecting BUILDSTATUS messages to stderr works
> around the issue, without using a third file handle.

Interestingly, the resource monitor is still confused...
(Assignee)

Comment 4

4 years ago
So, in fact, that doesn't really pan out. I was just lucky that it worked once.
Perhaps pymake should be coordinating writes to stdout/stderr via a lock or having all line-buffered output piped to a common process.
(Assignee)

Comment 6

4 years ago
Absolutely none of the last dozen builds i've done have working resource-usage info. All end with some variant of:
  Exception when writing resource usage file: u'compile_nspr'
with the key varying.
(Assignee)

Comment 7

4 years ago
Created attachment 8396876 [details] [diff] [review]
Remove directory tracking from build system

It's not used except with pymake, anyways.
Attachment #8396876 - Flags: review?(gps)
(Assignee)

Updated

4 years ago
Assignee: nobody → mh+mozilla
Status: NEW → ASSIGNED
(Assignee)

Comment 8

4 years ago
Created attachment 8396877 [details] [diff] [review]
Remove subtier tracking from build system

It's currently inaccurate, and soon, the very notion of subtier is going to
blow away in the compile/binaries targets.
Attachment #8396877 - Flags: review?(gps)
Attachment #8396876 - Flags: review?(gps) → review+
Comment on attachment 8396877 [details] [diff] [review]
Remove subtier tracking from build system

Review of attachment 8396877 [details] [diff] [review]:
-----------------------------------------------------------------

Please verify `mach resource-usage` still works with dirs and subtiers removed.
Attachment #8396877 - Flags: review?(gps) → review+
(Assignee)

Comment 10

4 years ago
Created attachment 8396891 [details] [diff] [review]
Remove directory tracking from build system

With resource-usage fixed.
Attachment #8396891 - Flags: review?(gps)
(Assignee)

Updated

4 years ago
Attachment #8396876 - Attachment is obsolete: true
(Assignee)

Comment 11

4 years ago
Created attachment 8396893 [details] [diff] [review]
Remove subtier tracking from build system

With resource-usage fixed.
Attachment #8396893 - Flags: review?(gps)
(Assignee)

Updated

4 years ago
Attachment #8396877 - Attachment is obsolete: true
Attachment #8396891 - Flags: review?(gps) → review+
Attachment #8396893 - Flags: review?(gps) → review+
(Assignee)

Comment 12

4 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/870f0aa64918
https://hg.mozilla.org/integration/mozilla-inbound/rev/f204394cac0c
(Assignee)

Comment 13

4 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/0102998866ac
https://hg.mozilla.org/mozilla-central/rev/870f0aa64918
https://hg.mozilla.org/mozilla-central/rev/f204394cac0c
https://hg.mozilla.org/mozilla-central/rev/0102998866ac
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla31
(Assignee)

Updated

3 years ago
Blocks: 1041960
You need to log in before you can comment on or make changes to this bug.