BUILDSTATUS messages may be interleaved with other output, confusing mach

RESOLVED FIXED in mozilla31

Status

defect
RESOLVED FIXED
6 years ago
2 years ago

People

(Reporter: glandium, Assigned: glandium)

Tracking

Trunk
mozilla31
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments, 2 obsolete attachments)

Filing in build config, but this is a joint bug with mach. Currently, it's not very likely to happen, because there aren't too many directories for which we print TIERDIR_START. With my wip for bug 907365, I made it such that every directory (or almost every) prints it, because, really, in a world where everything can be built in parallel, there's no reason not to.

The problem is that some of those TIERDIR_START/TIERDIR_STOP messages are interleaved with build logs from other make processes, most usually with compiler warnings. And mach doesn't catch them.

So it ends up failing with a backtrace like:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 808, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 761, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/mh/mozilla-central/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 698, in _processOutput
    self.processOutputLine(line.rstrip())
  File "/home/mh/mozilla-central/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 663, in processOutputLine
    handler(line)
  File "/home/mh/mozilla-central/python/mach/mach/mixin/process.py", line 86, in handleLine
    line_handler(line)
  File "/home/mh/mozilla-central/python/mozbuild/mozbuild/mach_commands.py", line 261, in on_line
    warning, state_changed, relevant = self.monitor.on_line(line)
  File "/home/mh/mozilla-central/python/mozbuild/mozbuild/controller/building.py", line 350, in on_line
    self.tiers.finish_dir(tier, subtier, d)
  File "/home/mh/mozilla-central/python/mozbuild/mozbuild/controller/building.py", line 152, in finish_dir
    self.active_dirs[subtier].remove(d)
KeyError: u'netwerk/streamconv/converters'

I think the best thing to do is to change BUILDSTATUS to do echo >&$(MACH_FD) and have mach set up a pipe on that MACH_FD and pass its number. That would, however make pymake unhappy, so we may want a native command for pymake.
Using a separate fd, while the most definitive way to avoid the issue, requires fiddling deep inside mozprocess.processhandler :(
So, in fact, it looks like redirecting BUILDSTATUS messages to stderr works around the issue, without using a third file handle.
(In reply to Mike Hommey [:glandium] from comment #2)
> So, in fact, it looks like redirecting BUILDSTATUS messages to stderr works
> around the issue, without using a third file handle.

Interestingly, the resource monitor is still confused...
So, in fact, that doesn't really pan out. I was just lucky that it worked once.
Perhaps pymake should be coordinating writes to stdout/stderr via a lock or having all line-buffered output piped to a common process.
Absolutely none of the last dozen builds i've done have working resource-usage info. All end with some variant of:
  Exception when writing resource usage file: u'compile_nspr'
with the key varying.
It's not used except with pymake, anyways.
Attachment #8396876 - Flags: review?(gps)
Assignee: nobody → mh+mozilla
Status: NEW → ASSIGNED
It's currently inaccurate, and soon, the very notion of subtier is going to
blow away in the compile/binaries targets.
Attachment #8396877 - Flags: review?(gps)
Attachment #8396876 - Flags: review?(gps) → review+
Comment on attachment 8396877 [details] [diff] [review]
Remove subtier tracking from build system

Review of attachment 8396877 [details] [diff] [review]:
-----------------------------------------------------------------

Please verify `mach resource-usage` still works with dirs and subtiers removed.
Attachment #8396877 - Flags: review?(gps) → review+
With resource-usage fixed.
Attachment #8396891 - Flags: review?(gps)
Attachment #8396876 - Attachment is obsolete: true
With resource-usage fixed.
Attachment #8396893 - Flags: review?(gps)
Attachment #8396877 - Attachment is obsolete: true
Attachment #8396891 - Flags: review?(gps) → review+
Attachment #8396893 - Flags: review?(gps) → review+
Blocks: 1041960
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.