794490 - When Windows builds fail, sometimes pymake returns -127 and no compiler output is seen

Status: RESOLVED DUPLICATE of bug 799189

(Firefox Build System :: General, defect)

Description

[Randell Jesup [:jesup] (needinfo me)]

Possibly the same or related to bug 787658

When building with pymake, fairly often when a compile fails on windows you see "return code -127" and see no compiler output.

See the URL for an example.  It's often quite repeatable for a particular failure; I did 5 runs on my local machine hitting it last weekend to see if it was flakey; all failed.

Comment 1

[(not currently active) Ted Mielczarek]

I meant to file this, but I totally forgot. glandium said he thought it was a race condition between the output being buffered and the process exit.

Comment 2

[(not currently active) Ted Mielczarek]

A previous example on alder:
https://tbpl.mozilla.org/php/getParsedLog.php?id=15293672&tree=Alder

the same failure on Win64 with the error output:
https://tbpl.mozilla.org/php/getParsedLog.php?id=15293711&tree=Alder

Comment 3

[Mike Hommey [:glandium]]

(In reply to Ted Mielczarek [:ted] from comment #1)
> I meant to file this, but I totally forgot. glandium said he thought it was
> a race condition between the output being buffered and the process exit.

I was mentioning something else, but that could certainly be the same root problem. We really need to unbuffer stdout and stderr. The two solutions to the same kind of problem in bug 483100 was either
1) stick -u in the hash-bang line at the top of the file
2) sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0) (goofy reopening of stdout)

In the case of pymake, unless we re-execute python, only the latter would work (while the former was used for bug 483100).

Comment 4

[Mike Hommey [:glandium]]

Attachment: Don't buffer stdout and stderr in pymake

Comment 5

[Rain [:rain1] (inactive)]

That should be inside the if name == '__main__' section.

Comment 6

[(not currently active) Ted Mielczarek]

Comment on attachment 665414 [details] [diff] [review]
Don't buffer stdout and stderr in pymake

Review of attachment 665414 [details] [diff] [review]:
-----------------------------------------------------------------

r=me with Sid's suggestion.

Comment 7

[Mike Hommey [:glandium]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/dc70ce87180c

Comment 8

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/dc70ce87180c

Comment 9

[Randell Jesup [:jesup] (needinfo me)]

Still fails:
https://tbpl.mozilla.org/?tree=Try&rev=9f97d7fdf3d6
ekr verified that the pymake in his push has the change.

Comment 10

[Mike Hommey [:glandium]]

A likely culprit is http://mxr.mozilla.org/mozilla-central/source/build/cl.py#26

I had a try build systematically not showing errors, and removing the cl.py wrapping [1] made them show errors properly.

1. http://mxr.mozilla.org/mozilla-central/source/config/config.mk#129

Comment 11

[Rain [:rain1] (inactive)]

Hm, quite possibly.

Comment 12

[Gregory Szorc [:gps]]

I filed bug 799189 for cl.py foo. I just realized this was reopened. It's been a long week.

Comment 15

[Rain [:rain1] (inactive)]

I just backed bug 585011 out, so this shouldn't be an issue on tinderboxes any more.

Comment 16

[Randell Jesup [:jesup] (needinfo me)]

Can we move forward on this?  Just got another confused dev hitting it (or some variant) in bug 814619

Comment 17

[(not currently active) Ted Mielczarek]

With the backout of bug 585011 this shouldn't be an issue.

Comment 18

[Gregory Szorc [:gps]]

Between comment 17 and bug 799189 switching cl.py to use mozprocess, I'm pretty sure this is not an issue and never will be.