Intermittent gmake[4]: *** [stage-cppunittests] Error 1 due to a "File truncated"

RESOLVED FIXED in Firefox 53

Status

defect
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: intermittent-bug-filer, Assigned: cmanchester)

Tracking

({intermittent-failure, regression})

Trunk
mozilla53
Dependency tree / graph

Firefox Tracking Flags

(firefox50 unaffected, firefox51 unaffected, firefox52 unaffected, firefox53 fixed)

Details

Attachments

(1 attachment)

I'm going to guess this is a regression from bug 1323581. Fortunately, the frequency looks pretty low.
Depends on: 1323581
Flags: needinfo?(cmanchester)
Keywords: regression
Blocks: 1323581
No longer depends on: 1323581
Version: unspecified → Trunk
this is primarily on android.  I believe Chris will be back next week- thanks for looking at this :gps.
I think I see the issue here. Testing a fix on try.

There's a step in symbol dumping where we modify the original binary. My hypothesis is that's happening for a cppunittest program exactly as we attempt to copy/strip it for the sake of the test archive, causing the latter to fail.
Assignee: nobody → cmanchester
Flags: needinfo?(cmanchester)
I wasn't able to reproduce this in a reasonable number of re-triggers on try, but I think this patch is worth landing to test the hypothesis.
Comment on attachment 8823476 [details]
Bug 1325801 - Fix intermittent failure staging cppunittests.

https://reviewboard.mozilla.org/r/102000/#review102612

I think this is fine to land, though most of the automation dependencies are here: https://dxr.mozilla.org/mozilla-central/rev/57ac9f63fc6953f4efeb0cc84a60192d3721251f/build/moz-automation.mk#62

We could add a similar rule, like automation/package-tests: automation/buildsymbols

That might be too big a hammer, though - I think your approach lets these steps operate mostly still in parallel.

If this does end up fixing the issue, is there a way to fix it in symbolstore.py instead? Maybe there's an easy way to ignore files staged by package-tests.

::: testing/testsuite-targets.mk:280
(Diff revision 1)
>  
>  ifdef OBJCOPY
>  ifneq ($(OBJCOPY), :) # see build/autoconf/toolchain.m4:102 for why this is necessary
>  ifndef PKG_SKIP_STRIP
>  STRIP_CPP_TESTS := 1
> +generatesymbols: stage-cppunittests

We'll need a comment here, since it's not really obvious what's going on (especially since generatesymbols is actually in the top-level Makefile).
Attachment #8823476 - Flags: review?(mshal) → review+
Comment on attachment 8823476 [details]
Bug 1325801 - Fix intermittent failure staging cppunittests.

https://reviewboard.mozilla.org/r/102000/#review102612

Yeah, if we run package-tests and buildsymbols entirely in serial we're going to give up a lot of speedup due to parallelism.

I don't see a straightforward way to do this is symbolstore.py -- I don't think the issue isn't that we're dumping symbols for binaries we shouldn't, the issue if I'm right is that we're modifying the binaries (with `objcopy --add-gnu-debuglink=...`) exactly as package-tests is trying copy/strip them.

> We'll need a comment here, since it's not really obvious what's going on (especially since generatesymbols is actually in the top-level Makefile).

Agreed.
Keywords: leave-open
Pushed by cmanchester@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/e0378b0c798e
Fix intermittent failure staging cppunittests. r=mshal
Backout by kwierso@gmail.com:
https://hg.mozilla.org/integration/autoland/rev/fa441ec96615
Backed out changeset e0378b0c798e for bustage a=backout
This introduces a new failure that's far more frequent, because it causes stage-cppunittests to run twice. I didn't remember this when writing the patch, but we're invoking each tier as a subprocess in moz-automation.mk.
Getting all these tiers into the same make process is kind of a mess, but Mike's original suggestion of adding `automation/package-tests: automation/buildsymbols` isn't a complete loss in terms of performance.
See Also: → 1329020
Attachment #8823476 - Flags: review+ → review?(mshal)
Comment on attachment 8823476 [details]
Bug 1325801 - Fix intermittent failure staging cppunittests.

https://reviewboard.mozilla.org/r/102000/#review103590

::: build/moz-automation.mk:84
(Diff revision 3)
>  automation/upload: automation/sdk
>  
> +# buildsymbols will modify our test binaries, which can interfere with
> +# packaging them. A finer-grained dependency can help performance here
> +# once bug 1329020 is fixed.
> +automation/package-tests: automation/buildsymbols

Looking at the rule below I think that this should probably be:

automation/pretty-package-tests: automation/buildsymbols

Otherwise the pretty version of package-tests will still collide with buildsymbols.

Though now that I look at it even more closely, I'm not sure we actually set MOZ_AUTOMATION_PRETTY=1 anywhere, so I think all the pretty stuff might be dead code. Can you confirm that? If so I can try to remove it in a followup.
Attachment #8823476 - Flags: review?(mshal) → review+
Comment on attachment 8823476 [details]
Bug 1325801 - Fix intermittent failure staging cppunittests.

https://reviewboard.mozilla.org/r/102000/#review103590

> Looking at the rule below I think that this should probably be:
> 
> automation/pretty-package-tests: automation/buildsymbols
> 
> Otherwise the pretty version of package-tests will still collide with buildsymbols.
> 
> Though now that I look at it even more closely, I'm not sure we actually set MOZ_AUTOMATION_PRETTY=1 anywhere, so I think all the pretty stuff might be dead code. Can you confirm that? If so I can try to remove it in a followup.

I also can't find any hits for `MOZ_AUTOMATION_PRETTY` getting set in searchfox, but I'll change this as you suggest before landing.
Comment on attachment 8823476 [details]
Bug 1325801 - Fix intermittent failure staging cppunittests.

https://reviewboard.mozilla.org/r/102000/#review103590

> I also can't find any hits for `MOZ_AUTOMATION_PRETTY` getting set in searchfox, but I'll change this as you suggest before landing.

Filed bug 1329355 on looking in to removing the pretty- targets.
Pushed by cmanchester@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1d126c201b0f
Fix intermittent failure staging cppunittests. r=mshal
https://hg.mozilla.org/mozilla-central/rev/1d126c201b0f
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla53
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.