Closed Bug 1228742 Opened 4 years ago Closed 4 years ago

Intermittent mozpack.errors.ErrorMessage: Error: Error while running startup cache precompilation

Categories

(Firefox Build System :: General, defect, blocker)

defect
Not set
blocker

Tracking

(firefox45 fixed)

RESOLVED FIXED
mozilla45
Tracking Status
firefox45 --- fixed

People

(Reporter: aryx, Assigned: glandium)

References

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

OS X 10.7 opt https://treeherder.mozilla.org/logviewer.html#?job_id=5958784&repo=fx-team

01:23:32     INFO -  resource://gre/modules/CrashReports.jsm
01:23:32     INFO -  resource://gre/modules/CrashSubmit.jsm
01:23:32     INFO -  resource://gre/modules/CrashTestUtils.jsm
01:24:22     INFO -  Traceback (most recent call last):
01:24:22     INFO -    File "/builds/slave/fx-team-m64-000000000000000000/build/src/toolkit/mozapps/installer/packager.py", line 406, in <module>
01:24:22     INFO -      main()
01:24:22     INFO -    File "/builds/slave/fx-team-m64-000000000000000000/build/src/toolkit/mozapps/installer/packager.py", line 400, in main
01:24:22     INFO -      args.source, gre_path, base)
01:24:22     INFO -    File "/builds/slave/fx-team-m64-000000000000000000/build/src/toolkit/mozapps/installer/packager.py", line 161, in precompile_cache
01:24:22     INFO -      errors.fatal('Error while running startup cache precompilation')
01:24:22     INFO -    File "/builds/slave/fx-team-m64-000000000000000000/build/src/python/mozbuild/mozpack/errors.py", line 103, in fatal
01:24:22     INFO -      self._handle(self.FATAL, msg)
01:24:22     INFO -    File "/builds/slave/fx-team-m64-000000000000000000/build/src/python/mozbuild/mozpack/errors.py", line 98, in _handle
01:24:22     INFO -      raise ErrorMessage(msg)
01:24:22     INFO -  mozpack.errors.ErrorMessage: Error: Error while running startup cache precompilation
01:24:22     INFO -  make[3]: *** [stage-package] Error 1
01:24:22     INFO -  make[2]: *** [postflight_all] Error 2
01:24:22     INFO -  make[1]: *** [realbuild] Error 2
01:24:22     INFO -  make: *** [build] Error 2
01:24:22     INFO -  349 compiler warnings present.
Nobody ever wants to be dragged into a startup cache precompilation failure, a land war in Southeast Asia is a much better bet than a crash with no stack, but it's hard not to notice that this has happened on six of twelve builds while bug 1209184 has been in the tree, and zero of infinity builds while it hasn't been in. I retriggered a few in https://treeherder.mozilla.org/#/jobs?repo=fx-team&revision=2e446ebafe6c&filter-searchStr=553af93acd4e756703af30dba01c32d142665fb8 to see if I can make it happen while that was backed out.

fx-team is closed, so we don't just blindly merge this around.
Severity: normal → blocker
It's very strange that this only happens on OS-X opt. My best guess is that those machines need a clobber or a ccache flush. I can't think of any reason those changesets would break OS-X but not other platforms.
I retriggered this a few of times on the latest fx-team checkin, and all passed. Of the 8 builds since I re-landed, only two have failed.
The usual reason for only-OS-X-opt is universal builds, though that also doesn't explain why you or why failing like this, since that usually leads to packaging failures. Since this failure does produce one of the same lines of failure message as the known packaging failure when someone removes a jarred file, they've actually been clobbered twice since the first of these failure happened.
I must just be luckier at retriggering them: out of 10 more retriggers on the tip, I got 2 more failures, and out of 10 more retriggers on the previous time I backed you out, I got 0 more failures. Then I realized what a crazy number of tests we're running doing this on fx-team, so I pushed the tip of fx-team to try, backed you out again, and pushed that to try, https://treeherder.mozilla.org/#/jobs?repo=try&revision=f0f238c69d15,9147b65fd122
Yeah, I'd already moved my tests to try. I was considering backing this out myself, though I'm still not sure how it could be causing the breakage.

I was able to reproduce this on try with two of the changesets backed out, leaving only https://hg.mozilla.org/try/rev/f191e66462b7 and https://hg.mozilla.org/try/rev/addfec116af7, but that makes even less sense to me than before.

The startup cache precompilation seems to be consistently failing after loading resource://gre/modules/CrashTestUtils.jsm, so I would expect to see crash test failures in some of the builds that succeed, but there aren't any.

I rebased all of the changesets on top of m-c, and so far 5 builds have completed without failing, but I started another 5 in case it was just luck.
Interestingly, I figured yesterday that CrashTestUtils.jsm is shipped by mistake, and really shouldn't. That would have the side effect of "fixing" this, if that is really the cause.
Assignee: nobody → mh+mozilla
Attachment #8693325 - Flags: review?(mshal)
(In reply to Mike Hommey [:glandium] from comment #7)
> Interestingly, I figured yesterday that CrashTestUtils.jsm is shipped by
> mistake, and really shouldn't. That would have the side effect of "fixing"
> this, if that is really the cause.

That would definitely be nice. So far I have this narrowed down to "this is triggered by the presence of a piece of code even when it doesn't actually run". But finding out exactly which piece is slow going, since I'm stuck bisecting commented out sections of code and running a bunch of builds to see if they crash, until I can reproduce this locally.
Attachment #8693325 - Flags: review?(mshal) → review+
https://hg.mozilla.org/mozilla-central/rev/ea00141d4b5a
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla45
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.