1228742 - Intermittent mozpack.errors.ErrorMessage: Error: Error while running startup cache precompilation

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

OS X 10.7 opt https://treeherder.mozilla.org/logviewer.html#?job_id=5958784&repo=fx-team

01:23:32     INFO -  resource://gre/modules/CrashReports.jsm
01:23:32     INFO -  resource://gre/modules/CrashSubmit.jsm
01:23:32     INFO -  resource://gre/modules/CrashTestUtils.jsm
01:24:22     INFO -  Traceback (most recent call last):
01:24:22     INFO -    File "/builds/slave/fx-team-m64-000000000000000000/build/src/toolkit/mozapps/installer/packager.py", line 406, in <module>
01:24:22     INFO -      main()
01:24:22     INFO -    File "/builds/slave/fx-team-m64-000000000000000000/build/src/toolkit/mozapps/installer/packager.py", line 400, in main
01:24:22     INFO -      args.source, gre_path, base)
01:24:22     INFO -    File "/builds/slave/fx-team-m64-000000000000000000/build/src/toolkit/mozapps/installer/packager.py", line 161, in precompile_cache
01:24:22     INFO -      errors.fatal('Error while running startup cache precompilation')
01:24:22     INFO -    File "/builds/slave/fx-team-m64-000000000000000000/build/src/python/mozbuild/mozpack/errors.py", line 103, in fatal
01:24:22     INFO -      self._handle(self.FATAL, msg)
01:24:22     INFO -    File "/builds/slave/fx-team-m64-000000000000000000/build/src/python/mozbuild/mozpack/errors.py", line 98, in _handle
01:24:22     INFO -      raise ErrorMessage(msg)
01:24:22     INFO -  mozpack.errors.ErrorMessage: Error: Error while running startup cache precompilation
01:24:22     INFO -  make[3]: *** [stage-package] Error 1
01:24:22     INFO -  make[2]: *** [postflight_all] Error 2
01:24:22     INFO -  make[1]: *** [realbuild] Error 2
01:24:22     INFO -  make: *** [build] Error 2
01:24:22     INFO -  349 compiler warnings present.

Comment 1

[Phil Ringnalda (:philor)]

Nobody ever wants to be dragged into a startup cache precompilation failure, a land war in Southeast Asia is a much better bet than a crash with no stack, but it's hard not to notice that this has happened on six of twelve builds while bug 1209184 has been in the tree, and zero of infinity builds while it hasn't been in. I retriggered a few in https://treeherder.mozilla.org/#/jobs?repo=fx-team&revision=2e446ebafe6c&filter-searchStr=553af93acd4e756703af30dba01c32d142665fb8 to see if I can make it happen while that was backed out.

fx-team is closed, so we don't just blindly merge this around.

Comment 2

[Kris Maglione [:kmag]]

It's very strange that this only happens on OS-X opt. My best guess is that those machines need a clobber or a ccache flush. I can't think of any reason those changesets would break OS-X but not other platforms.

Comment 3

[Kris Maglione [:kmag]]

I retriggered this a few of times on the latest fx-team checkin, and all passed. Of the 8 builds since I re-landed, only two have failed.

Comment 4

[Phil Ringnalda (:philor)]

The usual reason for only-OS-X-opt is universal builds, though that also doesn't explain why you or why failing like this, since that usually leads to packaging failures. Since this failure does produce one of the same lines of failure message as the known packaging failure when someone removes a jarred file, they've actually been clobbered twice since the first of these failure happened.

Comment 5

[Phil Ringnalda (:philor)]

I must just be luckier at retriggering them: out of 10 more retriggers on the tip, I got 2 more failures, and out of 10 more retriggers on the previous time I backed you out, I got 0 more failures. Then I realized what a crazy number of tests we're running doing this on fx-team, so I pushed the tip of fx-team to try, backed you out again, and pushed that to try, https://treeherder.mozilla.org/#/jobs?repo=try&revision=f0f238c69d15,9147b65fd122

Comment 6

[Kris Maglione [:kmag]]

Yeah, I'd already moved my tests to try. I was considering backing this out myself, though I'm still not sure how it could be causing the breakage.

I was able to reproduce this on try with two of the changesets backed out, leaving only https://hg.mozilla.org/try/rev/f191e66462b7 and https://hg.mozilla.org/try/rev/addfec116af7, but that makes even less sense to me than before.

The startup cache precompilation seems to be consistently failing after loading resource://gre/modules/CrashTestUtils.jsm, so I would expect to see crash test failures in some of the builds that succeed, but there aren't any.

I rebased all of the changesets on top of m-c, and so far 5 builds have completed without failing, but I started another 5 in case it was just luck.

Comment 7

[Mike Hommey [:glandium]]

Interestingly, I figured yesterday that CrashTestUtils.jsm is shipped by mistake, and really shouldn't. That would have the side effect of "fixing" this, if that is really the cause.

Comment 8

[Mike Hommey [:glandium]]

Attachment: Don't install CrashTestUtils.jsm to dist/bin/modules

Comment 9

[Intermittent Failures Robot]

6 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* fx-team: 6

Platform breakdown:
* osx-10-7: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1228742&startday=2015-11-23&endday=2015-11-29&tree=all

Comment 10

[Kris Maglione [:kmag]]

(In reply to Mike Hommey [:glandium] from comment #7)
> Interestingly, I figured yesterday that CrashTestUtils.jsm is shipped by
> mistake, and really shouldn't. That would have the side effect of "fixing"
> this, if that is really the cause.

That would definitely be nice. So far I have this narrowed down to "this is triggered by the presence of a piece of code even when it doesn't actually run". But finding out exactly which piece is slow going, since I'm stuck bisecting commented out sections of code and running a bunch of builds to see if they crash, until I can reproduce this locally.

Comment 11

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/ea00141d4b5a

Comment 12

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/ea00141d4b5a

Comment 13

[Kris Maglione [:kmag]]

This seems to have solved my problem: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b31975968135

Thanks!