1176638 - All opt tests will permafail when Gecko 41 merges to Beta

Status: RESOLVED WORKSFORME

(Testing :: General, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

Not sure what to make of this - I'm hoping the resident experts will have a better sense of what's going wrong here. This affects both the API9 and API11+ builds, but 4.3 debug is green, interestingly.
https://treeherder.mozilla.org/logviewer.html#?job_id=8741847&repo=try

Also interesting, the reftests appear to be hitting the non-local network crash trying to connect to location.services.mozilla.com.
https://treeherder.mozilla.org/logviewer.html#?job_id=8741841&repo=try

Calling this critical since it blocks being able to see if there are any other underlying Android issues lurking behind this.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

This is more widespread. OSX has the same issue (and a more-useful error from the looks of it):

https://treeherder.mozilla.org/logviewer.html#?job_id=8742639&repo=try

10:30:27 INFO - JavaScript error: http://mochi.test:8888/tests/SimpleTest/setup.js, line 110: ReferenceError: SpecialPowers is not defined
10:30:27 WARNING - TEST-UNEXPECTED-FAIL: setup.js | error parsing http://mochi.test:8888/tests.json (TypeError: RunSet is undefined)
10:30:27 INFO - JavaScript error: http://mochi.test:8888/tests/SimpleTest/setup.js, line 256: TypeError: RunSet is undefined
10:35:57 WARNING - TEST-UNEXPECTED-TIMEOUT | automation.py | application timed out after 330 seconds with no output

OSX reftests are hitting the same non-local crashes as well:
https://treeherder.mozilla.org/logviewer.html#?job_id=8742635&repo=try

And talos is hitting non-local crashes on snippets.cdn.mozilla.net:
https://treeherder.mozilla.org/logviewer.html#?job_id=8742684&repo=try

Comment 2

[Jonathan Griffin (:jgriffin)]

There seem to be several separate issues here.

The mochitest issue seems to be caused by lack of SpecialPowers, which may be some kind of packaging issue?

10:30:27 INFO - JavaScript error: http://mochi.test:8888/tests/SimpleTest/setup.js, line 110: ReferenceError: SpecialPowers is not defined

Both reftests and talos are failing with different url's involved in non-local crashes.

And the Android test problem seems to different as well.  needinfo'ing gbrown for the latter.

Comment 3

[Jonathan Griffin (:jgriffin)]

The mochitest issue seems related to:

10:30:22 INFO - 1434735022586 addons.xpi WARN Refusing to install staged add-on [e-mail] with signed state 0

Have we already enabled the requirement that addons must be signed?  This will definitely break all tests until we have a way to sign these.

Comment 4

[Geoff Brown [:gbrown]]

For Android, everything seems to point back to "Refusing to install staged add-on..."

Comment 5

[Lawrence Mandel [:lmandel] (use needinfo)]

ni Kev and mconnor to answer the question in comment 3.

Comment 7

[Ryan VanderMeulen [:RyanVM]]

Which is what bug 1132995 is for.

Comment 8

[Kev Needham [:kev]]

What Ryan said. The issue was raised by RelEng a couple months ago, and catlee and kmoir are on it. My understanding is that for 40, tests with unsigned addons will be performed with the pref off, but with Fx41 and forward there will be a solution in place to deal with signing requirements. Asking catlee and kmoir for more info.

Comment 9

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

(In reply to Geoff Brown [:gbrown] from comment #4)
> For Android, everything seems to point back to "Refusing to install staged
> add-on..."

Probably not bug 1175915, but posted just in case. I think add-ons don't need to use Android Intents and so it should be unrelated.

Comment 10

[Kim Moir [:kmoir] ET]

Regarding comment #8

I talked to Jordan (he'll also be working on this in Q3) today and we went over the plan so far.  We also talked to catlee and RyanVM.

The first step is to create builds with the build time option to disable requiring signed add ons in custom mozconfigs.  We could use Linux64 as a proof on concept platform and then enable for other platforms.

As part of the merge process to beta, we will push signed addons used in tests in tree so that the could be used for the regular builds.  Will coordinate with the addons team to sign them for us.  One possiblity is to store these binaries in tooltool.  This will be referenced in tree as part of the beta uplift. There are some addons that are generated in tree for testing purposes.  These could be disabled on the mozilla-beta and mozilla-release branches (according to catlee this is makefile hacking).  These in tree generated addons should be tested by the builds that have the signed addon preference disabled. So the plan is to have the testing switched to the builds with the signed addons buildtime preference disabled.  However, we'll ship the builds with the buildtime preference enabled that only allows signed addons to be installed. In summary

1) Generate new builds with option disabled on m-b and riding trains to m-r
2) Get addons used for testing signed by addons team, referenced in tree during uplift to beta
3) Store signed addons in tooltool? TBD
4) Run tests on new builds
5) Switch tests to only run on new builds on m-b and m-r
6) Actually ship build binaries with requirement for signed addons enabled