1393156 - Various extension xpcshell tests are going to permafail when Gecko 57 merges to Beta on 2017-09-20

Status: VERIFIED FIXED

(Toolkit :: Add-ons Manager, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

[Tracking Requested - why for this release]: Permafailing tests on the next merge day.

This was expected fallout from bug 1363897. The logs have some relevant-looking information:

INFO -  "CONSOLE_MESSAGE: (warn) [JavaScript Warning: "ReferenceError: reference to undefined property "applySoftBlock"" {file: "resource://gre/modules/addons/XPIProvider.jsm" line: 5024}]"
INFO -  "CONSOLE_MESSAGE: (info) 1503509043826	addons.xpi	WARN	disabling legacy extension [e-mail]"

https://treeherder.mozilla.org/logviewer.html#?job_id=125230804&repo=try
https://treeherder.mozilla.org/logviewer.html#?job_id=125230862&repo=try
https://treeherder.mozilla.org/logviewer.html#?job_id=125230816&repo=try

Patch needed to reproduce:
https://hg.mozilla.org/try/rev/9701715e2023391ad98b3dbb9632f34c1b0c4347

Comment 1

[Andrew Swan [:aswan]]

Attachment: Bug 1393156 Remove obsolete theme compatbility override feature

Review commit: https://reviewboard.mozilla.org/r/172932/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/172932/

Comment 2

[Andrew Swan [:aswan]]

Attachment: Bug 1393156 Remove addons manager tests specific to complete themes

Review commit: https://reviewboard.mozilla.org/r/172934/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/172934/

Comment 3

[Andrew Swan [:aswan]]

Attachment: Bug 1393156 Update addon DB rebuilding test cases

Update a bunch of tests that cover rebuilding the addons database to use
webextension themes instead of complete themes.  Discovered bug 1394117
along the way and disabled a bunch of code pending fixing that bug.

Review commit: https://reviewboard.mozilla.org/r/172936/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/172936/

Comment 4

[Andrew Swan [:aswan]]

Attachment: Bug 1393156 Update signing tests

test_signed_migrate.js was about migrating from a build that does not
enforce signatures to one that does.  This migration happened a while
ago so we can just get rid of this test.
test_signed_inject.js uses signed legacy extensions and needs to be
udpated to use webextensions, this is bug 1394122.

Review commit: https://reviewboard.mozilla.org/r/172938/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/172938/

Comment 5

[Andrew Swan [:aswan]]

Attachment: Bug 1393156 Legacy extension test loose ends

Change webextensions experiments test to use the shimmed certficiate DB
instead of the extensions.legacy.enabled pref.
In builds that don't honor the extensions.legacy.enabled pref, disable
test_legacy.js since that tests that flipping that preference works properly.

Review commit: https://reviewboard.mozilla.org/r/172940/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/172940/

Comment 6

[Kris Maglione [:kmag]]

Comment on attachment 8901472 [details]
Bug 1393156 Remove obsolete theme compatbility override feature

https://reviewboard.mozilla.org/r/172932/#review178282

Yay, deleted tests!

Comment 7

[Kris Maglione [:kmag]]

Comment on attachment 8901473 [details]
Bug 1393156 Remove addons manager tests specific to complete themes

https://reviewboard.mozilla.org/r/172934/#review178284

Comment 8

[Kris Maglione [:kmag]]

Comment on attachment 8901474 [details]
Bug 1393156 Update addon DB rebuilding test cases

https://reviewboard.mozilla.org/r/172936/#review178286

Comment 9

[Kris Maglione [:kmag]]

Comment on attachment 8901475 [details]
Bug 1393156 Update signing tests

https://reviewboard.mozilla.org/r/172938/#review178288

Comment 10

[Kris Maglione [:kmag]]

Comment on attachment 8901476 [details]
Bug 1393156 Legacy extension test loose ends

https://reviewboard.mozilla.org/r/172940/#review178290

::: toolkit/components/extensions/test/xpcshell/test_ext_experiments.js:8
(Diff revision 1)
>  /* globals browser */
>  
>  XPCOMUtils.defineLazyModuleGetter(this, "AddonManager",
>                                    "resource://gre/modules/AddonManager.jsm");
>  
> +Cu.import("resource://testing-common/AddonTestUtils.jsm");

This should probably be a lazy import in head.js

Comment 11

[Andrew Swan [:aswan]]

Comment on attachment 8901472 [details]
Bug 1393156 Remove obsolete theme compatbility override feature

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/172932/diff/1-2/

Comment 12

[Andrew Swan [:aswan]]

Comment on attachment 8901473 [details]
Bug 1393156 Remove addons manager tests specific to complete themes

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/172934/diff/1-2/

Comment 13

[Andrew Swan [:aswan]]

Comment on attachment 8901474 [details]
Bug 1393156 Update addon DB rebuilding test cases

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/172936/diff/1-2/

Comment 14

[Andrew Swan [:aswan]]

Comment on attachment 8901475 [details]
Bug 1393156 Update signing tests

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/172938/diff/1-2/

Comment 15

[Andrew Swan [:aswan]]

Comment on attachment 8901476 [details]
Bug 1393156 Legacy extension test loose ends

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/172940/diff/1-2/

Comment 16

[Andrew Swan [:aswan]]

The latest patches seem to not regress regular central tests but I can't get a clean central-as-beta build, perhaps due to bug 1350432:
https://treeherder.mozilla.org/logviewer.html#?job_id=126275400&repo=try

RyanVM, I dunno if you're aware of the beta breakage but in any case, do you prefer that I land these as-is or wait until we can get a good test run?

Comment 17

[Ryan VanderMeulen [:RyanVM]]

I think you can go ahead and land. I'll certainly let you know if there's other issues remaining :)

Comment 18

[Andrew Swan [:aswan]]

Comment on attachment 8901476 [details]
Bug 1393156 Legacy extension test loose ends

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/172940/diff/2-3/

Comment 19

[Andrew Swan [:aswan]]

Comment on attachment 8901476 [details]
Bug 1393156 Legacy extension test loose ends

https://reviewboard.mozilla.org/r/172940/#review178290

> This should probably be a lazy import in head.js

How careless of me, it is already in head.js, just removed this line.

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/eb1d6805d426
Remove obsolete theme compatbility override feature r=kmag
https://hg.mozilla.org/integration/autoland/rev/0f29beef0059
Remove addons manager tests specific to complete themes r=kmag
https://hg.mozilla.org/integration/autoland/rev/379d8bff976a
Update addon DB rebuilding test cases r=kmag
https://hg.mozilla.org/integration/autoland/rev/1dd01508a4ca
Update signing tests r=kmag
https://hg.mozilla.org/integration/autoland/rev/5d0d4f72554c
Legacy extension test loose ends r=kmag

Comment 21

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/eb1d6805d426
https://hg.mozilla.org/mozilla-central/rev/0f29beef0059
https://hg.mozilla.org/mozilla-central/rev/379d8bff976a
https://hg.mozilla.org/mozilla-central/rev/1dd01508a4ca
https://hg.mozilla.org/mozilla-central/rev/5d0d4f72554c

Comment 22

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

This is fixed, no need to track it.

Comment 23

[Ryan VanderMeulen [:RyanVM]]

This is mostly fixed now, but there's still a failure lingering in test_signed_updatepref.js for DevEdition builds. I've filed bug 1394784 to track that.