1610758 - TPS no longer works as its extension appears to not load.

Status: RESOLVED FIXED

(Firefox :: Sync, defect)

Description

[Isabel Rios[:isabel_rios]]

Steps
We are launching our sync integration tests as defined here:
https://github.com/mozilla-mobile/fenix/blob/master/docs/syncIntegrationTests.md
An stage FxA account is created, then Nightly is launched and the sign in should happen and success

Actual
Nightly is launched, the FxA account is correctly created but the sign in never happens.
These are the only logs generated:

1579524367852	[e-mail]	WARN	Loading extension '[e-mail]': Reading manifest: Invalid extension permission: networkStatus
1579524367868	[e-mail]	WARN	Loading extension '[e-mail]': Reading manifest: Invalid extension permission: mozillaAddons
1579524367868	[e-mail]	WARN	Loading extension '[e-mail]': Reading manifest: Invalid extension permission: telemetry
1579524367868	[e-mail]	WARN	Loading extension '[e-mail]': Reading manifest: Invalid extension permission: resource://pdf.js/
1579524367868	[e-mail]	WARN	Loading extension '[e-mail]': Reading manifest: Invalid extension permission: about:reader*
1579524369630	Marionette	INFO	Listening on port 2828

This started to happen after Thursday 18th, detected on Monday 20th. Will work on narrowing down where it stopped working in case that helps

Comment 1

[Isabel Rios[:isabel_rios]]

So, the latest build where it worked was from Wed 15th. On Thurs 16th the build is broken.
Using mozregression this is the broken build from the 16th: https://archive.mozilla.org/pub/firefox/nightly/2020/01/2020-01-16-21-45-49-mozilla-central/firefox-74.0a1.en-US.mac.dmg

Will try to narrow down the commits in which the error is introduced.

Comment 2

[Isabel Rios[:isabel_rios]]

This is the info from Mozregression:
INFO: Last good revision: dc2fdf33488c23aa0daf2060fa86f73577d05d5a
INFO: First bad revision: 85753283ef2e4ff5cd5c726a2cfe12b8b4623b4e

And the link given: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=dc2fdf33488c23aa0daf2060fa86f73577d05d5a&tochange=85753283ef2e4ff5cd5c726a2cfe12b8b4623b4e

Hope this helps Mark.
Let me know if I can help with some more info. Thanks!

Comment 3

[Mark Hammond [:markh] [:mhammond]]

Thanks Isabel, bug 1524327 certainly sounds like the culprit. Our extension at https://searchfox.org/mozilla-central/source/services/sync/tps/extensions/tps doesn't load which is what is causing the problem. I'm not yet sure how we can work around this.

Comment 4

[Andrew Swan [:aswan]]

Should be explained here:
https://mail.mozilla.org/pipermail/firefox-dev/2020-January/007302.html

Comment 5

[Mark Hammond [:markh] [:mhammond]]

(In reply to Andrew Swan [:aswan] from comment #4)

Should be explained here:
https://mail.mozilla.org/pipermail/firefox-dev/2020-January/007302.html

Thanks for that, but that pref is already set to true here - or maybe you mean there's something else we need to do?

Comment 6

[Shane Caraveo (:mixedpuppy)]

:mhammond

Whether the pref is allowed to be set depends on this: https://searchfox.org/mozilla-central/rev/e878e5b81bb319c141900ce9cfcde732df5c8449/toolkit/mozapps/extensions/internal/AddonSettings.jsm#84

It shouldn't have changed anything for you. How do you run this tps extension?

Comment 7

[Mark Hammond [:markh] [:mhammond]]

(In reply to Shane Caraveo (:mixedpuppy) from comment #6)

It shouldn't have changed anything for you. How do you run this tps extension?

Some of this stuff is still opaque to me. IIUC, the extension is automatically loaded via the 'mozprofile' package at https://searchfox.org/mozilla-central/source/testing/tps/tps/testrunner.py#248. In theory, this means we end up somewhere like https://searchfox.org/mozilla-central/source/testing/mozbase/mozprofile/mozprofile/profile.py#131, although for reasons I don't understand, TPS grabs mozprofile etc from pypi rather than directly using the versions in the tree.

Comment 8

[Andrew Swan [:aswan]]

I just noticed that this report is for testing on Fenix. I don't know where the GeckoView mozconfig is (or if it even uses mozconfig!) but I wouldn't be surprised if it requires signing but doesn't define NIGHTLY_BUILD or MOZ_DEV_EDITION.
ni Nick to see if he can provide a little GeckoView build background, but I suspect GeckoView wasn't considered in bug 1524327 and the clause Shane referenced in comment 6 will need to be updated (if GeckoView is going to support experiments)

Comment 9

[Andrew Swan [:aswan]]

Whoops, forgot the ni. See previous comment.

Comment 10

[Mark Hammond [:markh] [:mhammond]]

(In reply to Andrew Swan [:aswan] from comment #8)

I just noticed that this report is for testing on Fenix.

While a "kind of" TPS is used on Fenix, I don't think this web extension is, nor the code I've been pointing at. IIUC, the Fenix side of this is somewhat manual. As mentioned in comment 0

(In reply to Isabel Rios[:isabel_rios] from comment #0)

Steps
We are launching our sync integration tests as defined here:
https://github.com/mozilla-mobile/fenix/blob/master/docs/syncIntegrationTests.md
An stage FxA account is created, then Nightly is launched and the sign in should happen and success

I believe "Nightly" here refers to Desktop.

Comment 11

[Isabel Rios[:isabel_rios]]

(In reply to Andrew Swan [:aswan] from comment #8)

I just noticed that this report is for testing on Fenix. I don't know where the GeckoView mozconfig is (or if it even uses mozconfig!) but I wouldn't be surprised if it requires signing but doesn't define NIGHTLY_BUILD or MOZ_DEV_EDITION.
ni Nick to see if he can provide a little GeckoView build background, but I suspect GeckoView wasn't considered in bug 1524327 and the clause Shane referenced in comment 6 will need to be updated (if GeckoView is going to support experiments)

Yes, as said above, Nightly refers to Desktop.
Fennix has nothing to do with this issue, the tests have two parts. First, launch Nightly Desktop and then Fenix. The failure is in the first part. We also have similar tests on Firefox-iOS that fail with the same cause so it is for sure nothing related to Android or iOS.

Comment 12

[Shane Caraveo (:mixedpuppy)]

It would be great if tps were simply loadable in about:debugging, but when I try to do that it basically quits firefox leaving it in a strange state.

Looking at the console output when I do this, I see that the extension is loading with the experimental api.

CROSSWEAVE ERROR: [phase -1] TPS no longer takes arguments from the command line. instead you need to pass preferences  `testing.tps.{testFile,testPhase}` and optionally `testing.tps.{logFile,ignoreUnusedEngines}`.
 - JS Stack trace: [e-mail]:220:32
[e-mail]:31:11
[e-mail]:88:5
onStartup/<@ExtensionCommon.jsm:1405:17
promise callback*[e-mail]:1403:62

I think the easiest initial debugging on this is [for someone who knows the tps test stuff and has a setup with it] to modify AddonSettings.jsm (where pointed out in comment 6) and add some debug output to see what is happening at that point, is the pref being used or the constant? If the pref is being used, then look into Extension.jsm and maybe add some debug output for these:

https://searchfox.org/mozilla-central/rev/f98dad153b59a985efd4505912588d4651033395/toolkit/components/extensions/Extension.jsm#1874-1885

Comment 13

[Nick Alexander :nalexander [he/him]]

(In reply to Andrew Swan [:aswan] from comment #9)

Whoops, forgot the ni. See previous comment.

Looks like the right people are on this. For the record, (some) published GV versions should have NIGHTLY_BUILD enabled, and therefore should have unsigned extension support. Generally we expect extensions to be packed into the APK that consumes GV, but I'm not sure they need to be signed -- even when using a release or beta GV version.

Comment 14

[Isabel Rios[:isabel_rios]]

Hey Mark, do you have more/new info about this bug and its eventual fix? Thanks!!

Comment 15

[Isabel Rios[:isabel_rios]]

Finally we got a fix for this issue.

In our framework that runs the tests we can modify the pref that changed causing this issue in this file (thanks to Dave Hunt who pointed that out):
https://github.com/mozilla-mobile/firefox-ios/blob/master/SyncIntegrationTests/conftest.py

Specifically in this line:
https://github.com/mozilla-mobile/firefox-ios/blob/master/SyncIntegrationTests/conftest.py#L89

Changing that line to: extensions.experiments.enabled': True made the trick. TPS is correctly loaded and works as expected now. That will be done in two PR one for Firefox-iOS and another for Fenix where the framwork lives and is updated.

Closing this issue. Thanks Mark for your help to understand what was happening.