1488899 - Integration Tests: Tests are failing in tps step when trying to use the profile

Status: RESOLVED FIXED

(Firefox :: Sync, defect, P1)

Description

[Isabel Rios[:isabel_rios]]

Attachment: log(1).txt

All integration tests are failing due to this error:

```tps.py  33 INFO     Using profile at: /var/folder/md/29mfzzms0wn_1ljqddc61y840000gn/T/tmpAtuNJX.mozrunner
FAILED```

Complete log from Jenkins attached.

It is reproducible locally.

Comment 1

[Isabel Rios[:isabel_rios]]

This started to happen yesterday afternoon. Dave, any idea about changes that could have affected the tests?
Thanks!

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

The Firefox log contains:
JavaScript error: chrome://browser/content/browser-tabsintitlebar.js, line 302: TypeError: this._menuObserver is undefined, can't access property "disconnect" of it
JavaScript error: resource://gre/modules/ExtensionParent.jsm, line 1100: TypeError: this.chromeShell is null, can't access property "document" of it
JavaScript error: resource:///modules/aboutpages/AboutPrivateBrowsingHandler.jsm, line 26: TypeError: this.pageListener is undefined, can't access property "removeMessageListener" of it

At first it appears this is not related to a change in Firefox as both the last good and first bad builds are from revision bade35bc346f3bc1f5e8b59d3d49c7a1ba28166c.

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

The first failing build downloaded a newer version of tps.xpi:

Downloading from: https://queue.taskcluster.net/v1/task/J9stvaH-QbWU1tK0jL2sOQ/artifacts/public/tps.xpi

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

It could be because we're downloading a TPS extension based off mozilla-central, but using nightly builds. This means there could be some time between a change to the extension and the next nightly build.

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

The last good TPS extension was built from revision cd72be2928ea, and the first bad was built from c3734e0f634e. I worked this out from the task ids in the console logs for the TPS, and matched these with the recent builds of the extension in Treeherder. This makes the regression range: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=cd72be2928ea&tochange=c3734e0f634e

From this, it looks like TPS is now a web extension (bug 1451527). I suspect we need to update the sync integration test harness to match the changes to the TPS test harness. Perhaps :tcsc can advise?

Comment 6

[Mr. Jacob James Davila]

Thanks to all, note I am using a Windows RM-1073 stuck in stating running Windows 8.1, yet when reviewing logs states Explorer 11, along with this syncing to an older LapTop using Windows 8.1 and connects to Infrastructure, and Debugging Explorer, Firefox running in background. Chrome,AOL and Yahoo are disabled and Microsoft 2003 and 2007 versions are loaded, Office 2010 I have yet to run my disc yet the areas City Facility's have used Private Key and upgraded. IE:Texas Work Force has Microsoft Office 2017 for training, as it is an siphoning kiosk when asked to enter an Email address. Now I personally updated the City of San Antonio Housing Authorities Microsoft,Adobe Flash Players and Chrome Browsers as well as several City of San Antonio Public Library's which run different combinations of equipment from all Apple Desktops, Dell only Desktop and Laptops, and Microsoft and Apple Desktops back to back depending on what side of town you visit.  With Samsung and LG device's there is no issue of interpolation between my accounts. Devices plugged and played as well as connected to Network Wifi with no issue, as well as Encrypted BlueTooth Only for transfer of information. The Android devices I used run Developer Mode with encryption on device and SD card. This device is not being allowed to download Developer Mode, Chrome, Play store Apps or Firefox.

Comment 7

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I've tried changing the harness to use preferences instead of command line arguments, but Firefox is still failing to launch when the extension is present in the profile. See comment 2 for the logs from Firefox.

Comment 8

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Mr. Jacob James Davila from comment #6)
> Thanks to all, note I am using a Windows RM-1073 stuck in stating running
> Windows 8.1, yet when reviewing logs states Explorer 11, along with this
> syncing to an older LapTop using Windows 8.1 and connects to Infrastructure,
> and Debugging Explorer, Firefox running in background. Chrome,AOL and Yahoo
> are disabled and Microsoft 2003 and 2007 versions are loaded, Office 2010 I
> have yet to run my disc yet the areas City Facility's have used Private Key
> and upgraded. IE:Texas Work Force has Microsoft Office 2017 for training, as
> it is an siphoning kiosk when asked to enter an Email address. Now I
> personally updated the City of San Antonio Housing Authorities
> Microsoft,Adobe Flash Players and Chrome Browsers as well as several City of
> San Antonio Public Library's which run different combinations of equipment
> from all Apple Desktops, Dell only Desktop and Laptops, and Microsoft and
> Apple Desktops back to back depending on what side of town you visit.  With
> Samsung and LG device's there is no issue of interpolation between my
> accounts. Devices plugged and played as well as connected to Network Wifi
> with no issue, as well as Encrypted BlueTooth Only for transfer of
> information. The Android devices I used run Developer Mode with encryption
> on device and SD card. This device is not being allowed to download
> Developer Mode, Chrome, Play store Apps or Firefox.

You comment appears unrelated to this bug. Did you mean to comment on another issue?

Comment 9

[Isabel Rios[:isabel_rios]]

Thanks Dave for your help debugging this!

Comment 10

[Thom Chiovoloni [:tcsc] (ex-moco)]

Hrm, sorry about this, we changed TPS to be a webextension, and I meant to CC davehunt on it but guess I forgot (very sorry).

That said, if you changed the arguments to be passed as prefs, that should be all you need to do.

Leaving the needinfo so I know to dig into this.

Comment 11

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Here's a summary of my findings so far based on the latest Nightly and tps.xpi builds (after local modifications to use TPS preferences instead of command line options):

When I run the sync integration tests (without cache), https://archive.mozilla.org/pub/firefox/nightly/2018/09/2018-09-10-22-01-42-mozilla-central/firefox-64.0a1.en-US.mac.dmg is downloaded, along with https://queue.taskcluster.net/v1/task/c0doUuM5Seawyjk99a9KcA/artifacts/public/tps.xpi.

The harness creates a profile with the tps.xpi and attempts to launch Firefox using the profile. I see no Firefox launch, and the test fails because the tps log file is not present. The Firefox log file is empty.

Attempting to run Firefox from the command line with the same profile, I see Firefox trying to launch and then closes. The output in the console is:

> JavaScript error: jar:file:///private/var/folders/wz/3v_j7g2n2zx_q6qs8g7vmyg00000gn/T/pytest-of-dhunt/pytest-67/firefox0/Firefox%20Nightly.app/Contents/Resources/browser/omni.ja!/components/nsBrowserGlue.js, line 974: NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIIdleService.removeIdleObserver]

Further attempts fail in the same way, but with no console output.

Opening the profile in finder I see [e-mail] in the extensions directory. After removing it from this directory and repeating the attempt to launch Firefox from the console, Firefox successfully launches. The console output states that it's unable to load script: jar:file:///var/folders/wz/3v_j7g2n2zx_q6qs8g7vmyg00000gn/T/tmpid9FiF.mozrunner/extensions/[e-mail]!/api.js (understandable, as I have removed it).

After closing Firefox, restoring [e-mail] to the extensions directory, and repeating the attempt to launch Firefox results in a successful launch of Firefox with the TPS extension listed in about:addons.

I wonder if the addon is somehow conflicting with some first-launch logic in Firefox?

Comment 12

[Isabel Rios[:isabel_rios]]

Hi, Dave, Tom, any news about this? Any workaround for the SyncIntegration tests? they are still disabled...
Thanks!

Comment 13

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Thom and I spoke about this last week. I believe he's going to raise this to see if we can free up some time to work on it.

Comment 14

[Mark Hammond [:markh] [:mhammond]]

(In reply to Dave Hunt [:davehunt] ⌚️UTC+1 from comment #7)
> I've tried changing the harness to use preferences instead of command line
> arguments, but Firefox is still failing to launch when the extension is
> present in the profile. See comment 2 for the logs from Firefox.

This was tricky to track down as it was somewhat intermittent. Firefox is silently being killed immediately after launch by https://searchfox.org/mozilla-central/rev/3a54520d8d2319a4116866371ed3d9ed2ec0cc2b/services/sync/tps/extensions/tps/api.js#47. For some reason I don't understand, the import of FormAutofillSync.jsm at the top of tps.jsm was failing with some obscure error which was probably timing related. tps.jsm doesn't even use that, so :shrug.

I also discovered many other issues with tps being flakey and I think I've fixed most of them - however, there were some addon failures which look like are demonstrating a real bug - so I disabled these tests and opened bug 1498974.

Comment 15

[Mark Hammond [:markh] [:mhammond]]

Attachment: Bug 1488899 - Get TPS working. r?tcsc

Bug 1498974 - Part 2: Remove old async support, more effort to prevent Syncs during TPS and fix imports. r?tcsc

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5622861eb245
Get TPS working. r=tcsc

Comment 17

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/5622861eb245