1362711 - Intermittent AssertionError: BaseMarionetteTestRunner configuration (self.e10s) does not match browser appinfo

Status: RESOLVED FIXED

(Testing :: Marionette Client and Harness, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://treeherder.mozilla.org/logviewer.html#?job_id=97072021&repo=mozilla-inbound

03:12:02     INFO - mozversion platform_version: 55.0a1
03:12:02     INFO - Application command: Z:\task_1494037350\build\application\firefox\firefox.exe -no-remote -marionette -profile c:\users\genericworker\appdata\local\temp\tmpgpwzci
03:12:02     INFO - Profile path is c:\users\genericworker\appdata\local\temp\tmpgpwzci
03:12:02     INFO - Starting fixture servers
03:12:03     INFO - Fixture server listening on http://127.0.0.1:57220/
03:12:03     INFO - Fixture server listening on https://127.0.0.1:57221/
03:12:03     INFO - e10s is disabled
03:12:04    ERROR - Failure during harness execution
03:12:04    ERROR - Traceback (most recent call last):
03:12:04    ERROR - 
03:12:04    ERROR -   File "z:\task_1494037350\build\venv\lib\site-packages\marionette_harness\runtests.py", line 92, in cli
03:12:04    ERROR -     failed = harness_instance.run()
03:12:04    ERROR - 
03:12:04    ERROR -   File "z:\task_1494037350\build\venv\lib\site-packages\marionette_harness\runtests.py", line 72, in run
03:12:04    ERROR -     runner.run_tests(tests)
03:12:04    ERROR - 
03:12:04    ERROR -   File "z:\task_1494037350\build\venv\lib\site-packages\marionette_harness\runner\base.py", line 846, in run_tests
03:12:04    ERROR -     raise AssertionError(message_e10s)
03:12:04    ERROR - 
03:12:04    ERROR - AssertionError: BaseMarionetteTestRunner configuration (self.e10s) does not match browser appinfo
03:12:04    ERROR - 
03:12:04    ERROR - Return code: 1

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)]

For the tests e10s gets disabled via the command line flag of Marionette, but the browser told us otherwise. 

This all happened on Windows 10 nodes and TC jobs. There might be still some issues with the new machines? CC'ing Rob for information.

Comment 2

[Rob Thijssen [:grenade (EET/UTC+0300)]]

apologies, i don't understand the question. i don't think the instance configuration tries to change e10s enable/disable. is there some setting i should check?

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2] (away 07/21 - 08/08)]

I checked again this failure, and it's actually the other way around. The job wants Firefox to be started in e10s mode, but it's disabled. The logger entry is here:

https://dxr.mozilla.org/mozilla-central/rev/17d8a1e278a9c54a6fdda9d390abce4077e55b20/testing/marionette/harness/marionette_harness/runner/base.py#840

It means appinfo's property "browserTabsRemoteAutostart" is false instead of true.

Here are the pref which we are setting:

https://dxr.mozilla.org/mozilla-central/rev/17d8a1e278a9c54a6fdda9d390abce4077e55b20/testing/marionette/harness/marionette_harness/runner/base.py#553-556

So maybe something prevents Firefox from turning e10s mode on, which might be dependent on a system setting or whatever for those new Windows 10 machines. We have never seen this problem somewhere else so far.

Comment 4

[Pete Moore [:pmoore][:pete]]

Would this be a good test for me to check this theory?

1) I grab a win10 worker from the pool
2) Start up firefox
3) Go to about:config and set browser.tabs.remote.autostart to true
4) Restart the browser
5) Check the value in about:config of browser.tabs.remote.autostart, to see if it is still "true" or not

Is there a better way of validating that the browser can run in e10s mode?

Comment 5

[Pete Moore [:pmoore][:pete]]

Actually, I suspect this is a genuine issue, rather than a test failure. Just discovered bug 1263278 ....

Comment 6

[Pete Moore [:pmoore][:pete]]

We should confirm that this is indeed the same problem by seeing if the option is greyed out or not (and if it isn't, we can remove the bug dependency).

Comment 7

[Pete Moore [:pmoore][:pete]]

I notice the following line in the log:

Application command: Z:\task_1494037350\build\application\firefox\firefox.exe -no-remote -marionette -profile c:\users\genericworker\appdata\local\temp\tmpgpwzci

I'm surprised/concerned to see that the appdata folder is on the C: drive, since that would persist between tasks (we format the Z: drive between task runs).

Maybe we are not setting the APPDATA folder location for the GenericWorker location to the Z: drive during machine setup? Or maybe something in tree is building the path incorrectly.

Comment 8

[Pete Moore [:pmoore][:pete]]

Attachment: Github Pull Request for OpenCloudConfig

This patch is just for *gecko-t-win10-64-beta* worker type only, and sets 3 known folder locations - LocalAppData, RoamingAppData, LocalAppDataLow - for the GenericWorker user.

If this patch helps things, we'll need to decide on a better way to integrate it, since at the moment it requires hardcoding the GenericWorker password - hence why I've only applied it to a beta worker type. It probably isn't too complicated to integrate it some other way, but I didn't want to invest heavily on implementation until we're sure it helps (since running as current user is hopefully only a temporary workaround anyway, and this change isn't needed if we run as a task user, since g-w already sets known foler locations for task users).

Comment 9

[Rob Thijssen [:grenade (EET/UTC+0300)]]

Comment on attachment 8866302 [details] [review]
Github Pull Request for OpenCloudConfig

merged: https://github.com/mozilla-releng/OpenCloudConfig/commit/e3b965e315645db37ce2aec5dbc3770af574050f

Comment 10

[Pete Moore [:pmoore][:pete]]

Rerunning original task against this new worker type:

https://tools.taskcluster.net/task-inspector/#PY4_-t4XRdW81IHqk4m4jA/

Comment 11

[Pete Moore [:pmoore][:pete]]

So one strange thing I noticed, might be totally unrelated.

I installed Firefox 53.0.2 (64-bit) on Windows 10, and then went to about:support and saw that e10s was already enabled:

    Multiprocess Windows: 1/1

I then went to about:config and saw that browser.tabs.remote.autostart was set to false (default) and that there was a setting browser.tabs.remote.autostart.1 set to true.

So it looks like browser.tabs.remote.autostart isn't definitive as to whether e10s is enabled.

I then set browser.tabs.remote.autostart to true and restarted, and it remained set to true (browser.tabs.remote.autostart.1 also set to true).

Not really sure if this is related to the issue, but it does seem weird.

This was on a vanilla Windows10 VM (not a worker) because I wanted to see if this was a problem in general on Win10 64 bit or not.

Comment 12

[Pete Moore [:pmoore][:pete]]

(In reply to Pete Moore [:pmoore][:pete] from comment #10)
> Rerunning original task against this new worker type:
> 
> https://tools.taskcluster.net/task-inspector/#PY4_-t4XRdW81IHqk4m4jA/

gecko-t-win10-64-beta was *not* running as current user, so updated worker type config, killed old instance, and have retriggered task:

https://tools.taskcluster.net/task-inspector/#KN1ND_dkQMi8zwMLjigltw/

Comment 13

[Pete Moore [:pmoore][:pete]]

This correctly set LOCALAPPDATA but not APPDATA due to this mistake! D'oh!!!

https://github.com/mozilla-releng/OpenCloudConfig/pull/69

Will retrigger again when the new worker type is running in production.

Comment 14

[Intermittent Failures Robot]

8 failures in 879 pushes (0.009 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-inbound: 6
* mozilla-central: 1
* autoland: 1

Platform breakdown:
* linux64: 4
* windows10-64-vm: 3
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1362711&startday=2017-05-08&endday=2017-05-14&tree=all

Comment 15

[Joel Maher ( :jmaher ) (UTC -8)]

I believe this is resolved?

Comment 16

[BMO Automation]

Moving bug to Testing::Marionette Client and Harness component per bug 1815831.