1893223 - Some marionette manifests are not included by the master manifest

Status: RESOLVED FIXED

(Testing :: Marionette Client and Harness, task, P3)

Description

[Geoff Brown [:gbrown]]

In bug 1892038 we noticed that some marionette tests could not be run in CI because they were not included in testing/marionette/harness/marionette_harness/tests/unit-tests.toml. It is not clear to me if all marionette manifests should be included there or not, so just noting here for someone to review and consider.

https://searchfox.org/mozilla-central/search?q=&path=marionette**manifest*.toml&case=true&regexp=false

Comment 1

[Rob Wu [:robwu]]

Is this a correct assessment of the issue? It is surprising to learn that the file needs to be listed here. In other test types, listing it in the directory-specific toml file is enough for it to be picked up in CI.

I previously wondered about the test selection behavior (locally) at https://phabricator.services.mozilla.com/D196188#6507426

If it ultimately turns out that the test needs to be listed in unit-tests.toml, then that should be documents.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

As we discussed we do not have the time to change the underlying packaging behavior for Marionette tests. As such we suggest that the unit-test.toml file gets updated so that it contains all the missing references to tests that are not run yet in CI. As suggested we should also document that and maybe this could directly be done in the same file.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Bug 1893223 - [marionette] Fix master manifest for in-tree Marionette tests.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b609a929be9f
[marionette] Fix master manifest for in-tree Marionette tests. r=webdriver-reviewers,application-update-reviewers,Sasha,nalexander

Comment 5

[Iulian Moraru]

Backed out for causing marionette failures on test_exitcode.py.

• Push with failures

• Failure log

• Backout link

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

That's interesting. This test was running before as well and did not fail. My patch only changed the order between the update and the XRE tests. And I have to admit that I missed to have a final check for the try build results. There it's failing as well but only for MacOS and no other platform. I'll take a look what's wrong here. Most likely that the update tests do not correctly clean-up.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I filed bug 1898477 to cover the specific test failure that happens when the update test is run before the xre test. For now I will keep the update test to run as last test to prevent that.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a9f0952d79a4
[marionette] Fix master manifest for in-tree Marionette tests. r=webdriver-reviewers,application-update-reviewers,Sasha,nalexander

Comment 9

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/a9f0952d79a4