1093012 - TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/content-policy/test-plugins-policy.js | test-plugins-policy.js::setupModule

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[aleth [:aleth]]

also 

TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/content-tabs/test-plugin-blocked.js | test-plugin-blocked.js::test_blocklisted_plugin_notification

TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/content-tabs/test-plugin-crashing.js | test-plugin-crashing.js::setupModule

TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/content-tabs/test-plugin-outdated.js | test-plugin-outdated.js::setupModule

Comment 1

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=7859
repository: comm-central
start_time: 2015-01-19T05:01:09
who: aleth[at]instantbird[dot]org
machine: talos-mtnlion-r5-012
buildname: TB Rev5 MacOSX Mountain Lion 10.8 comm-central opt test mozmill
revision: d2889360a981

ConfigParser.NoSectionError: No section: 'App'
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/content-policy/test-plugins-policy.js | test-plugins-policy.js::setupModule
TEST-UNEXPECTED-FAIL | (runtestlist.py) | Exited with code 1 during directory run
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/content-tabs/test-plugin-blocked.js | test-plugin-blocked.js::test_blocklisted_plugin_notification
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/content-tabs/test-plugin-crashing.js | test-plugin-crashing.js::setupModule
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/content-tabs/test-plugin-outdated.js | test-plugin-outdated.js::setupModule
TEST-UNEXPECTED-FAIL | (runtestlist.py) | Exited with code 1 during directory run
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/testBasicFunctionality.js | testBasicFunctionality.js::teardownTest
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/testBasicFunctionality.js | testBasicFunctionality.js::testSmokeTest
TEST-UNEXPECTED-FAIL | (runtestlist.py) | Exited with code 1 during directory run
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/testLocalICS.js | testLocalICS.js::<TOP_LEVEL>
TEST-UNEXPECTED-FAIL | (runtestlist.py) | Exited with code 1 during directory run
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/testTodayPane.js | testTodayPane.js::<TOP_LEVEL>
TEST-UNEXPECTED-FAIL | (runtestlist.py) | Exited with code 1 during directory run

Comment 2

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=7836
repository: comm-central
start_time: 2015-01-18T13:37:12
who: aleth[at]instantbird[dot]org
machine: talos-mtnlion-r5-050
buildname: TB Rev5 MacOSX Mountain Lion 10.8 comm-central opt test mozmill
revision: d2889360a981

ConfigParser.NoSectionError: No section: 'App'
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/content-policy/test-plugins-policy.js | test-plugins-policy.js::setupModule
TEST-UNEXPECTED-FAIL | (runtestlist.py) | Exited with code 1 during directory run
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/content-tabs/test-plugin-blocked.js | test-plugin-blocked.js::test_blocklisted_plugin_notification
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/content-tabs/test-plugin-crashing.js | test-plugin-crashing.js::setupModule
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/content-tabs/test-plugin-outdated.js | test-plugin-outdated.js::setupModule
TEST-UNEXPECTED-FAIL | (runtestlist.py) | Exited with code 1 during directory run
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/testBasicFunctionality.js | testBasicFunctionality.js::teardownTest
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/testBasicFunctionality.js | testBasicFunctionality.js::testSmokeTest
TEST-UNEXPECTED-FAIL | (runtestlist.py) | Exited with code 1 during directory run
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/testLocalICS.js | testLocalICS.js::<TOP_LEVEL>
TEST-UNEXPECTED-FAIL | (runtestlist.py) | Exited with code 1 during directory run
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/testTodayPane.js | testTodayPane.js::<TOP_LEVEL>
TEST-UNEXPECTED-FAIL | (runtestlist.py) | Exited with code 1 during directory run

Comment 3

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=comm-central&job_id=7822
repository: comm-central
start_time: 2015-01-18T12:25:45
who: aleth[at]instantbird[dot]org
machine: talos-mtnlion-r5-090
buildname: TB Rev5 MacOSX Mountain Lion 10.8 comm-central debug test mozmill
revision: d2889360a981

ConfigParser.NoSectionError: No section: 'App'
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/content-policy/test-plugins-policy.js | test-plugins-policy.js::setupModule
TEST-UNEXPECTED-FAIL | (runtestlist.py) | Exited with code 1 during directory run
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/content-tabs/test-plugin-blocked.js | test-plugin-blocked.js::test_blocklisted_plugin_notification
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/content-tabs/test-plugin-crashing.js | test-plugin-crashing.js::setupModule
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/content-tabs/test-plugin-outdated.js | test-plugin-outdated.js::setupModule
TEST-UNEXPECTED-FAIL | (runtestlist.py) | Exited with code 1 during directory run
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/testBasicFunctionality.js | testBasicFunctionality.js::teardownTest
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/testBasicFunctionality.js | testBasicFunctionality.js::testSmokeTest
TEST-UNEXPECTED-FAIL | (runtestlist.py) | Exited with code 1 during directory run
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/testLocalICS.js | testLocalICS.js::<TOP_LEVEL>
TEST-UNEXPECTED-FAIL | (runtestlist.py) | Exited with code 1 during directory run
TEST-UNEXPECTED-FAIL | /builds/slave/talos-slave/test/build/mozmill/testTodayPane.js | testTodayPane.js::<TOP_LEVEL>
TEST-UNEXPECTED-FAIL | (runtestlist.py) | Exited with code 1 during directory run

Comment 4

[aleth [:aleth]]

Do you remember what fixed these failures? They are still broken on esr38.

Comment 5

[Magnus Melin [:mkmelin]]

No sorry.

Comment 6

[aleth [:aleth]]

Actually may not be fixed, still showing up in OSX debug on c-c.

Comment 7

[aleth [:aleth]]

Also fails on OSX opt, but not when run locally.

These are all
   EXCEPTION: plugin is null

which is from get_test_plugin() in test-content-tab-helpers.js

Comment 8

[aleth [:aleth]]

(In reply to aleth [:aleth] from comment #7)
> Also fails on OSX opt, but not when run locally.
> 
> These are all
>    EXCEPTION: plugin is null
> 
> which is from get_test_plugin() in test-content-tab-helpers.js

The only plugin that is found is the Default Browser Helper. Maybe we are not packaging the test plugin for some reason on OS X?

Comment 9

[aleth [:aleth]]

An OSX local build finds (on running the mozmill test) the following plugins:
* Default Browser Helper
* Java Applet Plug-in
* Shockwave Flash
* Java Test Plug-in
* Flash Test Plug-in
* Second Test Plug-in
* Test Plug-in

These are located in objdir/dist/plugins and objdir/_tests/mozmill/mozmillprofile/plugins.

Comment 10

[aleth [:aleth]]

(In reply to aleth [:aleth] from comment #9)
> objdir/_tests/mozmill/mozmillprofile/plugins.

This is generated when the mozmill test is run, the plugin dir location is passed via a flag (testsuite-targets.mk).

Comment 11

[aleth [:aleth]]

In automation, mozmill is run by runtest.py, called by runtestlist.py. Adding some logging here shows that the PLUGINS_PATH is not set, and so the plugins are not copied to the mozmillprofile, causing the failure.

Comment 12

[aleth [:aleth]]

Attachment: Set plugins-path for mozmill tests run via mozharness

This sets the plugins-path command line parameter for runtestlist.py, which should hopefully resolve the problem. (It's unfortunately not possible to test this on try-comm-central.)

Comment 13

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8719077 [details] [diff] [review]
Set plugins-path for mozmill tests run via mozharness

Review of attachment 8719077 [details] [diff] [review]:
-----------------------------------------------------------------

This looks reasonable, but I think jlund should review here.

Comment 14

[Jordan Lund (:jlund)]

Comment on attachment 8719077 [details] [diff] [review]
Set plugins-path for mozmill tests run via mozharness

Review of attachment 8719077 [details] [diff] [review]:
-----------------------------------------------------------------

also looks good to me. that's too bad this can't be tested via comm-try :(

Comment 15

[aleth [:aleth]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/fdeaea80d8a1a4d566a2f118f9214d90ccce4203
Bug 1093012 - Set plugins-path for mozmill tests run via mozharness. r=jlund

Comment 16

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/fdeaea80d8a1

Comment 17

[aleth [:aleth]]

Exception: Plugins path "/builds/slave/test/build/application/Daily.app/Contents/Resources/plugins" does not exist.
TEST-UNEXPECTED-FAIL | (runtestlist.py) | Exited with code 1 during directory run

Comment 18

[aleth [:aleth]]

On Linux, this patch works:
PLUGINS_PATH is  /builds/slave/test/build/application/thunderbird/plugins
and the plugins are indeed found there.

Not sure what goes wrong on OSX at this point (some universal build issue?), or why this failure didn't occur for Linux before the plugin path was set.

Comment 19

[aleth [:aleth]]

These plugins are packaged in common.tests.zip/bin/plugins, both for Linux and Mac. So the suspicion has to be that they aren't copied to the right place on Mac.

Comment 20

[aleth [:aleth]]

OSX mozmill

20:24:51     INFO - Running main action method: run_tests
20:24:51     INFO - mkdir: /builds/slave/test/build/application/Daily.app/Contents/MacOS/plugins
20:24:51     INFO - copying tree: /builds/slave/test/build/tests/bin/plugins to /builds/slave/test/build/application/Daily.app/Contents/MacOS/plugins

Linux mozmill

09:38:26     INFO - mkdir: /builds/slave/test/build/application/thunderbird/plugins
09:38:26     INFO - copying tree: /builds/slave/test/build/tests/bin/plugins to /builds/slave/test/build/application/thunderbird/plugins

OSX xpcshell

03:38:17     INFO - Running main action method: run_tests
03:38:17     INFO - mkdir: /builds/slave/test/build/application/Daily.app/Contents/Resources/plugins
03:38:17     INFO - copying tree: /builds/slave/test/build/tests/bin/plugins to /builds/slave/test/build/application/Daily.app/Contents/Resources/plugins

Comment 21

[aleth [:aleth]]

Attachment: Ensure plugins and extensions are placed in Contents/Resources on OSX for mozmill tests

I think this should do it. I'm not sure 'components' is actually required for mozmill, and has nothing to do with this bug, but it is present for xpcshell and so its absence for mozmill seems dubious.

Comment 24

[Jordan Lund (:jlund)]

Comment on attachment 8721712 [details] [diff] [review]
Ensure plugins and extensions are placed in Contents/Resources on OSX for mozmill tests

Review of attachment 8721712 [details] [diff] [review]:
-----------------------------------------------------------------

this logic looks very similar to preflight_xpcshell()[1]

do you think we could rip this out into some common helper method?

also, do you think we will need a mozmill equivalent bin copy that is similar to xpcshell:
    shutil.copy2(os.path.join(dirs['abs_test_bin_dir'], c['xpcshell_name']),
                     os.path.join(abs_app_dir, c['xpcshell_name']))


[1] https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/desktop_unittest.py#517

Comment 25

[aleth [:aleth]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ca87e93a883e

Comment 26

[aleth [:aleth]]

(In reply to Jordan Lund (:jlund) from comment #24)
> also, do you think we will need a mozmill equivalent bin copy that is
> similar to xpcshell:
>     shutil.copy2(os.path.join(dirs['abs_test_bin_dir'], c['xpcshell_name']),
>                      os.path.join(abs_app_dir, c['xpcshell_name']))

I don't think so - there doesn't seem to be any mozmill equivalent to the xpcshell binary in common.tests.zip/bin (and if there was much more should have been broken).

Comment 27

[aleth [:aleth]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a39048f2f294

Comment 28

[aleth [:aleth]]

Attachment: Ensure plugins and extensions are placed in Contents/Resources on OSX for mozmill tests

Comment 29

[aleth [:aleth]]

Attachment: Ensure plugins and extensions are placed in Contents/Resources on OSX for mozmill tests

Comment 30

[aleth [:aleth]]

Attachment: Ensure plugins and extensions are placed in Contents/Resources on OSX for mozmill tests

Comment 31

[Jordan Lund (:jlund)]

Comment on attachment 8722467 [details] [diff] [review]
Ensure plugins and extensions are placed in Contents/Resources on OSX for mozmill tests

Review of attachment 8722467 [details] [diff] [review]:
-----------------------------------------------------------------

ooo .. I like this much more! :)

Comment 32

[Jordan Lund (:jlund)]

Comment on attachment 8722467 [details] [diff] [review]
Ensure plugins and extensions are placed in Contents/Resources on OSX for mozmill tests

if it doesn't break xpcshell on try, ship-it!

Comment 33

[aleth [:aleth]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=879c7309e23f

Comment 34

[aleth [:aleth]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/13f2bf958d0f2cd816b24aaebf4e2237263d4638
Bug 1093012 - Ensure plugins and extensions are placed in Contents/Resources on OSX for mozmill tests. r=jlund

Comment 35

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/13f2bf958d0f

Comment 36

[aleth [:aleth]]

Mozmill is finally green. Changing the component to be able to request uplift.

Comment 37

[aleth [:aleth]]

Comment on attachment 8719077 [details] [diff] [review]
Set plugins-path for mozmill tests run via mozharness

Approval Request Comment
[Feature/regressing bug #]: Makes mozmill tests run properly on OS X
[User impact if declined]: Test-only changes
[Describe test coverage new/current, TreeHerder]: OS X mozmill now green on c-c
[Risks and why]: low (change affects only mozmill test harness)

Comment 38

[Sylvestre Ledru [:Sylvestre]]

Comment on attachment 8719077 [details] [diff] [review]
Set plugins-path for mozmill tests run via mozharness

Improve the testsuite, taking it.
Should be in 45 beta 10

Comment 39

[Carsten Book [:Tomcat]]

has problems uplifting, could you take a look ?

grafting 330231:13f2bf958d0f "Bug 1093012 - Ensure plugins and extensions are placed in Contents/Resources on OSX for mozmill tests. r=jlund"
merging testing/mozharness/scripts/desktop_unittest.py
warning: conflicts while merging testing/mozharness/scripts/desktop_unittest.py! (edit, then use 'hg resolve --mark')
abort: unresolved conflicts, can't continue
(use hg resolve and hg graft --continue)

Comment 40

[aleth [:aleth]]

Attachment: Ensure plugins and extensions are placed in Contents/Resources on OSX for mozmill tests -- mozilla-aurora version

Version for mozilla-aurora (one line of context was changed by https://hg.mozilla.org/mozilla-central/rev/4026ceb5a083).

Comment 41

[Jorg K (CEST = GMT+2)]

Aleth, why don't you just check it in? It's been reviewed and approved, so surely that covers an adaption to Aurora.

Comment 42

[Magnus Melin [:mkmelin]]

Only believe only sheriffs check in things for the m-c repos other than inbound (and the like).

Comment 43

[Carsten Book [:Tomcat]]

remote:   https://hg.mozilla.org/releases/mozilla-aurora/rev/96ebff8f55bf
remote:   https://hg.mozilla.org/releases/mozilla-aurora/rev/79d93a35ff02

remote:   https://hg.mozilla.org/releases/mozilla-beta/rev/d978b460ac48
remote:   https://hg.mozilla.org/releases/mozilla-beta/rev/70d9804d8757