1224746 - mozmill/resources/jsbridge install failure on mozmill OSX opt

Status: RESOLVED FIXED

(Thunderbird :: Testing Infrastructure, defect)

Description

[aleth [:aleth]]

Happens in the run-tests step. Doesn't happen for debug builds.

INFO -  Exception:
RROR -  Traceback (most recent call last):
INFO -    File "/builds/slave/test/build/venv/lib/python2.7/site-packages/pip-1.5.5-py2.7.egg/pip/basecommand.py", line 122, in main
INFO -      status = self.run(options, args)
INFO -    File "/builds/slave/test/build/venv/lib/python2.7/site-packages/pip-1.5.5-py2.7.egg/pip/commands/install.py", line 257, in run
INFO -      InstallRequirement.from_line(name, None))
INFO -    File "/builds/slave/test/build/venv/lib/python2.7/site-packages/pip-1.5.5-py2.7.egg/pip/req.py", line 172, in from_line
INFO -      return cls(req, comes_from, url=url, prereleases=prereleases)
INFO -    File "/builds/slave/test/build/venv/lib/python2.7/site-packages/pip-1.5.5-py2.7.egg/pip/req.py", line 70, in __init__
INFO -      req = pkg_resources.Requirement.parse(req)
INFO -    File "/builds/slave/test/build/venv/lib/python2.7/site-packages/pip-1.5.5-py2.7.egg/pip/_vendor/pkg_resources.py", line 2667, in parse
INFO -      reqs = list(parse_requirements(s))
INFO -    File "/builds/slave/test/build/venv/lib/python2.7/site-packages/pip-1.5.5-py2.7.egg/pip/_vendor/pkg_resources.py", line 2593, in parse_requirements
INFO -      raise ValueError("Missing distribution spec", line)
INFO -  ValueError: ('Missing distribution spec', '/builds/slave/test/build/tests/mozmill/resources/jsbridge')

"Missing distribution spec" usually means "file not found or path incorrect", however the same pip install command runs fine on debug builds.

Comment 1

[aleth [:aleth]]

The problem could be that (just a little earlier in the log)

INFO - copying tree: /builds/slave/test/build/tests/extensions to /builds/slave/test/build/application/DailyDebug.app/Contents/MacOS/extensions
INFO - rmtree: /builds/slave/test/build/application/DailyDebug.app/Contents/MacOS/extensions
 
is missing for opt builds, if that's where the jsbridge extension is supposed to live?

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

I don't know anything about mozmill, but I know a guy who does ....

aleth, could you please provide a link to a log for full context.

Comment 3

[aleth [:aleth]]

(In reply to Nick Thomas [:nthomas] from comment #2)
> aleth, could you please provide a link to a log for full context.

e.g. https://treeherder.mozilla.org/logviewer.html#?job_id=26265&repo=comm-central

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I have no idea about the testing infrastructure for Thunderbird and what has changed to cause this bustage. And I'm fairly sure this problem is part of it and has nothing to do with the framework itself. So you might want to find someone who knows all about it. Maybe Joshua can help here.

Comment 5

[aleth [:aleth]]

(In reply to aleth [:aleth] from comment #1)
> The problem could be that (just a little earlier in the log)
> 
> INFO - copying tree: /builds/slave/test/build/tests/extensions to
> /builds/slave/test/build/application/DailyDebug.app/Contents/MacOS/extensions
> INFO - rmtree:
> /builds/slave/test/build/application/DailyDebug.app/Contents/MacOS/extensions
>  
> is missing for opt builds, if that's where the jsbridge extension is
> supposed to live?

Looks to me like this check
https://dxr.mozilla.org/comm-central/source/mozilla/testing/mozharness/scripts/desktop_unittest.py#576
must be failing for opt builds, as otherwise we'd see a line
https://dxr.mozilla.org/comm-central/source/mozilla/testing/mozharness/mozharness/base/script.py#613
in the logs.

However, it doesn't seem like much can go wrong where this dir is set?
https://dxr.mozilla.org/comm-central/source/mozilla/testing/mozharness/scripts/desktop_unittest.py#205

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

These are probably the important lines in the opt log which are missing in the debug log:

14:59:00     INFO - Copy/paste: unzip -q -o /builds/slave/test/build/thunderbird-45.0a1.en-US.mac.common.tests.zip bin/* certs/* extensions/* modules/* mozbase/* config/* mozmill/*
14:59:00     INFO - Calling ['unzip', '-q', '-o', u'/builds/slave/test/build/thunderbird-45.0a1.en-US.mac.common.tests.zip', 'bin/*', 'certs/*', 'extensions/*', 'modules/*', 'mozbase/*', 'config/*', 'mozmill/*'] with output_timeout 1760
14:59:01     INFO -  caution: filename not matched:  extensions/*
14:59:01     INFO -  caution: filename not matched:  mozmill/*
14:59:01     INFO - Return code: 11

So mozmill isn't getting packaged. That's about where my expertise ends.

Comment 7

[aleth [:aleth]]

(In reply to Nick Thomas [:nthomas] from comment #6)
> So mozmill isn't getting packaged. That's about where my expertise ends.

Aha! Well spotted, thanks. Let's see if the buildbotcustom fix that just landed for the build orange helps, it's possible we're simply not reaching that packaging step.

Comment 8

[aleth [:aleth]]

(In reply to aleth [:aleth] from comment #7)
> Aha! Well spotted, thanks. Let's see if the buildbotcustom fix that just
> landed for the build orange helps, it's possible we're simply not reaching
> that packaging step.

For future reference: that wasn't it.

Comment 9

[Stefan Sitter [:ssitter]]

See Bug 1224844 for a similar test failure that happens only in 'OS X opt' builds. The cause for failure might be that files for the Provider for Google Calendar extension cannot be found, indicating it might be same root cause as in this bug.

Comment 10

[aleth [:aleth]]

Missing in the opt make-package step:

make -C ./mail/test/mozmill stage-package
../../../config/nsinstall -D ../../../dist/test-stage/mozmill/
make -C ./calendar/lightning stage-package
../../config/nsinstall -D ../../dist/test-stage/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103}
(cd ../../dist/xpi-stage/lightning && tar -chf - *) | (cd ../../dist/test-stage/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103} && tar -xf -)
grep -v em:targetPlatform ../../dist/xpi-stage/lightning/install.rdf > ../../dist/test-stage/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103}/install.rdf
../../config/nsinstall -D ../../dist/test-stage/mozmill/shared-modules
(cd /builds/slave/tb-c-cen-m64-d-000000000000000/build/calendar/test/mozmill && tar -chf - `cat /builds/slave/tb-c-cen-m64-d-000000000000000/build/calendar/test/mozmill/mozmilltests.list`) | (cd ../../dist/test-stage/mozmill && tar -xf -)
(cd /builds/slave/tb-c-cen-m64-d-000000000000000/build/calendar/test/mozmill/shared-modules && tar -chf - *) | (cd ../../dist/test-stage/mozmill/shared-modules && tar -xf -)
/builds/slave/tb-c-cen-m64-d-000000000000000/build/objdir-tb/_virtualenv/bin/python -m mozbuild.action.buildlist ../../dist/test-stage/mozmill/mozmilltests.list testBasicFunctionality.js testTodayPane.js testLocalICS.js
make -C ./calendar/providers/gdata stage-package
../../../config/nsinstall -D ../../../dist/test-stage/extensions/{a62ef8ec-5fdc-40c2-873c-223b8a6925cc}
(cd ../../../dist/xpi-stage/gdata-provider && tar -chf - *) | (cd ../../../dist/test-stage/extensions/{a62ef8ec-5fdc-40c2-873c-223b8a6925cc} && tar -xf -)

Comment 11

[aleth [:aleth]]

Attachment: Package mozmill and calendar tests for OS X universal builds too

Comment 12

[aleth [:aleth]]

(In reply to aleth [:aleth] from comment #11)
> Created attachment 8696132 [details] [diff] [review]
> Package mozmill and calendar tests for OS X universal builds too

This seems an obvious guess for how to fix the problem.

The question is why the ifdef was there in the first place, though.

Comment 13

[aleth [:aleth]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a89b89f1d925

Comment 14

[Joshua Cranmer [:jcranmer]]

(In reply to aleth [:aleth] from comment #12)
> (In reply to aleth [:aleth] from comment #11)
> > Created attachment 8696132 [details] [diff] [review]
> > Package mozmill and calendar tests for OS X universal builds too
> 
> This seems an obvious guess for how to fix the problem.
> 
> The question is why the ifdef was there in the first place, though.

Going through hg blame is a good way to uncover history. The short of the history:

Originally, the code was
    1.41 +ifndef UNIVERSAL_BINARY
    1.42 +PKG_STAGE = $(DIST)/test-package-stage
    1.43 +package-tests:: stage-mozmill
    1.44 +else
    1.45 +# This staging area has been built for us by universal/flight.mk
    1.46 +PKG_STAGE = $(DIST)/universal/test-package-stage
    1.47 +endif

This morphed into the present form when I did the c-c pseudo-merge, based on getting it to work with try builds. At the time, it aligned with mozilla-central, but Ted appears to have changed how package generation worked in bug 1201224.

Comment 15

[aleth [:aleth]]

(In reply to Joshua Cranmer [:jcranmer] from comment #14)
> mozilla-central, but Ted appears to have changed how package generation
> worked in bug 1201224.

Ah, that's the bit I didn't find, that wasn't apparent from hg blame. Thanks.

Anyway, from the try push it looks like this works!

Comment 16

[aleth [:aleth]]

(In reply to aleth [:aleth] from comment #15)
> Anyway, from the try push it looks like this works!

The relevant try push was https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=8d6f5bed4a15

Comment 17

[aleth [:aleth]]

https://hg.mozilla.org/comm-central/rev/8d1f4274456945fb10dc79bd719c89ad6351ae3a
Bug 1224746 - Package mozmill and calendar tests for OS X universal builds too. r=jcranmer

Comment 18

[aleth [:aleth]]

Comment on attachment 8696132 [details] [diff] [review]
Package mozmill and calendar tests for OS X universal builds too

[Approval Request Comment]
Regression caused by (bug #): see blocking bug from mozilla 43

Comment 19

[Kent James (:rkent)]

Comment on attachment 8696132 [details] [diff] [review]
Package mozmill and calendar tests for OS X universal builds too

https://hg.mozilla.org/releases/comm-beta/rev/55507ff7483e
https://hg.mozilla.org/releases/comm-aurora/rev/f52db1e76dea