523402 - when doing a make package-tests, it doesn't include the mobile specific browser chrome tests

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

When trying to run browser-chrome mochitests written for fennec, we don't find them in the package-tests file.  This looks to be the case because all of the unittests live in the xulrunner objdir, and our mobile specific tests live int he mobile objdir.  

I tried a make package-tests in the mobile objdir and received errors in the script while it was trying to cd ../../dist/bin/plugins.

My recommendation is to create a cleaner make package-tests that can be run from the mobile objdir.  If we go this route, we will need releng to create another tests binary for us for each build in addition to the normal tests binary.

Ted, you might have ideas here.

Comment 1

[(not currently active) Ted Mielczarek]

I'm surprised that we didn't already have a bug on this, but yeah, as currently implemented "package-tests" can't know about the Fennec objdir, because it's completely separate from the xulrunner objdir. It's the same situation as "make package", really, where the mobile tree jumps through some hoops to package the xulrunner bits.

(Also, I think we should just change Fennec builds to not be a two-pass build, but that belongs in a separate bug.)

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: package mobile tests

patch to package tests for mobile similar to 'make package-tests' but in the mobile tree.

Run it by doing:
cd $(objdir)/mobile/mobile/installer
make package-mobile-tests
cd $(objdir)/mobile/dist
ls *tests*.bz2

One caveat: if you run this first, then $(objdir)/mobile/make package, make package will delete the $(objdir)/mobile/dist folder and your bz2 file will disappear.

Another thing to note is this will turn the tests into the same testing structure as the xulrunner (firefox) *tests.tar.bz2 file.  So if you untar this in the same directory, it will add the additional mobile browser-chrome tests to mochitest/browser/mobile/*.

I testing this with a fresh clobber build on mozilla-central and 1.9.2 for desktop linux Fennec.  I ran the tests in a virgin directory (make package + make package-mobile-tests only- untar'd to a fresh dir) like so:
cd mochitest
python runtests.py --appname=../fennec/fennec --xre-path=../fennec/xulrunner --browser-chrome --autorun

Comment 3

[Mark Finkle (:mfinkle) (use needinfo?)]

Comment on attachment 407663 [details] [diff] [review]
package mobile tests 

Seems ok to me, but Ted should really have a look

Comment 4

[Mark Finkle (:mfinkle) (use needinfo?)]

It would be nice if we could run the make package-mobile-tests in the objdir too. This certainly gets us moving in the right direction: running browser-chrome tests!

Comment 5

[(not currently active) Ted Mielczarek]

Comment on attachment 407663 [details] [diff] [review]
package mobile tests 

Bleh. Just include $(topsrcdir)/testing/testsuite-targets.mk, and do

package-mobile-tests: stage-mochitest

Then tar up the result? Ideally you could just reuse the package-tests target, but we'd have to do some trickery in there to not try to package all the reftest stuff, etc.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

I would like to have it reuse the same logic and code if possible.  I found that what we do for mochitest stage-package (http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/Makefile.in#127), has issues with the /bin and /certs stuff since dist is $(objdir)/mobile vs $(objdir)/xulrunner.

Comment 7

[(not currently active) Ted Mielczarek]

Yeah. I'd take a patch that adds ifdefs or handles failures there silently to make it work.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: package-tests for the mobile-browser branch

patch to allow $(objdir)/mobile/mobile: make package-mobile-tests reuse the same logic for package-tests as xulrunner.

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: package-tests for mozilla-central/1.9.2

this is the other part of the patch that uses #ifdef's to fix the 'make package-mobile-tests' from the mobile objdir.

Comment 10

[(not currently active) Ted Mielczarek]

Comment on attachment 408537 [details] [diff] [review]
package-tests for mozilla-central/1.9.2

+#ifdef WINCE
+DIST_BIN = $(DIST)/bin/xulrunner
+#endif
+
+#ifdef MOZ_PLATFORM_HILDON
+DIST_BIN = $(DIST)/bin/xulrunner
+#endif

This is no good. For one, we already have Firefox WinCE builds, and we'll want to run tests on those. Also, when do the xulrunner bits get stuck in the Fennec dist/bin? You'll want to make sure there isn't an ordering problem between running "make package" and "make package-tests".

I think if you just did
DIST_BIN = $(DIST)/bin, then you could just have the mobile Makefile call stage-mochitest directly, like
package-mobile-tests:
    $(MAKE) stage-mochitest DIST_BIN=$(DIST)/bin/xulrunner
and that would be ok.

Comment 11

[(not currently active) Ted Mielczarek]

Comment on attachment 408536 [details] [diff] [review]
package-tests for the mobile-browser branch

This looks ok, although per my other review comment you might want to change the way this calls stage-mochitest.

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: package-mobile-tests (mobile-browser) patch

update for package-mobile-tests for the mobile-browser branch from last review comments.

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: package-mobile-tests for mozilla-central/1.9.2

updated patch for package-mobile-tests on mozilla-central and 1.9.2.  Much simpler and tested from fresh clobber builds on both branches.

Comment 14

[Joel Maher ( :jmaher ) (UTC -8)]

just verified with a fresh m-c/m-b sync up and clean clobber build that this has no bitrot for patches or functionality.  And the mobile browser-chrome tests still pass!

mfinkle, please check this in when you get a chance.

Comment 15

[Mark Finkle (:mfinkle) (use needinfo?)]

pushed m-b:
https://hg.mozilla.org/mobile-browser/rev/0a42c675f8a7

pushed m-c:
http://hg.mozilla.org/mozilla-central/rev/15726bd6943c

pushed m-192:
http://hg.mozilla.org/releases/mozilla-1.9.2/rev/2f5c1da8fe1c