695351 - update buildbot configs for android mochitests to use an in-tree manifest

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

now that bug 694741 is landed, we don't need to hardcode our test paths in android.  This is only for mozilla-central/inbound right now, not for aurora or beta.

Here is an example of a command line that we are running:
python mochitest/runtestsremote.py --deviceIP 10.250.49.85 --xre-path ../hostutils/xre --utility-path ../hostutils/bin --certificate-path certs --app org.mozilla.fennec --console-level INFO --http-port 30097 --ssl-port 31097 --pidfile /builds/tegra-097/test/../runtestsremote.pid --test-path dom/tests/mochitest/ajax/mochikit


We now need to run something similar to this:
python mochitest/runtestsremote.py --deviceIP 10.250.49.85 --xre-path ../hostutils/xre --utility-path ../hostutils/bin --certificate-path certs --app org.mozilla.fennec --console-level INFO --http-port 30097 --ssl-port 31097 --pidfile /builds/tegra-097/test/../runtestsremote.pid --run-only-tests=android.json --total-chunks=8 --this-chunk=1 --chunk-by-dir

note the --this-chunk value will change for all of the current 8 chunks.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

this is 2.5 months old, can we get this in?  Why was this prioritized lower?
If we do this for native android tests only vs xul it would solve a lot of problems.  This would allow people to test stuff on try and to turn on tests which fail due to e10s issues on xul only.

Comment 2

[Mike Taylor [:bear]]

(In reply to Joel Maher (:jmaher) from comment #1)
> this is 2.5 months old, can we get this in?  Why was this prioritized lower?
> If we do this for native android tests only vs xul it would solve a lot of
> problems.  This would allow people to test stuff on try and to turn on tests
> which fail due to e10s issues on xul only.

It wasn't "prioritized lower" it was just missed in all the other things that "had to be done *NOW* for android over the last couple of months.  

And yes, I will work on this now.

Comment 3

[Mike Taylor [:bear]]

Attachment: buildbotcustom change

Comment 4

[Mike Taylor [:bear]]

Attachment: buildbot-configs change

Comment 5

[Mike Taylor [:bear]]

Attachment: buildbot-configs change

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

as a non expert on this stuff, it looks like it is using this for android tests which is good.  Does it do the '--total-chunks=8 --this-chunk=x' ?  similar to the 5 chunks in desktop mochitests.

Comment 7

[Mike Taylor [:bear]]

Attachment: buildbotcustom change with chunking

Comment 8

[Mike Taylor [:bear]]

Attachment: buildbot-configs change with chunking

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 586460 [details] [diff] [review]
buildbot-configs change with chunking

Review of attachment 586460 [details] [diff] [review]:
-----------------------------------------------------------------

hey, this is looking great from my point of view.

Comment 10

[Mike Taylor [:bear]]

I have put this on the tegra staging server to run today

Comment 11

[Aki Sasaki (not active)]

Comment on attachment 586459 [details] [diff] [review]
buildbotcustom change with chunking

This looks good, assuming the tests run well.
Could you add a

    if totalChunks:
        assert 1 <= thisChunk <= totalChunks

like line 796 in steps/unittest.py ?
Also, do we want to make testPath and testManifest mutually exclusive?  The code as written doesn't do that.

Comment 12

[Aki Sasaki (not active)]

Comment on attachment 586460 [details] [diff] [review]
buildbot-configs change with chunking

Thought about how we could specify (the overlapping) non-mochitests in one list, and then have the android-xul and android dicts specify the mochitest portions and extend the non-mochitest list.

Not sure how much we gain there, especially since we want android-xul to go away after a few releases. r=me.

Comment 13

[Aki Sasaki (not active)]

Comment on attachment 586459 [details] [diff] [review]
buildbotcustom change with chunking

r- for not specifying thisChunk / totalChunks for mochitests.

Comment 14

[Mike Taylor [:bear]]

Attachment: buildbotcustom change with chunking

this time I remember to *pass* the chunking parameters for mochitest - d'oh!

Comment 15

[Aki Sasaki (not active)]

Comment on attachment 587129 [details] [diff] [review]
buildbotcustom change with chunking

r=me with the assert and passing tests with correct command line options (comment 11)

Comment 16

[Mike Taylor [:bear]]

(In reply to Aki Sasaki [:aki] from comment #15)
> Comment on attachment 587129 [details] [diff] [review]
> buildbotcustom change with chunking
> 
> r=me with the assert and passing tests with correct command line options
> (comment 11)

added assert and will update test master to include new version of code

Comment 17

[Mike Taylor [:bear]]

this is being tested on the tegra staging foopy last night and tonight and is currently in a cage match with hwine's patch to add robocop.apk

will check on it tonight

Comment 18

[Mike Taylor [:bear]]

joel,

this what i'm seeing in staging now:

python mochitest/runtestsremote.py --deviceIP 10.250.50.94 --xre-path ../hostutils/xre --utility-path ../hostutils/bin --certificate-path certs --app org.mozilla.fennec --console-level INFO --http-port 30184 --ssl-port 31184 --pidfile /builds/tegra-184/test/../runtestsremote.pid --run-only-tests android.json --symbols-path=../symbols --total-chunks 8 --this-chunk 4

looks ok to me - can you sanity check the above?

Comment 19

[Joel Maher ( :jmaher ) (UTC -8)]

bear, that looks correct!  Thanks for double checking.

Comment 20

[Mike Taylor [:bear]]

(In reply to Joel Maher (:jmaher) from comment #19)
> bear, that looks correct!  Thanks for double checking.

cool, let me get this landed so it will be picked up during the next reconfig

Comment 21

[Mike Taylor [:bear]]

Comment on attachment 586460 [details] [diff] [review]
buildbot-configs change with chunking

committed changeset 5297:f784166f4b2e

Comment 22

[Mike Taylor [:bear]]

Comment on attachment 587129 [details] [diff] [review]
buildbotcustom change with chunking

committed changeset 1997:ca617f34ac5b

Comment 23

[Aki Sasaki (not active)]

(In reply to Mike Taylor [:bear] from comment #22)
> Comment on attachment 587129 [details] [diff] [review]
> buildbotcustom change with chunking
> 
> committed changeset 1997:ca617f34ac5b

I suspect you didn't 'hg push' .

Comment 24

[Aki Sasaki (not active)]

Live in production.

Comment 25

[Mike Taylor [:bear]]

spotted in tbpl \o/