Closed Bug 1064002 Opened 10 years ago Closed 9 years ago

Add {mochitest-gl,robocop,jsreftest,crashtest} suite_definitions to in-tree configurations

Categories

(Release Engineering :: Applications: MozharnessCore, defect)

All
Android
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: nalexander, Assigned: armenzg)

References

Details

(Whiteboard: [easier-mozharness])

Attachments

(18 files, 6 obsolete files)

761 bytes, patch
ahal
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
9.26 KB, patch
Details | Diff | Splinter Review
12.38 KB, patch
ahal
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
964 bytes, patch
chmanchester
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
5.15 KB, patch
armenzg
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
1.16 KB, patch
armenzg
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
38 bytes, text/x-review-board-request
Details
38 bytes, text/x-review-board-request
Details
38 bytes, text/x-review-board-request
Details
38 bytes, text/x-review-board-request
Details
38 bytes, text/x-review-board-request
Details
38 bytes, text/x-review-board-request
Details
38 bytes, text/x-review-board-request
armenzg
: review+
Details
38 bytes, text/x-review-board-request
armenzg
: review+
Details
38 bytes, text/x-review-board-request
armenzg
: review+
Details
38 bytes, text/x-review-board-request
armenzg
: review+
Details
38 bytes, text/x-review-board-request
armenzg
: review+
Details
38 bytes, text/x-review-board-request
armenzg
: review+
Details
In mozharness, we have robocop definitions [1] like:

289         "robocop-1": {
290             "category": "mochitest",
291             "extra_args": ["--total-chunks", "4", "--this-chunk", "1", "--robocop-path=../..",
292                 "--robocop-ids=fennec_ids.txt", "--robocop=robocop.ini"],
293         },

I'd like a "category": "robocop" that includes the "--robocop" arguments.  I think said categories come from the in tree "suite_definitions" defined at [2]; can somebody confirm that?  (ahal?)

This would make it easier to purge fennec_ids.txt from the tree and from automation.  It might also make it easier to configure Robocop.

It seems like the jsreftest definitions could use the same love.

In my opinion, the goal should be for the mozharness configurations that are *not* in tree to include only the chunking information, but perhaps this is not a reasonable state of affairs.

[1] http://mxr.mozilla.org/build/source/mozharness/configs/android/androidarm.py#289

[2] http://mxr.mozilla.org/mozilla-central/source/testing/config/mozharness/android_arm_config.py#7
Flags: needinfo?(ahalberstadt)
Yeah, that looks like the file you want. The mozharness script uses the "category" value to find the proper directory, so we'd likely have to restructure how this works a bit (or else define 'abs_robocop_dir' = 'abs_mochitest_dir'):
http://mxr.mozilla.org/build/source/mozharness/scripts/android_emulator_unittest.py#387

Ideally I'd like everything (including chunking) defined in-tree. We have a vague notion that this is important and should be done, but haven't exactly nailed down where it belongs in Q4 goals.
Flags: needinfo?(ahalberstadt)
Assignee: nobody → armenzg
Whiteboard: [easier-mozharness]
The only difference between mochitest and robocop jobs are the ones defined in the mh config.
They grab common values from the "mochitest" category.

I think the best approach here is to create a "robocop" category in the in-tree config which defines:
--robocop-path=../.. --robocop-ids=fennec_ids.txt --robocop=robocop.ini
while the mochitest category defines:
--run-only-tests=android23.json

We would need to uplift the changes to every branch before landing the mozharness changes.

From mochitest-1:
/builds/slave/test/build/venv/bin/python -u /builds/slave/test/build/tests/mochitest/runtestsremote.py --autorun --close-when-done --dm_trans=sut --console-level=INFO --app=org.mozilla.fennec --remote-webserver=10.0.2.2 --xre-path=/builds/slave/test/build/hostutils/xre --utility-path=/builds/slave/test/build/hostutils/bin --deviceIP=127.0.0.1 --devicePort=20701 --http-port=8854 --ssl-port=4454 --certificate-path=/builds/slave/test/build/tests/certs --symbols-path=https://ftp-ssl.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android/1415892129/fennec-36.0a1.en-US.android-arm.crashreporter-symbols.zip --quiet --log-raw=/builds/slave/test/build/blobber_upload_dir/raw_structured_logs.log --total-chunks=16 --this-chunk=1 --run-only-tests=android23.json

From robocop-1:
/builds/slave/test/build/venv/bin/python -u /builds/slave/test/build/tests/mochitest/runtestsremote.py --autorun --close-when-done --dm_trans=sut --console-level=INFO --app=org.mozilla.fennec --remote-webserver=10.0.2.2 --xre-path=/builds/slave/test/build/hostutils/xre --utility-path=/builds/slave/test/build/hostutils/bin --deviceIP=127.0.0.1 --devicePort=20701 --http-port=8854 --ssl-port=4454 --certificate-path=/builds/slave/test/build/tests/certs --symbols-path=https://ftp-ssl.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android/1415892129/fennec-36.0a1.en-US.android-arm.crashreporter-symbols.zip --quiet --log-raw=/builds/slave/test/build/blobber_upload_dir/raw_structured_logs.log --total-chunks=4 --this-chunk=1 --robocop-path=../.. --robocop-ids=fennec_ids.txt --robocop=robocop.ini
For xcpshell, we will have to land a different patch for mozilla-release's in-tree config.
We will have to add this:
--manifest=tests/xpcshell_android.ini

mc: http://hg.mozilla.org/mozilla-central/file/default/testing/config/mozharness/android_arm_config.py#l35
ma: http://hg.mozilla.org/releases/mozilla-aurora/file/9c775c8888ef/testing/config/mozharness/android_arm_config.py#l34
mb: http://hg.mozilla.org/releases/mozilla-beta/file/117eb4e49c72/testing/config/mozharness/android_arm_config.py#l35
mr: http://hg.mozilla.org/releases/mozilla-release/file/f2bdbc945da2/testing/config/mozharness/android_arm_config.py#l34

In m-c:
/builds/slave/test/build/venv/bin/python -u /builds/slave/test/build/tests/xpcshell/remotexpcshelltests.py --deviceIP=127.0.0.1 --devicePort=20701 --xre-path=/builds/slave/test/build/hostutils/xre --testing-modules-dir=/builds/slave/test/build/tests/modules --apk=/builds/slave/test/build/fennec-36.0a1.en-US.android-arm.apk --no-logfiles --symbols-path=https://ftp-ssl.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android/1415892129/fennec-36.0a1.en-US.android-arm.crashreporter-symbols.zip --manifest=tests/xpcshell.ini --log-raw=/builds/slave/test/build/blobber_upload_dir/raw_structured_logs.log --total-chunks=3 --this-chunk=1

In m-b:
/builds/slave/test/build/venv/bin/python -u /builds/slave/test/build/tests/xpcshell/remotexpcshelltests.py --deviceIP=127.0.0.1 --devicePort=20701 --xre-path=/builds/slave/test/build/hostutils/xre --testing-modules-dir=/builds/slave/test/build/tests/modules --apk=/builds/slave/test/build/fennec-34.0.en-US.android-arm.apk --no-logfiles --symbols-path=https://ftp-ssl.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-beta-android/1415891288/fennec-34.0.en-US.android-arm.crashreporter-symbols.zip --manifest=tests/xpcshell.ini --total-chunks=3 --this-chunk=1

In m-r:
/builds/slave/test/build/venv/bin/python -u /builds/slave/test/build/tests/xpcshell/remotexpcshelltests.py --deviceIP=127.0.0.1 --devicePort=20701 --xre-path=/builds/slave/test/build/hostutils/xre --testing-modules-dir=/builds/slave/test/build/tests/modules --apk=/builds/slave/test/build/fennec-33.1.en-US.android-arm.apk --no-logfiles --symbols-path=https://ftp-ssl.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-release-android/1415656612/fennec-33.1.en-US.android-arm.crashreporter-symbols.zip --total-chunks=3 --this-chunk=1 --manifest=tests/xpcshell_android.ini
Attachment #8522461 - Flags: review?(ahalberstadt)
Attached file MozReview Request: bz://1064002/mynick (obsolete) —
Attachment #8522475 - Flags: review?(ahalberstadt)
/r/601 - Bug 1064002 - Modify Android in-tree configs to allow further experimentation

Pull down this commit:

hg pull review -r 0ec9371ab0c8cf52c2f7bb47015570752bcf38f1
Attached file MozReview Request: bz://1064002/mynick (obsolete) —
/r/605 - Bug 1064002 - Remove information of androidarm.py which exists in the tree. We can't land this w/o landing previous patch

Pull down this commit:

hg pull review -r 2437947ec53662eef1969b32421761c991724532
Attachment #8522475 - Flags: review?(ahalberstadt) → review+
https://reviewboard.mozilla.org/r/599/#review287

This is awesome! I'm kind of surprised that adding '--total-chunks' works, but I guess you've tested on try. Is it possible to add '--total-chunks' to desktop unittests as well? If so is there a bug on file?
Comment on attachment 8522461 [details] [diff] [review]
[m-r] add android manifest

Review of attachment 8522461 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/config/mozharness/android_arm_config.py
@@ +31,5 @@
>              "options": ["--deviceIP=%(device_ip)s", "--devicePort=%(device_port)s",
>                  "--xre-path=%(xre_path)s", "--testing-modules-dir=%(modules_dir)s",
>                  "--apk=%(installer_path)s", "--no-logfiles",
>                  "--symbols-path=%(symbols_path)s",
> +                "--manifest=tests/xpcshell_android.ini",

So I removed xpcshell_android.ini awhile ago, but I think this makes sense because this is what the default is if it isn't defined in-tree. Just clarifying. Though, I'm surprised this wasn't defined already.
Attachment #8522461 - Flags: review?(ahalberstadt) → review+
I wonder what I enabled that it post my pushes to the bugs.

I found a syntax typo in my previous patch which I'm retesting on Ash and Try.

If all goes well, I will land the trunk changes and start uplifting.

Without the mozharness change:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=104e5feaa4ac

With the mozharness change:
https://tbpl.mozilla.org/?tree=Ash&rev=3b9d91bee0e1
(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #18)
> Comment on attachment 8522461 [details] [diff] [review]
> [m-r] add android manifest
> 
> https://hg.mozilla.org/releases/mozilla-release/rev/85d4fdc6d572

https://hg.mozilla.org/releases/mozilla-esr31/rev/f01ae373216d
/r/637 - Bug 1064002 - Modify Android in-tree configs to allow further experimentation
/r/639 - Fix syntax error
/r/647 - Specify testsdir for each Android suite

Pull down these commits:

hg pull review -r 93d21fa78e0c785bdd9141c7b9dd8088cecc6826
/r/651 - Bug 1064002 - Remove information of androidarm.py which exists in the tree. We can't land this w/o landing previous patch
/r/653 - Read testsdir if it exists
/r/655 - Use suite_name instead of suite_category and fallback to suite_category if testsdir is not defined

Pull down these commits:

hg pull review -r d9e78846c07c14d1b25ba6d9cb4c830b2903c7d1
This is now working on Ash.
/r/637 - Bug 1064002 - Modify Android in-tree configs to allow further experimentation
/r/639 - Fix syntax error
/r/647 - Specify testsdir for each Android suite

Pull down these commits:

hg pull review -r 93d21fa78e0c785bdd9141c7b9dd8088cecc6826
/r/651 - Bug 1064002 - Remove information of androidarm.py which exists in the tree. We can't land this w/o landing previous patch
/r/653 - Read testsdir if it exists
/r/655 - Use suite_name instead of suite_category and fallback to suite_category if testsdir is not defined

Pull down these commits:

hg pull review -r d9e78846c07c14d1b25ba6d9cb4c830b2903c7d1
/r/651 - Bug 1064002 - Remove information of androidarm.py which exists in the tree. We can't land this w/o landing previous patch
/r/653 - Read testsdir if it exists
/r/655 - Use suite_name instead of suite_category and fallback to suite_category if testsdir is not defined

Pull down these commits:

hg pull review -r d9e78846c07c14d1b25ba6d9cb4c830b2903c7d1
Attachment #8523185 - Flags: review?(ahalberstadt)
/r/651 - Bug 1064002 - Remove information of androidarm.py which exists in the tree. We can't land this w/o landing previous patch
/r/653 - Read testsdir if it exists
/r/655 - Use suite_name instead of suite_category and fallback to suite_category if testsdir is not defined

Pull down these commits:

hg pull review -r d9e78846c07c14d1b25ba6d9cb4c830b2903c7d1
Attachment #8523180 - Flags: review?(ahalberstadt)
/r/637 - Bug 1064002 - Modify Android in-tree configs to allow further experimentation
/r/639 - Fix syntax error
/r/647 - Specify testsdir for each Android suite

Pull down these commits:

hg pull review -r 93d21fa78e0c785bdd9141c7b9dd8088cecc6826
/r/651 - Bug 1064002 - Remove information of androidarm.py which exists in the tree. We can't land this w/o landing previous patch
/r/653 - Read testsdir if it exists
/r/655 - Use suite_name instead of suite_category and fallback to suite_category if testsdir is not defined

Pull down these commits:

hg pull review -r d9e78846c07c14d1b25ba6d9cb4c830b2903c7d1
Attachment #8523185 - Flags: review?(jlund)
Attachment #8522477 - Attachment is obsolete: true
/r/637 - Bug 1064002 - Modify Android in-tree configs to allow further experimentation
/r/639 - Fix syntax error
/r/647 - Specify testsdir for each Android suite
/r/707 - Add testsdir to crashtests for Android configs

Pull down these commits:

hg pull review -r a2c5f56ec27599c063834fb5bada3a6a04f95afd
/r/651 - Bug 1064002 - Remove information of androidarm.py which exists in the tree. We can't land this w/o landing previous patch
/r/653 - Read testsdir if it exists
/r/655 - Use suite_name instead of suite_category and fallback to suite_category if testsdir is not defined
/r/709 - Fix typo

Pull down these commits:

hg pull review -r 204353fcab3cacfd17bc86b61252d99a3b14cdf8
https://reviewboard.mozilla.org/r/635/#review319

::: testing/config/mozharness/android_arm_config.py
(Diff revision 2)
> +                "--run-only-tests=android23.json"

nit: would be nice to add trailing commas to all the last list items just for consistency
Attachment #8523180 - Flags: review?(ahalberstadt) → review+
Comment on attachment 8523185 [details]
MozReview Request: bz://1064002/armenzg

This looks good to me, but I'd rather have Jordan do the final sign off.
Attachment #8523185 - Flags: review?(ahalberstadt) → feedback+
Attachment #8523185 - Flags: review?(jlund) → review+
Keywords: leave-open
Attachment #8523185 - Flags: review?(jlund)
Attachment #8523185 - Flags: review?(ahalberstadt)
Attachment #8523185 - Flags: review+
Attachment #8523185 - Flags: feedback+
/r/651 - Bug 1064002 - Remove information of androidarm.py which exists in the tree. We can't land this w/o landing previous patch
/r/653 - Read testsdir if it exists
/r/655 - Use suite_name instead of suite_category and fallback to suite_category if testsdir is not defined
/r/709 - Fix typo
/r/813 - Remove --manifest for xcpshell

Pull down these commits:

hg pull review -r b8daccc949472300d61bf33ec3cec39f6e99dc5f
sorry had to back this out in https://hg.mozilla.org/mozilla-central/rev/7d17b594834f for causing Bug 1102115

Seems the problem is 

05:26 < vaibhav1994> Tomcat|sheriffduty: basically --test-manifest and --run-only-tests cannot be used together
Flags: needinfo?(armenzg)
Blocks: 1102115
My apologies about that! I had tested it on try. I will review the issues.
Flags: needinfo?(armenzg)
Attachment #8523185 - Flags: review?(ahalberstadt)
It was on my try push as well. Unfortunately I though that the 1 and 2 that are green after they were confirming that it was an intermittent while they actually were different jobs.

Pushing a new patch to try:
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=cec4080ae299

And to Ash for post-landing mozharness work:
https://tbpl.mozilla.org/?tree=Ash&rev=207a65587960
Summary: Add {robocop,jsreftest} suite_definitions to in-tree configurations → Add {mochitest-gl,robocop,jsreftest,crashtest} suite_definitions to in-tree configurations
The tests on try all passed, including the mochitest-gl ones running with the right arguments.
Attachment #8523180 - Flags: review+ → review?(ahalberstadt)
/r/637 - Bug 1064002 - We need to remove the parameters that are specific to the original suites until we land the mozharness changes
/r/639 - Add note with regards to not yet in use sections

Pull down these commits:

hg pull review -r bac5fac8730fdeb9495f13343e1a5e8303a43357
Attachment #8523180 - Flags: review?(ahalberstadt) → review+
https://reviewboard.mozilla.org/r/635/#review383

Thanks, looks good!

::: testing/config/mozharness/android_arm_config.py
(Diff revision 3)
> +        # Bug 1064002 - Not yet in use 

nit: these comments all have an extra whitespace
/r/637 - Bug 1064002 - We need to remove the parameters that are specific to the original suites until we land the mozharness changes
/r/639 - Add note with regards to not yet in use sections
/r/829 - Fix extra white spaces

Pull down these commits:

hg pull review -r 1650517126a3729b378e09361dc584f21c69b39b
Attachment #8523180 - Flags: review+ → review?(ahalberstadt)
Comment on attachment 8523180 [details]
MozReview Request: bz://1064002/armenzg

Recuperating missing r+
Attachment #8523180 - Flags: review?(ahalberstadt) → review+
Attachment #8523180 - Flags: review+ → review?(ahalberstadt)
/r/637 - Bug 1064002 - We need to remove the parameters that are specific to the original suites until we land the mozharness changes
/r/639 - Add note with regards to not yet in use sections
/r/829 - Fix extra white spaces
/r/831 - Fix extra white spaces

Pull down these commits:

hg pull review -r 5ac226783cfa830a8db44f00fbc99a5e86f4c6c7
Comment on attachment 8523180 [details]
MozReview Request: bz://1064002/armenzg

Silly reviewboard.
Attachment #8523180 - Flags: review?(ahalberstadt) → review+
Comment on attachment 8523185 [details]
MozReview Request: bz://1064002/armenzg

I'm going to push a different mozharness review request.
Attachment #8523185 - Flags: review?(jlund)
/r/651 - Remove --manifest for xcpshell
/r/653 - Bug 1064002 - Put back the original suites to its original values

Pull down these commits:

hg pull review -r e8c3575bce60231969152c9cc20e6e4607e059ba
Attachment #8523185 - Flags: review?(jlund)
Once the mozharness bit lands this will follow.
I will need to test on try once the mozharness bits land.

Oh! how I long for pinning mozharness from the tree! :)
Testing on Ash the mozharness changes and I can see it working.
https://tbpl.mozilla.org/?tree=Ash&jobname=Android&rev=1f6c9176557a

Crashtest job using the right parameters:
https://tbpl.mozilla.org/php/getParsedLog.php?id=53176580&tree=Ash&full=1
10:51:39     INFO - Running on test-1 the command /builds/slave/test/build/venv/bin/python -u /builds/slave/test/build/tests/reftest/remotereftest.py --app=org.mozilla.fennec --ignore-window-size --bootstrap --remote-webserver=10.0.2.2 --xre-path=/builds/slave/test/build/hostutils/xre --utility-path=/builds/slave/test/build/hostutils/bin --deviceIP=127.0.0.1 --devicePort=20701 --http-port=8854 --ssl-port=4454 --httpd-path reftest/components --symbols-path=https://ftp-ssl.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/ash-android/1416592334/fennec-36.0a1.en-US.android-arm.crashreporter-symbols.zip --total-chunks=2 tests/testing/crashtest/crashtests.list --this-chunk=1

From mozharness:
10:47:55     INFO -  'test_suite_definitions': {'crashtest-1': {'category': 'crashtest',
10:47:55     INFO -                                             'extra_args': ('--this-chunk=1',)},

From in-tree:
    80                 "--total-chunks=2",
    81                 "tests/testing/crashtest/crashtests.list",
    82             ],
    83         },


Reftest job using the right parameters:
https://tbpl.mozilla.org/php/getParsedLog.php?id=53176993&tree=Ash&full=1
10:51:40     INFO - Running on test-1 the command /builds/slave/test/build/venv/bin/python -u /builds/slave/test/build/tests/reftest/remotereftest.py --app=org.mozilla.fennec --ignore-window-size --bootstrap --remote-webserver=10.0.2.2 --xre-path=/builds/slave/test/build/hostutils/xre --utility-path=/builds/slave/test/build/hostutils/bin --deviceIP=127.0.0.1 --devicePort=20701 --http-port=8854 --ssl-port=4454 --httpd-path reftest/components --symbols-path=https://ftp-ssl.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/ash-android/1416592334/fennec-36.0a1.en-US.android-arm.crashreporter-symbols.zip --total-chunks=16 --this-chunk=1 tests/layout/reftests/reftest.list

From mozharness:
10:48:00     INFO -                             'reftest-1': {'category': 'reftest',
10:48:00     INFO -                                           'extra_args': ('--total-chunks=16',
10:48:00     INFO -                                                          '--this-chunk=1',
10:48:00     INFO -                                                          'tests/layout/reftests/reftest.list')},

From in-tree:
    63                 "--symbols-path=%(symbols_path)s",
    64                 # Bug 1064002 - Land once mozharness changes land
    65                 #"--total-chunks=16",
    66                 #"tests/layout/reftests/reftest.list",
Attachment #8523185 - Flags: review?(jlund) → review+
(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #62)
> https://hg.mozilla.org/build/mozharness/rev/96810ab51f0d

This change once merge into production will do the following.
- Every suite that is *not* mochitest, reftest and xpcshell will start reading their info from the tree instead of from mozharness for *every* branch

We now have to land another change that will switch mochitest, reftest and xpcshell.
It will require uplifts and then a mozharness change.
Attachment #8522475 - Attachment is obsolete: true
Comment on attachment 8523180 [details]
MozReview Request: bz://1064002/armenzg

This was reviewed by ahal in reviewboard. I don't know why it says that I reviewed it.
Attachment #8523180 - Flags: checked-in+
Comment on attachment 8523185 [details]
MozReview Request: bz://1064002/armenzg

Checked-in. Not yet in production.
Attachment #8523185 - Flags: checked-in+
Comment on attachment 8523185 [details]
MozReview Request: bz://1064002/armenzg

and backed out :(

default: https://hg.mozilla.org/build/mozharness/rev/0049e8c8e2e9
production: https://hg.mozilla.org/build/mozharness/rev/cdbe77f88f8e

reason: robocop failure: http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-aurora-android/1416872493/mozilla-aurora_ubuntu64_vm_mobile_test-robocop-3-bm118-tests1-linux64-build48.txt.gz

looks like we still somewhere depend on abs_%s_dir being: mochitest || reftest || xpcshell

not sure why this test didn't show up in your tests.
Attachment #8523185 - Flags: checked-in+ → checked-in-
Well, it seems that with all the micro-commits and hg handling I did locally I ended up landing missing the changes to the script.

I'm re-testing on Ash (so far so good):
https://tbpl.mozilla.org/?tree=Ash&rev=6a32858f85ff
here's the missing part of the patch:
http://hg.mozilla.org/users/armenzg_mozilla.com/mozharness/rev/fc37f42c4b0d
(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #70)
> https://hg.mozilla.org/build/mozharness/rev/6c0f0bca4f54

I triaged all issues that could be related in https://tbpl.mozilla.org/?tree=Ash&rev=6a32858f85ff

Landing now the patch as originally intended.
Comment on attachment 8526893 [details] [diff] [review]
Start using the proper categories for each suite (cannot land before attachment 8545428 [details] [diff] [review] and needs to be uplifted)

Once the mozharness change makes it into the production branch we can land this change.
Attachment #8526893 - Flags: review?(ahalberstadt)
Once attachment 8226893 is landed and uplifted everywhere.
Attachment #8529108 - Flags: review?(jlund)
No longer blocks: 1040383
(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #74)
> https://tbpl.mozilla.org/?tree=Try&rev=1c840a6a08d3

The try push went well. Waiting for the reviews.
<philor> uh oh, did anything that might have involved android mochitest-gl land in this morning's reconfig?
<philor> armenzg: could a side-effect of bug 1064002 have been "make android 2.3 mochitest-gl1 and -gl2 fail every single run on every tree"?
<armenzg> philor, let me check
<bhearsum> yes, it did
<philor> armenzg: https://treeherder.mozilla.org/logviewer.html#?job_id=4289120&repo=mozilla-inbound - successful run marked as failure

I'm re-triggering on trunk and Aurora to determine if it is due to the mozharness change:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&jobname=Android.*gl
https://tbpl.mozilla.org/?tree=Mozilla-Aurora&jobname=Android.*gl

The only difference so far is the order of the arguments:
--test-manifest=gl.json --this-chunk=1
--this-chunk=1 --test-manifest=gl.json

gl-1 failing job:
http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android/1417182580/mozilla-inbound_ubuntu64_vm_mobile_test-mochitest-gl-1-bm118-tests1-linux64-build128.txt.gz
Command:
/builds/slave/test/build/venv/bin/python -u /builds/slave/test/build/tests/mochitest/runtestsremote.py --autorun --close-when-done --dm_trans=sut --console-level=INFO --app=org.mozilla.fennec --remote-webserver=10.0.2.2 --xre-path=/builds/slave/test/build/hostutils/xre --utility-path=/builds/slave/test/build/hostutils/bin --deviceIP=127.0.0.1 --devicePort=20701 --http-port=8854 --ssl-port=4454 --certificate-path=/builds/slave/test/build/tests/certs --symbols-path=https://ftp-ssl.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android/1417182580/fennec-36.0a1.en-US.android-arm.crashreporter-symbols.zip --quiet --log-raw=/builds/slave/test/build/blobber_upload_dir/raw_structured_logs.log --total-chunks=2 --test-manifest=gl.json --this-chunk=1

gl-1 successful job:
https://tbpl.mozilla.org/php/getParsedLog.php?id=53753300&tree=Mozilla-Inbound&full=1

/builds/slave/test/build/venv/bin/python -u /builds/slave/test/build/tests/mochitest/runtestsremote.py --autorun --close-when-done --dm_trans=sut --console-level=INFO --app=org.mozilla.fennec --remote-webserver=10.0.2.2 --xre-path=/builds/slave/test/build/hostutils/xre --utility-path=/builds/slave/test/build/hostutils/bin --deviceIP=127.0.0.1 --devicePort=20701 --http-port=8854 --ssl-port=4454 --certificate-path=/builds/slave/test/build/tests/certs --symbols-path=https://ftp-ssl.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android/1417181261/fennec-36.0a1.en-US.android-arm.crashreporter-symbols.zip --quiet --log-raw=/builds/slave/test/build/blobber_upload_dir/raw_structured_logs.log --total-chunks=2 --this-chunk=1 --test-manifest=gl.json
(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #73)
> Created attachment 8529108 [details] [diff] [review]
> Make reftest, mochitest and xpchsell read the in-tree configs
> 
> Once attachment 8226893 is landed and uplifted everywhere.

sanity check. My interpretation is that previous dependent work on this had a regression and was backed out in comment 78. Are you still wanting a review on 8529108 or do you need to fix something first?
Comment on attachment 8529108 [details] [diff] [review]
Make reftest, mochitest and xpchsell read the in-tree configs (cannot land before attachment 8526893 [details] [diff] [review] is landed)

I don't need the review right now. It is likely not to change but I will first land and test everything blocking this patch.
Attachment #8529108 - Flags: review?(jlund)
Comment on attachment 8526893 [details] [diff] [review]
Start using the proper categories for each suite (cannot land before attachment 8545428 [details] [diff] [review] and needs to be uplifted)

Review of attachment 8526893 [details] [diff] [review]:
-----------------------------------------------------------------

This is so awesome. Can't wait for all jobs to have in-tree defined chunks.
Attachment #8526893 - Flags: review?(ahalberstadt) → review+
Depends on: 1112200
It seems that for some harnesses the order of parameters can be troublesome. [1]

This fails:
--total-chunks=2 --test-manifest=gl.json --this-chunk=1

This succeeds:
--total-chunks=2 --this-chunk=1 --test-manifest=gl.json

[1]
https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=bd4f7fbc017b
/builds/slave/test/build/venv/bin/python -u /builds/slave/test/build/tests/mochitest/runtestsremote.py --autorun --close-when-done --dm_trans=sut --console-level=INFO --app=org.mozilla.fennec --remote-webserver=10.0.2.2 --xre-path=/builds/slave/test/build/hostutils/xre --utility-path=/builds/slave/test/build/hostutils/bin --deviceIP=127.0.0.1 --devicePort=20701 --http-port=8854 --ssl-port=4454 --certificate-path=/builds/slave/test/build/tests/certs --symbols-path=https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-bd4f7fbc017b/try-android-api-9/fennec-37.0a1.en-US.android-arm.crashreporter-symbols.zip --quiet --log-raw=/builds/slave/test/build/blobber_upload_dir/raw_structured_logs.log --total-chunks=2 --test-manifest=gl.json --this-chunk=1
No parsing? No passing.
Attachment #8538684 - Flags: review?(jlund)
Attachment #8538684 - Flags: review?(jlund) → review+
Blocks: 1083347
This is what my current try push is testing.

If this works, then we will follow up with:
* attachment 8526893 [details] [diff] [review] - uplift everywhere
* attachment 8529108 [details] [diff] [review]
Attachment #8538684 - Attachment is obsolete: true
I hit a hiccup since I merged code from default and it had issues (bug 1109346).
I'm re-triggering few jobs to see where we stand.
Comment on attachment 8523185 [details]
MozReview Request: bz://1064002/armenzg

This patch does not show properly on MozReview.
I will re-attach.
Attachment #8523185 - Attachment is obsolete: true
Comment on attachment 8545428 [details] [diff] [review]
Original patch + add mochitest-gl as a parsing category

This has worked on Try:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=ecdb1c21ae76&filter-searchStr=gl1

ahal, this patch is the combination of https://reviewboard.mozilla.org/r/649 and attachment 8538684 [details] [diff] [review] which already have been r+, however, MozReview does not show it clearly (probably a bug) and would like to get your approval one more time before landing it.

I will not land this before I see all remaining jobs running green, however, I'm confident of it after seeing that gl1 job run properly.
Attachment #8545428 - Attachment description: wip - android configs → Original patch + add mochitest-gl as a parsing category
Attachment #8545428 - Flags: review?(ahalberstadt)
Attachment #8526893 - Attachment description: android_mc.diff.final → Start using the proper categories for each suite (cannot land before attachment 8545428 and needs to be uplifted)
Attachment #8529108 - Attachment description: Make reftest, mochitest and xpchsell read the in-tree configs → Make reftest, mochitest and xpchsell read the in-tree configs (cannot land before attachment 8526893 is landed)
I just noticed this removed the raw log option from the in-tree android_arm_config.py for xpcshell, so we aren't getting logs from those jobs. Is there a patch pending here to put that back in?
Flags: needinfo?(armenzg)
Comment on attachment 8545428 [details] [diff] [review]
Original patch + add mochitest-gl as a parsing category

Review of attachment 8545428 [details] [diff] [review]:
-----------------------------------------------------------------

Lgtm!
Attachment #8545428 - Flags: review?(ahalberstadt) → review+
Attached patch xpcshell.diffSplinter Review
Not intentional.
It needs to be uplifted.
I can take care of it on Monday.
Flags: needinfo?(armenzg)
Attachment #8546692 - Flags: review?(cmanchester)
Comment on attachment 8546692 [details] [diff] [review]
xpcshell.diff

Review of attachment 8546692 [details] [diff] [review]:
-----------------------------------------------------------------

Great, thanks!
Attachment #8546692 - Flags: review?(cmanchester) → review+
Attachment #8545428 - Flags: checked-in+
Attachment #8546692 - Flags: checked-in+
(In reply to Wes Kocher (:KWierso) from comment #105)
> (In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from
> comment #102)
> > https://hg.mozilla.org/releases/mozilla-esr31/rev/97610709b8af
> 
> This appears to have broken xpcshell and robocop tests on esr31:
> https://treeherder.mozilla.org/#/jobs?repo=mozilla-
> esr31&revision=97610709b8af&filter-searchStr=android%202.3%20armv6

Support for "--log-raw" merged to central in October, this should be backed out of the esr31.
Thanks for fixing it. My apologies!
Flags: needinfo?(armenzg)
Comment on attachment 8547812 [details] [diff] [review]
Start using the proper categories for each suite (cannot land before attachment 8545428 [details] [diff] [review] and needs to be uplifted)

We can uplift it after it merges to central.

All jobs are running properly:
Mochitest [1]
Mochitest-gl [2]
Robocop-1 [3]
Xpcshell [4]
Crashtest [5]
Jsreftest [6]
Reftest [7]

[1]
08:44:42     INFO - Running on test-1 the command /builds/slave/test/build/venv/bin/python -u /builds/slave/test/build/tests/mochitest/runtestsremote.py --autorun --close-when-done --dm_trans=sut --console-level=INFO --app=org.mozilla.fennec --remote-webserver=10.0.2.2 --xre-path=/builds/slave/test/build/hostutils/xre --utility-path=/builds/slave/test/build/hostutils/bin --deviceIP=127.0.0.1 --devicePort=20701 --http-port=8854 --ssl-port=4454 --certificate-path=/builds/slave/test/build/tests/certs --symbols-path=https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-f970daecdc34/try-android-api-9/fennec-38.0a1.en-US.android-arm.crashreporter-symbols.zip --quiet --log-raw=/builds/slave/test/build/blobber_upload_dir/mochitest-14_raw.log --total-chunks=16 --run-only-tests=android23.json --this-chunk=14

[2]
10:44:54     INFO - Running on test-1 the command /builds/slave/test/build/venv/bin/python -u /builds/slave/test/build/tests/mochitest/runtestsremote.py --autorun --close-when-done --dm_trans=sut --console-level=INFO --app=org.mozilla.fennec --remote-webserver=10.0.2.2 --xre-path=/builds/slave/test/build/hostutils/xre --utility-path=/builds/slave/test/build/hostutils/bin --deviceIP=127.0.0.1 --devicePort=20701 --http-port=8854 --ssl-port=4454 --certificate-path=/builds/slave/test/build/tests/certs --symbols-path=https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-f970daecdc34/try-android-api-9/fennec-38.0a1.en-US.android-arm.crashreporter-symbols.zip --quiet --log-raw=/builds/slave/test/build/blobber_upload_dir/mochitest-gl-1_raw.log --total-chunks=2 --test-manifest=gl.json --this-chunk=1

[3]
08:44:55     INFO - Running on test-1 the command /builds/slave/test/build/venv/bin/python -u /builds/slave/test/build/tests/mochitest/runtestsremote.py --autorun --close-when-done --dm_trans=sut --console-level=INFO --app=org.mozilla.fennec --remote-webserver=10.0.2.2 --xre-path=/builds/slave/test/build/hostutils/xre --utility-path=/builds/slave/test/build/hostutils/bin --deviceIP=127.0.0.1 --devicePort=20701 --http-port=8854 --ssl-port=4454 --certificate-path=/builds/slave/test/build/tests/certs --symbols-path=https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-f970daecdc34/try-android-api-9/fennec-38.0a1.en-US.android-arm.crashreporter-symbols.zip --quiet --log-raw=/builds/slave/test/build/blobber_upload_dir/robocop-1_raw.log --total-chunks=4 --robocop-path=../.. --robocop-ids=fennec_ids.txt --robocop=robocop.ini --this-chunk=1

[4]
08:45:07     INFO - Running on test-1 the command /builds/slave/test/build/venv/bin/python -u /builds/slave/test/build/tests/xpcshell/remotexpcshelltests.py --deviceIP=127.0.0.1 --devicePort=20701 --xre-path=/builds/slave/test/build/hostutils/xre --testing-modules-dir=/builds/slave/test/build/tests/modules --apk=/builds/slave/test/build/fennec-38.0a1.en-US.android-arm.apk --no-logfiles --symbols-path=https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-f970daecdc34/try-android-api-9/fennec-38.0a1.en-US.android-arm.crashreporter-symbols.zip --manifest=tests/xpcshell.ini --log-raw=/builds/slave/test/build/blobber_upload_dir/xpcshell-3_raw.log --total-chunks=3 --this-chunk=3

[5]
07:50:39     INFO - Running on test-1 the command /builds/slave/test/build/venv/bin/python -u /builds/slave/test/build/tests/reftest/remotereftest.py --app=org.mozilla.fennec --ignore-window-size --bootstrap --remote-webserver=10.0.2.2 --xre-path=/builds/slave/test/build/hostutils/xre --utility-path=/builds/slave/test/build/hostutils/bin --deviceIP=127.0.0.1 --devicePort=20701 --http-port=8854 --ssl-port=4454 --httpd-path /builds/slave/test/build/tests/modules --symbols-path=https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-f970daecdc34/try-android-api-9/fennec-38.0a1.en-US.android-arm.crashreporter-symbols.zip --total-chunks=2 tests/testing/crashtest/crashtests.list --this-chunk=1

[6]
07:50:16     INFO - Running on test-1 the command /builds/slave/test/build/venv/bin/python -u /builds/slave/test/build/tests/reftest/remotereftest.py --app=org.mozilla.fennec --ignore-window-size --bootstrap --remote-webserver=10.0.2.2 --xre-path=/builds/slave/test/build/hostutils/xre --utility-path=/builds/slave/test/build/hostutils/bin --deviceIP=127.0.0.1 --devicePort=20701 --http-port=8854 --ssl-port=4454 --httpd-path /builds/slave/test/build/tests/modules --symbols-path=https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-f970daecdc34/try-android-api-9/fennec-38.0a1.en-US.android-arm.crashreporter-symbols.zip ../jsreftest/tests/jstests.list --total-chunks=6 --extra-profile-file=jsreftest/tests/user.js --this-chunk=1

[7]
07:50:10     INFO - Running on test-1 the command /builds/slave/test/build/venv/bin/python -u /builds/slave/test/build/tests/reftest/remotereftest.py --app=org.mozilla.fennec --ignore-window-size --bootstrap --remote-webserver=10.0.2.2 --xre-path=/builds/slave/test/build/hostutils/xre --utility-path=/builds/slave/test/build/hostutils/bin --deviceIP=127.0.0.1 --devicePort=20701 --http-port=8854 --ssl-port=4454 --httpd-path /builds/slave/test/build/tests/modules --symbols-path=https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/try-builds/armenzg@mozilla.com-f970daecdc34/try-android-api-9/fennec-38.0a1.en-US.android-arm.crashreporter-symbols.zip --total-chunks=16 tests/layout/reftests/reftest.list --this-chunk=1
Attachment #8547812 - Flags: checked-in+
Attachment #8526893 - Attachment is obsolete: true
Attached patch robocop.diffSplinter Review
The last of the release jobs that is trying to use --log-raw.
Attachment #8548422 - Flags: review?(cmanchester)
Attachment #8548422 - Flags: review?(cmanchester)
Attachment #8548422 - Flags: review+
Attachment #8548422 - Flags: checked-in+
Mass comment: With mozharness moving into the tree [1] and pinning of mozharness we can assume that anything left (if any) in this bug will be easily taken care of.

[1] http://jordan-lund.ghost.io/mozharness-steps-into-the-forest
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Attachment #8523180 - Attachment is obsolete: true
Attachment #8618307 - Flags: review+
Attachment #8618308 - Flags: review+
Attachment #8618309 - Flags: review+
Attachment #8618310 - Flags: review+
Attachment #8618311 - Flags: review+
Attachment #8618312 - Flags: review+
Removing leave-open keyword from resolved bugs, per :sylvestre.
Keywords: leave-open
You need to log in before you can comment on or make changes to this bug.