Closed Bug 1471112 Opened 6 years ago Closed 6 years ago

Split mochitest-gl into sub-jobs

Categories

(Testing :: General, enhancement, P1)

Product:
Testing
Component:
General
Version:
Version 3
Type:
enhancement

Tracking

(firefox63 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla63
Tracking Flags:
Tracking Status
firefox63 --- fixed

People

(Reporter: jgilbert, Assigned: jgilbert)

References

Details

Attachments

(2 files, 1 obsolete file)

Bug 1471112 - Update manifest generation and regenerate.
59 bytes, text/x-review-board-request
jgilbert
: review+
Details
Bug 1471112 - Split mochitest-gl into mochitest-webgl[1,2]-[core,ext] -
59 bytes, text/x-review-board-request
gbrown
: review+
Details 
(I'm not sure what component this should be in)

I believe this will help us keep more-important subsets of tests running on more platforms. For instance, we can probably run on Android and Linux again if we restrict what tests we run to a subset.

We can also reduce our machine load, since we can run less useful tests less frequently.
Comment on attachment 8987721 [details]
Bug 1471112 - Split mochitest-gl into mochitest-webgl[1,2]-[core,ext] -

https://reviewboard.mozilla.org/r/252980/#review259522


Code analysis found 3 defects in this patch:
 - 3 defects found by mozlint

You can run this analysis locally with:
 - `./mach lint path/to/file` (JS/Python)


If you see a problem in this automated review, please report it here: http://bit.ly/2y9N9Vx


::: taskcluster/ci/test/mochitest.yml:517
(Diff revision 1)
> +    suite: mochitest/mochitest-webgl2-deqp
> +    treeherder-symbol: M(gl2d)
> +    run-on-projects:
> +        by-test-platform:
> +            windows10-64-ccov/.*: []  # Do not run on Windows ccov, see bug 1419475.
> +            default: [] # Don't run this for now.

Warning: Too few spaces before comment [yamllint: comments]

::: testing/mozharness/configs/android/android_common.py:27
(Diff revision 1)
> +            "--log-errorsummary=%(error_summary_file)s",
> +            "--screenshot-on-fail",
> +            "--subsuite=" + name,
> +        ],
> +    }
> +    

Error: Blank line contains whitespace [flake8: W293]

::: testing/mozharness/configs/android/android_hw.py:27
(Diff revision 1)
> +            "--screenshot-on-fail",
> +            "--subsuite=" + name,
> +            "--deviceSerial=%(device_serial)s",
> +        ]
> +    }
> +        

Error: Blank line contains whitespace [flake8: W293]
Comment on attachment 8987721 [details]
Bug 1471112 - Split mochitest-gl into mochitest-webgl[1,2]-[core,ext] -

https://reviewboard.mozilla.org/r/252980/#review260048

No serious concerns here, but let's work through another iteration.

::: taskcluster/ci/test/mochitest.yml:357
(Diff revision 1)
>              windows10-64-ccov/.*: []  # Do not run on Windows ccov, see bug 1419475.
>              default: built-projects
>      virtualization: virtual-with-gpu
>      chunks:
>          by-test-platform:
>              android-em-7.0-x86/opt: 1

Same as default - let's remove it.

::: taskcluster/ci/test/mochitest.yml:358
(Diff revision 1)
>              default: built-projects
>      virtualization: virtual-with-gpu
>      chunks:
>          by-test-platform:
>              android-em-7.0-x86/opt: 1
> -            android-em.*: 10
> +            android-em.*: 2

Since these are not running currently, let's remove the android-em* also.

Then this can collapse down to chunks: 1.

::: taskcluster/ci/test/mochitest.yml:368
(Diff revision 1)
> +            default: true
> +    loopback-video: true
> +    max-run-time:
> +        by-test-platform:
> +            windows.*: 5400
> +            android-em.*: 7200

Similarly, unless you have plans to run webgl on android emulators, maybe this is a good opportunity to remove all the android-em settings. (Just something to consider.)

::: taskcluster/ci/test/mochitest.yml:482
(Diff revision 1)
> +    virtualization: virtual-with-gpu
> +    chunks:
> +        by-test-platform:
> +            android-em-7.0-x86/opt: 1
> +            android-em.*: 8
> +            default: 4

Are 4 chunks needed on linux/osx? Will it be enough on windows? Can you show me a try run to give me an idea of run-time?

::: taskcluster/taskgraph/test/test_try_option_syntax.py:39
(Diff revision 1)
>      unittest_task('mochitest-browser-chrome-e10s', 'linux64/opt'),
>      unittest_task('mochitest-chrome', 'linux/debug', 'debug'),
> -    unittest_task('mochitest-webgl', 'linux/debug', 'debug'),
> +    unittest_task('mochitest-gl', 'linux/debug', 'debug'),
> +    unittest_task('mochitest-webgl1-core', 'linux/debug', 'debug'),
> +    unittest_task('mochitest-webgl1-ext', 'linux/debug', 'debug'),
> +    unittest_task('mochitest-webgl1-deqp', 'linux/debug', 'debug'),

I think this is the only reference to mochitest-webgl1-deqp. Accidental?

::: taskcluster/taskgraph/test/test_try_option_syntax.py:193
(Diff revision 1)
> -        parameters = {'try_options': parse_message('try: -u mochitest-webgl')}
> +        parameters = {'try_options': parse_message('try: -u mochitest-webgl1-core')}
>          tos = TryOptionSyntax(parameters, graph_with_jobs, GRAPH_CONFIG)
> -        self.assertEqual(sorted(tos.unittests), sorted([{'test': 'mochitest-webgl'}]))
> +        self.assertEqual(sorted(tos.unittests), sorted([{'test': 'mochitest-webgl1-core'}]))
>  
>      def test_u_alias(self):
> -        "-u mochitest-gl sets unittests=[mochitest-webgl]"
> +        "-u mochitest-gl sets unittests=[mochitest-webgl1-core]"

Test description is wrong/incomplete.
Attachment #8987721 - Flags: review?(gbrown) → review-
Assignee: nobody → jgilbert
Priority: -- → P1
Oops, I'll push the fixed csets up tomorrow. Here's the latest try run: (nice and green)
https://treeherder.mozilla.org/#/jobs?repo=try&revision=476ac511afe6af66e28145f85f466c66548acd17
Thanks. Yes, no concerns there.
Comment on attachment 8987721 [details]
Bug 1471112 - Split mochitest-gl into mochitest-webgl[1,2]-[core,ext] -

https://reviewboard.mozilla.org/r/252980/#review260458
Attachment #8987721 - Flags: review?(gbrown) → review+
Comment on attachment 8987721 [details]
Bug 1471112 - Split mochitest-gl into mochitest-webgl[1,2]-[core,ext] -

https://reviewboard.mozilla.org/r/252980/#review260460


Code analysis found 1 defect in this patch:
 - 1 defect found by mozlint

You can run this analysis locally with:
 - `./mach lint path/to/file` (JS/Python)


If you see a problem in this automated review, please report it here: http://bit.ly/2y9N9Vx


::: testing/mozbase/moztest/moztest/resolve.py:25
(Diff revision 2)
>  
>  
>  MOCHITEST_CHUNK_BY_DIR = 4
>  MOCHITEST_TOTAL_CHUNKS = 5
>  
> +def WebglSuite(name):

Error: Expected 2 blank lines, found 1 [flake8: E302]
Pushed by jgilbert@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/df70300f00cd
Update manifest generation and regenerate.
https://hg.mozilla.org/integration/mozilla-inbound/rev/4c92c4b611c6
Split mochitest-gl into mochitest-webgl[1,2]-[core,ext] - r=gbrown
:gbrown, what's our target ideal chunk duration range? [10,30]?
Flags: needinfo?(gbrown)
Comment on attachment 8987721 [details]
Bug 1471112 - Split mochitest-gl into mochitest-webgl[1,2]-[core,ext] -

https://reviewboard.mozilla.org/r/252980/#review260048

> Are 4 chunks needed on linux/osx? Will it be enough on windows? Can you show me a try run to give me an idea of run-time?

The runtimes look OK to me. We can adjust them if not.
Backed out 2 changesets (bug 1471112) for mochitest failures on test_2_conformance2__textures__image__tex-3d-rgba16f-rgba-float.html.

Backout: https://hg.mozilla.org/integration/mozilla-inbound/rev/f7e24207d24528a6481cdc2510a74c040224a194

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=4c92c4b611c6c423b567572c4cc5d49a87af1fdc&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=success&selectedJob=185419725

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=185419725&repo=mozilla-inbound&lineNumber=1540

[task 2018-06-28T19:36:51.074Z]     INFO -  1 INFO TEST-START | dom/canvas/test/webgl-conf/generated/test_2_conformance2__textures__image__tex-3d-rgba16f-rgba-float.html
[task 2018-06-28T19:37:11.710Z]     INFO -  Buffered messages logged at 19:36:57
[task 2018-06-28T19:37:11.710Z]     INFO -  2 INFO TEST-PASS | dom/canvas/test/webgl-conf/generated/test_2_conformance2__textures__image__tex-3d-rgba16f-rgba-float.html | A valid string reason is expected
[task 2018-06-28T19:37:11.711Z]     INFO -  3 INFO TEST-PASS | dom/canvas/test/webgl-conf/generated/test_2_conformance2__textures__image__tex-3d-rgba16f-rgba-float.html | Reason cannot be empty
[task 2018-06-28T19:37:11.712Z]     INFO -  Buffered messages finished
[task 2018-06-28T19:37:11.712Z]     INFO -  4 INFO TEST-UNEXPECTED-FAIL | dom/canvas/test/webgl-conf/generated/test_2_conformance2__textures__image__tex-3d-rgba16f-rgba-float.html | Unable to fetch WebGL rendering context for Canvas
Flags: needinfo?(jgilbert)
(In reply to Jeff Gilbert [:jgilbert] from comment #12)
> :gbrown, what's our target ideal chunk duration range? [10,30]?

I don't know of an authoritative answer. [10,30] seems reasonable. I might say more like [20,30]. Too long is bad for waiting for results / sheriffing / backouts. Too short is inefficient -- there's probably about 3 minutes of machine time overhead, setting up the task, preparing the python environment, etc. I think you've made reasonable choices for your number of chunks.
Flags: needinfo?(gbrown)
In addition to the failures in comment 14, there were failures in test_try_option_syntax.py:

https://taskcluster-artifacts.net/AppPKzwsSgGogDxyr53Bgw/0/public/logs/live_backing.log

You can run that test with

  mach python-test --python 2 --subsuite taskgraph

Here's a quick fix. If you like it, land it with r=me or r=you.
Attachment #8988587 - Flags: review?(jgilbert)
Comment on attachment 8988587 [details] [diff] [review]
follow-up for test failure in test_try_option_syntax.py

Review of attachment 8988587 [details] [diff] [review]:
-----------------------------------------------------------------

Cool, I'll just fold it in.
Attachment #8988587 - Flags: review?(jgilbert) → review+
'common-tests' is really 'linux-tests'. :/

That should do it:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c18569fbd87cbc8d6c14ee061c17442b2dcf316c
Flags: needinfo?(jgilbert)
Attachment #8988587 - Attachment is obsolete: true
Attachment #8987720 - Flags: review?(jgilbert)
Comment on attachment 8987720 [details]
Bug 1471112 - Update manifest generation and regenerate.

https://reviewboard.mozilla.org/r/252978/#review260602
Attachment #8987720 - Flags: review?(jgilbert) → review+
Pushed by jgilbert@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1f26d0868935
Update manifest generation and regenerate. r=jgilbert
https://hg.mozilla.org/integration/autoland/rev/7f2d98706144
Split mochitest-gl into mochitest-webgl[1,2]-[core,ext] - r=gbrown
https://hg.mozilla.org/mozilla-central/rev/1f26d0868935
https://hg.mozilla.org/mozilla-central/rev/7f2d98706144
Status: NEW → RESOLVED
Closed: 6 years ago
status-firefox63: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla63
Backout by rgurzau@mozilla.com:
https://hg.mozilla.org/mozilla-central/rev/a009b5249a4b
Backed out 2 changesets for failing webgl2 tests on OSX
Backed out 2 changesets (bug 1471112) for failing webgl2 tests on OSX 

Backout link: https://hg.mozilla.org/mozilla-central/rev/a009b5249a4b78a889fdc5ffcf55ad51715cc686

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=7f2d9870614484433ef66a83639907a33f9c2852

Log link: https://treeherder.mozilla.org/logviewer.html#?job_id=185585975&repo=autoland&lineNumber=1717

Log snippet: 
07:30:51     INFO - TEST-START | dom/canvas/test/webgl-conf/generated/test_2_conformance2__textures__image_bitmap_from_image_bitmap__tex-3d-r11f_g11f_b10f-rgb-float.html
07:30:52     INFO - GECKO(751) | JavaScript error: http://mochi.test:8888/tests/dom/canvas/test/webgl-conf/checkout/js/tests/tex-image-and-sub-image-utils.js, line 27: ReferenceError: WebGLTestUtils is not defined
07:30:52     INFO - GECKO(751) | JavaScript error: http://mochi.test:8888/tests/dom/canvas/test/webgl-conf/checkout/js/tests/tex-image-and-sub-image-3d-with-image-bitmap-from-image-bitmap.js, line 25: ReferenceError: WebGLTestUtils is not defined
07:36:18     INFO - TEST-INFO | started process screencapture
07:36:18     INFO - TEST-INFO | screencapture: exit 0
07:36:18     INFO - Buffered messages logged at 07:30:51
07:36:18     INFO - TEST-PASS | dom/canvas/test/webgl-conf/generated/test_2_conformance2__textures__image_bitmap_from_image_bitmap__tex-3d-r11f_g11f_b10f-rgb-float.html | A valid string reason is expected 
07:36:18     INFO - TEST-PASS | dom/canvas/test/webgl-conf/generated/test_2_conformance2__textures__image_bitmap_from_image_bitmap__tex-3d-r11f_g11f_b10f-rgb-float.html | Reason cannot be empty 
07:36:18     INFO - Buffered messages finished
07:36:18     INFO - TEST-UNEXPECTED-FAIL | dom/canvas/test/webgl-conf/generated/test_2_conformance2__textures__image_bitmap_from_image_bitmap__tex-3d-r11f_g11f_b10f-rgb-float.html | Test timed out.
Flags: needinfo?(jgilbert)
Status: RESOLVED → REOPENED
status-firefox63: fixed → ---
Resolution: FIXED → ---
This literally isn't allowed to happen:

> 07:30:51     INFO - TEST-START | dom/canvas/test/webgl-conf/generated/test_2_conformance2__textures__image_bitmap_from_image_bitmap__tex-3d-r11f_g11f_b10f-rgb-float.html
> 07:30:52     INFO - GECKO(751) | JavaScript error: http://mochi.test:8888/tests/dom/canvas/test/webgl-conf/checkout/js/tests/tex-image-and-sub-image-utils.js, line 27: ReferenceError: WebGLTestUtils is not defined
> 07:30:52     INFO - GECKO(751) | JavaScript error: http://mochi.test:8888/tests/dom/canvas/test/webgl-conf/checkout/js/tests/tex-image-and-sub-image-3d-with-image-bitmap-from-image-bitmap.js, line 25: ReferenceError: WebGLTestUtils is not defined
> 07:36:18     INFO - TEST-INFO | started process screencapture
> 07:36:18     INFO - TEST-INFO | screencapture: exit 0
> 07:36:18     INFO - Buffered messages logged at 07:30:51
> 07:36:18     INFO - TEST-PASS | dom/canvas/test/webgl-conf/generated/test_2_conformance2__textures__image_bitmap_from_image_bitmap__tex-3d-r11f_g11f_b10f-rgb-float.html | A valid string reason is expected 
> 07:36:18     INFO - TEST-PASS | dom/canvas/test/webgl-conf/generated/test_2_conformance2__textures__image_bitmap_from_image_bitmap__tex-3d-r11f_g11f_b10f-rgb-float.html | Reason cannot be empty 
> 07:36:18     INFO - Buffered messages finished
> 07:36:18     INFO - TEST-UNEXPECTED-FAIL | dom/canvas/test/webgl-conf/generated/test_2_conformance2__textures__image_bitmap_from_image_bitmap__tex-3d-r11f_g11f_b10f-rgb-float.html | Test timed out. 

Somehow WebGLTestUtils is not being loaded properly, but only sometimes.
I'm going to disable this wing of the test suite for now, but this is not my bug.
Flags: needinfo?(jgilbert)
Also MozReview refuses to let me upload new copies of patches now, because it doesn't realize it was backed out.
Pushed by jgilbert@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/90a68ae60daf
Update manifest generation and regenerate. r=jgilbert
https://hg.mozilla.org/integration/autoland/rev/f773c716cbcc
Split mochitest-gl into mochitest-webgl[1,2]-[core,ext] - r=gbrown
https://hg.mozilla.org/mozilla-central/rev/90a68ae60daf
https://hg.mozilla.org/mozilla-central/rev/f773c716cbcc
Status: REOPENED → RESOLVED
Closed: 6 years ago → 6 years ago
status-firefox63: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla63
Sweet, as far as webgl intermittents are concerned, that's a success.
Depends on: 1476318
Blocks: 1479078
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: