1400967 - Promote linux64 test-verify (TV) task to tier 2

Status: RESOLVED FIXED

(Testing :: General, enhancement)

Description

[Geoff Brown [:gbrown]]

Most of the obvious bugs in test verification have been worked out now. TV is currently running as tier 3, but I think it meets requirements for tier 2. Let's check and if it's okay, move to tier 2.

Comment 1

[Geoff Brown [:gbrown]]

I'm pretty happy with https://developer.mozilla.org/en-US/docs/Test_Verification and that is linked from https://developer.mozilla.org/docs/Mozilla/QA/Automated_testing.

I'd be even happier to hear good or bad feedback on my efforts there. :)

Comment 2

[Geoff Brown [:gbrown]]

TV successfully detected bug 1400528, first on inbound and then on merges:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=0533dbf2956818762fbbe7dbcfad738fe8383d89&filter-searchStr=test-verify&filter-tier=1&filter-tier=2&filter-tier=3

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=27e7b4261e4b5a016e09adb5332a181e43ab520e&filter-searchStr=test-verify&filter-tier=1&filter-tier=2&filter-tier=3

https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=5e5e2372b2c05bf68b4cd0908bdf22d8c4575a9d&filter-searchStr=test-verify&filter-tier=1&filter-tier=2&filter-tier=3

Comment 3

[Geoff Brown [:gbrown]]

TV predicted failures in browser/extensions/formautofill/test/mochitest/test_basic_creditcard_autocomplete_form.html, new test added by bug 1393374:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=c0769d228dd3ba666d6e9b02896bebef7e996ad5&filter-searchStr=test-verify&filter-tier=1&filter-tier=2&filter-tier=3

...the test was backed out for frequent failures: https://bugzilla.mozilla.org/show_bug.cgi?id=1393374#c38.

Comment 4

[Geoff Brown [:gbrown]]

Here's a more complicated issue:

Bug 1373853, https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=f1e37c409e4f46f7a646dcc0ec131e817fa0f325&filter-searchStr=test-verify&filter-tier=1&filter-tier=2&filter-tier=3 modified two tests: browser_extension_controlled.js and browser_homepages_filter_aboutpreferences.js. browser_extension_controlled.js failed verification on all platforms; on debug, browser_homepages_filter_aboutpreferences.js failed due to leaks.

The changeset was backed out for leaks: https://bugzilla.mozilla.org/show_bug.cgi?id=1373853#c20.

Interestingly, browser_homepages_filter_aboutpreferences.js failed verification on the backout, due to a pre-existing intermittent failure, bug 1297634:

https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=47f15c732421cedfdf82b8799c89942caf57c655&filter-searchStr=test-verify&filter-tier=1&filter-tier=2&filter-tier=3

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

wow, that is interesting!  I think this will happen more often as we have older tests which do not pass test-verify.

Comment 6

[Geoff Brown [:gbrown]]

Attachment: run test-verify as tier 2 (tier 3 -> tier 2 promotion)

I think test-verify satisfies the requirements for tier 2:
 - I am the owner;
 - logs are generated by existing mochitest/reftest/xpcshell test harnesses; treeherder failure summaries look good to me -- same as existing test suites;
 - docs at https://developer.mozilla.org/en-US/docs/Test_Verification.

I don't think it will cause trouble.

But test-verify is "different" from a lot of other test jobs, and I want to make sheriffs aware of how it works and how it is unique:
 - TV is intended as an "early warning system" for identifying the introduction of intermittent test failures.
 - When a mochitest, reftest, or xpcshell test is modified on a push, TV runs that particular test over and over until it fails (orange job, standard failure messages), or until max iterations are achieved (green job, all's well), or until TV runs out of time (green job, maybe all's well?).
 - As a consequence, when a faulty test is introduced, TV will turn orange and report the test failure on that push, but on the next and subsequent pushes, TV may not run or may run green (because it is now running other tests - the ones modified on those pushes).
 - When no supported tests have been modified, TV runs green and finishes quickly (a few minutes). When one, short-duration test is modified, TV can usually verify that in just a few minutes more. But if the test is longer-duration, or there are many tests to verify, the job will run longer. After about an hour, if there's more work to do, TV will give up and not verify the remaining tests. (If no failures found, the job will be green.) This happens sometimes on merges, because collectively, there are lots of modified tests. Treeherder's "Job Details" pane has a good summary of what was or was not verified.

I'll announce all this on dev.platform, but wanted to reach out to you/sheriffs as well. If any questions or concerns arise, please let me know.

Comment 7

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Comment on attachment 8912833 [details] [diff] [review]
run test-verify as tier 2 (tier 3 -> tier 2 promotion)

R+, pending a post to dev-platform. Maybe also send an email to [e-mail]?

Comment 8

[Geoff Brown [:gbrown]]

https://groups.google.com/forum/#!topic/mozilla.dev.platform/sf3SjDAKL60

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/2d88b252dcc3
Run test-verify as a tier-2 test task for linux64; r=kwierso

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/295bbf8b45ef
Follow-up for decision task bustage on a CLOSED TREE

Comment 11

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backed out for failing gecko decision task: test-windows10-64-asan/opt-test-verify-e10s (tier 2) cannot depend on build-win64-asan/opt (tier 3) / for failing gecko decision task: No test-platform matching 'windows10-64-asan/opt' nor 'default' found while determining item run-on-projects in test-verify:

https://hg.mozilla.org/integration/mozilla-inbound/rev/28698488ac9d3c2009d33f3c40caa4b67bf5ffef
https://hg.mozilla.org/integration/mozilla-inbound/rev/749f9f45479040ff05ee93b04e5e6549c548e830

Failure log follow-up: https://treeherder.mozilla.org/logviewer.html#?job_id=134449371&repo=mozilla-inbound
> Exception: No test-platform matching 'windows10-64-asan/opt' nor 'default' found while determining item run-on-projects in test-verify

Failure log initial push: https://treeherder.mozilla.org/logviewer.html#?job_id=134441043&repo=mozilla-inbound
> Exception: test-windows10-64-asan/opt-test-verify-e10s (tier 2) cannot depend on build-win64-asan/opt (tier 3)

Comment 12

[Geoff Brown [:gbrown]]

Sorry about that.

I see now that most test tasks have a special tier entry for windows10-64-asan. test-verify was previously exempt from that because it was running as tier 3 anyway. Now the windows10-64-asan tier needs to be explicit.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1298f4c322f1ea47f92764ad3f1c19b2a9e91a05
https://treeherder.mozilla.org/#/jobs?repo=try&revision=8be9c34b796ee5de98a84baf01c23cf2d6296827

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8222a2787c3c
Run test-verify as a tier-2 test task for linux64; r=kwierso

Comment 14

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/8222a2787c3c