Closed Bug 1406213 Opened 3 years ago Closed 3 years ago

test-verify sometimes times out, still

Categories

(Testing :: General, defect)

defect
Not set

Tracking

(firefox58 fixed)

RESOLVED FIXED
mozilla58
Tracking Status
firefox58 --- fixed

People

(Reporter: gbrown, Assigned: gbrown)

References

(Blocks 1 open bug)

Details

Attachments

(1 file, 1 obsolete file)

As in bug 1398933, but now very rare.

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=b25870b6590be5e881987ec7092be735e668ee80&filter-searchStr=test-verify

https://public-artifacts.taskcluster.net/PaDjHz2-TRyOcTdXQvuKbA/0/public/logs/live_backing.log

[task 2017-10-05T11:19:55.848Z] 11:19:55     INFO - Running post-action listener: find_tests_for_verification
...
[task 2017-10-05T12:48:27.679Z] 12:48:27     INFO -  :::
[task 2017-10-05T12:48:27.680Z] 12:48:27     INFO -  ::: Test verification summary for: /builds/worker/workspace/build/tests/xpcshell/tests/toolkit/mozapps/extensions/test/xpcshell/test_system_update_newset.js
[task 2017-10-05T12:48:27.681Z] 12:48:27     INFO -  :::
[task 2017-10-05T12:48:27.683Z] 12:48:27     INFO -  ::: 1. Run each test 10 times, sequentially. : Pass
[task 2017-10-05T12:48:27.684Z] 12:48:27     INFO -  ::: 2. Run each test 10 times, sequentially, in chaos mode. : Pass
[task 2017-10-05T12:48:27.685Z] 12:48:27     INFO -  :::
[task 2017-10-05T12:48:27.686Z] 12:48:27     INFO -  ::: Test verification PASSED
[task 2017-10-05T12:48:27.687Z] 12:48:27     INFO -  :::

...

[task 2017-10-05T12:48:42.165Z] 12:48:42     INFO -  TEST-PASS | xpcshell.ini:toolkit/mozapps/extensions/test/xpcshell/test_system_repository.js | took 1494ms
[task 2017-10-05T12:48:42.186Z] 12:48:42     INFO -  TEST-START | xpcshell.ini:toolkit/mozapps/extensions/test/xpcshell/test_system_repository.js

[taskcluster:error] Task timeout after 5400 seconds. Force killing container.
[taskcluster 2017-10-05 12:48:42.603Z] === Task Finished ===
[taskcluster 2017-10-05 12:48:42.604Z] Unsuccessful task run with exit code: -1 completed in 5401.75 seconds
I think the existing strategy is sound, as is the implementation for detecting when the max verification time is exceeded.

The trouble is that there are *3* loops to break out of when that max time is exceeded:

for suite_category ...:
    for suite in category ...:
        for file-to-be-verified ...:

This should break out of all of them.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7f9f38687f6c1f84cd47332350448429f8526ed1
Attachment #8916192 - Flags: review?(jmaher)
Found and fixed another small problem. Now it's perfect!
Attachment #8916192 - Attachment is obsolete: true
Attachment #8916192 - Flags: review?(jmaher)
Attachment #8916202 - Flags: review?(jmaher)
Comment on attachment 8916202 [details] [diff] [review]
break out of all loops when verification time exceeded + do not reset start_time

Review of attachment 8916202 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/mozharness/scripts/desktop_unittest.py
@@ +763,4 @@
>                          # Verification has run out of time. That is okay! Stop running
>                          # tests so that a task timeout is not triggered, and so that
>                          # (partial) results are made available in a timely manner.
>                          self.info("TinderboxPrint: Verification too long: Not all tests were verified.<br/>")

shouldn't this be an error vs an info, we probably will never see this message.
Attachment #8916202 - Flags: review?(jmaher) → review+
An error will turn the job orange. TV normally runs out of time on a merge, or other change that affects lots of files -- not conditions that we want to cause bug filing, etc. I want the job to be green, and I don't mind if we are sometimes unaware that verification was not complete.
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7fd42c49a756
Ensure test verification gives up after max time exceeded; r=jmaher
Blocks: 1204281
https://hg.mozilla.org/mozilla-central/rev/7fd42c49a756
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla58
Blocks: 1411358
No longer blocks: 1411358
You need to log in before you can comment on or make changes to this bug.