Closed Bug 1398933 Opened 8 years ago Closed 8 years ago

test-verify sometimes times out

Categories

(Testing :: General, defect)

defect
Not set
normal

Tracking

(firefox57 fixed)

RESOLVED FIXED
mozilla57
Tracking Status
firefox57 --- fixed

People

(Reporter: gbrown, Assigned: gbrown)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

We expect test-verify to sometimes have more tests to verify than it can verify in a reasonable amount of time. In these cases, test-verify should give up and complete before timing out. I see some exceptions, where the job runs for too long. https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=296f26c8422126c7e80812173ebc38e548566234&filter-searchStr=test-verify&filter-tier=1&filter-tier=2&filter-tier=3 This is a merge, so having lots of tests to verify is not unexpected. https://public-artifacts.taskcluster.net/bkKYRtAIStim-PpcHhxxIg/0/public/logs/live_backing.log [task 2017-09-09T10:22:55.667898Z] 10:22:55 INFO - Calling ['/builds/worker/workspace/build/venv/bin/python', '-u', '/builds/worker/workspace/build/tests/mochitest/runtests.py', ... '--verify-max-time=900', '--verify', u'intl/l10n/test/dom/test_domloc_mutations.html'] with output_timeout 1000 ... [task 2017-09-09T11:41:12.971029Z] 11:41:12 INFO - Calling ['/builds/worker/workspace/build/venv/bin/python', '-u', '/builds/worker/workspace/build/tests/xpcshell/runxpcshelltests.py', ... '--verify-max-time=900', '--verify', u'toolkit/components/extensions/test/xpcshell/test_ext_unknown_permissions.js'] with output_timeout 1000 ... [task 2017-09-09T11:46:59.887397Z] 11:46:59 INFO - Calling ['/builds/worker/workspace/build/venv/bin/python', '-u', '/builds/worker/workspace/build/tests/xpcshell/runxpcshelltests.py', ... '--verify-max-time=900', '--verify', u'services/sync/tests/unit/test_syncengine_sync.js'] with output_timeout 1000 ... [task 2017-09-09T11:50:09.064917Z] 11:50:09 INFO - TEST-START | services/sync/tests/unit/test_syncengine_sync.js [taskcluster:error] Task timeout after 5400 seconds. Force killing container.
We triggered "Verification too long", but didn't stop verification: toolkit/components/extensions/test/mochitest/test_ext_notifications.html ... [task 2017-09-09T11:28:26.407889Z] 11:28:26 INFO - TinderboxPrint: Verification too long: Not all tests were verified.<br/> ... intl/l10n/test/test_localization.js I think it bailed out of one suite (mochitests) but continued with the next suite (xpcshell tests).
A small fix for a small problem!
Attachment #8906800 - Flags: review?(jmaher)
Comment on attachment 8906800 [details] [diff] [review] bail out of nested loop when verify max time exceeded Review of attachment 8906800 [details] [diff] [review]: ----------------------------------------------------------------- oh, this should solve many of the issues.
Attachment #8906800 - Flags: review?(jmaher) → review+
Blocks: test-verify
Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/b805cc36637b Ensure test-verify bails out before timing out; r=jmaher
Pushed by gbrown@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/f559248aac81 Follow-up, fix cannot compare timedelta bustage, on a CLOSED TREE
See Also: → 1406213
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: