test-verify sometimes times out

RESOLVED FIXED in Firefox 57

Status

RESOLVED FIXED
a year ago
a year ago

People

(Reporter: gbrown, Assigned: gbrown)

Tracking

(Blocks: 1 bug)

Trunk
mozilla57
Points:
---

Firefox Tracking Flags

(firefox57 fixed)

Details

Attachments

(1 attachment)

(Assignee)

Description

a year ago
We expect test-verify to sometimes have more tests to verify than it can verify in a reasonable amount of time. In these cases, test-verify should give up and complete before timing out.

I see some exceptions, where the job runs for too long.

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=296f26c8422126c7e80812173ebc38e548566234&filter-searchStr=test-verify&filter-tier=1&filter-tier=2&filter-tier=3

This is a merge, so having lots of tests to verify is not unexpected.

https://public-artifacts.taskcluster.net/bkKYRtAIStim-PpcHhxxIg/0/public/logs/live_backing.log

[task 2017-09-09T10:22:55.667898Z] 10:22:55     INFO - Calling ['/builds/worker/workspace/build/venv/bin/python', '-u', '/builds/worker/workspace/build/tests/mochitest/runtests.py', ... '--verify-max-time=900', '--verify', u'intl/l10n/test/dom/test_domloc_mutations.html'] with output_timeout 1000
...
[task 2017-09-09T11:41:12.971029Z] 11:41:12     INFO - Calling ['/builds/worker/workspace/build/venv/bin/python', '-u', '/builds/worker/workspace/build/tests/xpcshell/runxpcshelltests.py', ... '--verify-max-time=900', '--verify', u'toolkit/components/extensions/test/xpcshell/test_ext_unknown_permissions.js'] with output_timeout 1000
...
[task 2017-09-09T11:46:59.887397Z] 11:46:59     INFO - Calling ['/builds/worker/workspace/build/venv/bin/python', '-u', '/builds/worker/workspace/build/tests/xpcshell/runxpcshelltests.py', ... '--verify-max-time=900', '--verify', u'services/sync/tests/unit/test_syncengine_sync.js'] with output_timeout 1000
...
[task 2017-09-09T11:50:09.064917Z] 11:50:09     INFO -  TEST-START | services/sync/tests/unit/test_syncengine_sync.js

[taskcluster:error] Task timeout after 5400 seconds. Force killing container.
(Assignee)

Comment 1

a year ago
We triggered "Verification too long", but didn't stop verification:

toolkit/components/extensions/test/mochitest/test_ext_notifications.html
...
[task 2017-09-09T11:28:26.407889Z] 11:28:26     INFO - TinderboxPrint: Verification too long: Not all tests were verified.<br/>
...
intl/l10n/test/test_localization.js

I think it bailed out of one suite (mochitests) but continued with the next suite (xpcshell tests).
(Assignee)

Comment 2

a year ago
Created attachment 8906800 [details] [diff] [review]
bail out of nested loop when verify max time exceeded

A small fix for a small problem!
Attachment #8906800 - Flags: review?(jmaher)
Comment on attachment 8906800 [details] [diff] [review]
bail out of nested loop when verify max time exceeded

Review of attachment 8906800 [details] [diff] [review]:
-----------------------------------------------------------------

oh, this should solve many of the issues.
Attachment #8906800 - Flags: review?(jmaher) → review+
(Assignee)

Updated

a year ago
Blocks: 1357513

Comment 4

a year ago
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/b805cc36637b
Ensure test-verify bails out before timing out; r=jmaher

Comment 5

a year ago
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/f559248aac81
Follow-up, fix cannot compare timedelta bustage, on a CLOSED TREE
https://hg.mozilla.org/mozilla-central/rev/b805cc36637b
https://hg.mozilla.org/mozilla-central/rev/f559248aac81
Status: NEW → RESOLVED
Last Resolved: a year ago
status-firefox57: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla57
(Assignee)

Updated

a year ago
See Also: → bug 1406213
You need to log in before you can comment on or make changes to this bug.