Closed Bug 1233953 Opened 9 years ago Closed 8 years ago

Intermittent Linux debug command timed out: 7200 seconds elapsed running ['/tools/buildbot/bin/python', '-u', 'scripts/scripts/desktop_unittest.py', '--cfg', 'unittests/linux_unittest.py', '--mochitest-suite', 'plain-chunked', '--total-chunks', '5'

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: aryx, Unassigned)

References

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

patch_Bug_1233953_bb.patch
1.48 KB, patch
rail
: review+
Details | Diff | Splinter Review 
https://treeherder.mozilla.org/logviewer.html#?job_id=18814565&repo=mozilla-inbound

command timed out: 7200 seconds elapsed running ['/tools/buildbot/bin/python', 'scripts/scripts/desktop_unittest.py', '--cfg', 'unittests/linux_unittest.py', '--mochitest-suite', 'plain-chunked', '--total-chunks', '5', '--this-chunk', '2', '--blob-upload-branch', 'mozilla-inbound', '--download-symbols', 'true'], attempting to kill
patch in bug 1073442 to increase chunks on debug mochitest.
Okay, this one is in a worrying increasing trend yet again :(
Seems that lately new jobs from reftest-no-accel suite need more than 7200 seconds to finish and that's why we have new failed tests
(example https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1233953&startday=2016-07-04&endday=2016-07-10&tree=all).
Similar with the solution from Bug 1073442 we have 2 options: extend the timeout from 7200 -> 9000  or split reftest-no-accel chunks.
splitting seems to be the most ideal route, looking at m-c, all 8 chunks run in <15 minutes for linux64 debug Ru.  I assume the Ru jobs were timing out/hanging for some other reason.
From what I found seems that on linux64 debug Ru things are going well but for Linux32 debug Ru things are slower.
https://treeherder.mozilla.org/#/jobs?repo=mozilla-esr45&revision=d5b5d3e88b564dde53c37f7a9a682faf9377bea2&filter-searchStr=ru
I checked the raw logs for both(Linux32 and Linux64) and it seems that Linux32 take more time to do the same 6800 tests (for example same tests between 1000 and 2000 were done in 10 minutes for Linux64 debug and in 18 minutes for Linux32 debug),so I think that a splitting from 2 to (4 or 6) chunks for Linux 32 debug could be helpfull.
Flags: needinfo?(jmaher)
oh, this is esr45, that would explain the confusion.  In that case we should go to 6 chunks.  Oddly we started running these tests on June 3rd, are they required for esr45?  I will be on pto, I would verify they are wanted to run for linux debug (not sure if they are really needed) and if they are, split them to 6 chunks.  ask RyanVM for review, I am on PTO for 3 days.
Flags: needinfo?(jmaher)
This patch will split reftest_no_accell tests from 2 to 6 chunks for Linux 32 debug for mochitests 48+.
Attachment #8770447 - Flags: review?(ryanvm)
Comment on attachment 8770447 [details] [diff] [review]
patch_Bug_1233953_bb.patch

Not a peer, sorry.
Attachment #8770447 - Flags: review?(ryanvm) → review?(rail)
Comment on attachment 8770447 [details] [diff] [review]
patch_Bug_1233953_bb.patch

LGTM!
Attachment #8770447 - Flags: review?(rail) → review+
Summary: Intermittent Linux debug command timed out: 7200 seconds elapsed running ['/tools/buildbot/bin/python', 'scripts/scripts/desktop_unittest.py', '--cfg', 'unittests/linux_unittest.py', '--mochitest-suite', 'plain-chunked', '--total-chunks', '5', '--this-chu → Intermittent Linux debug command timed out: 7200 seconds elapsed running ['/tools/buildbot/bin/python', '-u', 'scripts/scripts/desktop_unittest.py', '--cfg', 'unittests/linux_unittest.py', '--mochitest-suite', 'plain-chunked', '--total-chunks', '5'
Fixed by bug 1317568.
Blocks: 1317568
Status: NEW → RESOLVED
Closed: 8 years ago
No longer depends on: 1211889
Resolution: --- → FIXED
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: