1141941 - reduce desktop builds max timeout limit without output to a sane amount

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Jordan Lund (:jlund)]

at the very least, we should revert https://bugzilla.mozilla.org/attachment.cgi?id=8443714&action=edit

but we should experiment with lowering this number to something even less than 2h

Comment 1

[Jordan Lund (:jlund)]

looking at a nightly build (clobbered and built from scratch), it appears like we have very few gaps where there is no output.

let's try something fun first: 30min timeout!

I'm not sure how seriously we should consider such a low timeout. e.g. if the internets is slow and our slave's shared repos are empty, checking out src could take that time I'd imagine. However, this try push should at least show what our build + automation targets can do with less than 30min of no output.

note, I changed it at the mozharness level so I can easily test this on try. we should fix this bug at overall buildbot level once we figure out the number we want.

try push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=d99508892fa2

Comment 2

[Jordan Lund (:jlund)]

> try push:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=d99508892fa2

looks pretty green. however taking a look at PGO builds (run outside of try), it looks like there are some larger time gaps:

17:20:42     INFO -  touch -t `date +%Y%m%d%H%M.%S -d 'now+5seconds'` pgo.relink
17:20:42     INFO -  c:/builds/moz2_slave/m-in-w32-pgo-00000000000000000/build/src/obj-firefox/_virtualenv/Scripts/python.exe c:/builds/moz2_slave/m-in-w32-pgo-00000000000000000/build/src/config/nsinstall.py -t -m 755 'jsapi-tests.exe' '../../../dist/bin'
17:20:42     INFO -  mozmake.EXE[6]: Leaving directory 'c:/builds/moz2_slave/m-in-w32-pgo-00000000000000000/build/src/obj-firefox/js/src/jsapi-tests'
17:49:59     INFO -     Creating library xul.lib and object xul.exp
17:49:59     INFO -  Generating code

18:15:46     INFO -  Finished generating code
18:15:46     INFO -  c:/builds/moz2_slave/m-in-w32-pgo-00000000000000000/build/src/obj-firefox/_virtualenv/Scripts/python.exe c:/builds/moz2_slave/m-in-w32-pgo-00000000000000000/build/src/config/nsinstall.py -t -m 755 'jsapi-tests.exe' '../../../dist/bin'
18:15:46     INFO -  mozmake.EXE[6]: Leaving directory 'c:/builds/moz2_slave/m-in-w32-pgo-00000000000000000/build/src/obj-firefox/js/src/jsapi-tests'
18:41:40     INFO -  Still linking, 30 minutes passed...
19:11:42     INFO -  Still linking, 30 minutes passed...
19:18:23     INFO -  PGOMGR : warning PG0188: No .PGC files matching 'xul!*.pgc' were found.
19:18:23     INFO -    Reading PGD file 1: xul.pgd
19:18:23     INFO -     Creating library xul.lib and object xul.exp
19:18:23     INFO -  Generating code


so while we link, we do have logging output every 30min though I suspect I'll need to add a buffer to my 30min max timeout limit :)

I think we should set the default to 40m (which is a lot better than 3 hours!). From there we should configure the timeout on a platform variant basis.

So, rather than setting this at the buildbot level, we should set the timeout like I did in comment 0 at mh step. This way, if there is regressions or certain variants need longer, we can quickly fix those instances.

patch incoming

Comment 3

[Jordan Lund (:jlund)]

Attachment: 150316_bug_1141941_build_timeout-mh.patch

what could go wrong? :)

seriously though, if we do have fallout, we should configure a max_build_output_timeout at a per platform variant level rather than adjusting the `60 * 40` default.

Comment 4

[Massimo Gervasini [:massimo]]

Comment on attachment 8578262 [details] [diff] [review]
150316_bug_1141941_build_timeout-mh.patch

Ship it!

Comment 5

[Jordan Lund (:jlund)]

Comment on attachment 8578262 [details] [diff] [review]
150316_bug_1141941_build_timeout-mh.patch

thanks!

default: https://hg.mozilla.org/build/mozharness/rev/6a2f35325f28

Comment 6

[Jordan Lund (:jlund)]

mozharness production tag moved to: https://hg.mozilla.org/build/mozharness/rev/production

Comment 7

[Phil Ringnalda (:philor)]

And the platform in per platform would, unsurprisingly, be Windows. Not only bug 1145507, but RyanVM disabled b-2008-ix-0149 for hitting it twice.

Of course, it's also possible it will just smoke out a relatively small number of broken Windows build slaves, wouldn't be the first time we've had that.

Comment 8

[Jordan Lund (:jlund)]

over irc we agreed to let 1145507 do its thing to catch the frequency the timeout is hit on a per platform basis. If we hit it alot by tomorrow, we will intervene and bump up the platforms that timeout the most

Comment 9

[Jordan Lund (:jlund)]

Attachment: 150326_bug_1141941_build_timeout-windows_bump-mh.patch

based on: https://bugzilla.mozilla.org/show_bug.cgi?id=1145507#c38

Comment 10

[Jordan Lund (:jlund)]

Comment on attachment 8582878 [details] [diff] [review]
150326_bug_1141941_build_timeout-windows_bump-mh.patch

thanks!

on default: remote:   https://hg.mozilla.org/build/mozharness/rev/28e8e8169134

Comment 11

[[e-mail] (:bhearsum)]

this looks fixed