534395 - Lengthen buildsymbols and client.py checkout timeouts

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Robert Kaiser]

The SeaMonkey buildbots did run into some problems when we turned on mozilla-central trees as for some reason, the buildsymbols step takes double the time on trunk than on 1.9.1 branch, and so we did run into the default 20min timeout of buildbots steps at least for Windows.
Bumping the timeout for that step to an hour fixes that (though an even better solution would be for the symbol script to send progress output to stdout/stderr, so that buildbot doesn't think it's dead).

Also, when client.py checkout needs to clone a complete mozilla-central tree (e.g. after a clobber), that can take some time and also made us run into the timeout on our sometimes a bit slow boxes. Therefore, I upped the timeout on that step to an hour as well.

I think it's finally time to get that patch upstream, we've been free of such timeout problems since I applied it locally on the SeaMonkey buildmaster.

(Marking blocking bug 532676 even though I'll fix that one with the patch applied locally again.)

Comment 1

[Robert Kaiser]

Attachment: make all those timeouts be 1h

I already explained the reason for this patch in comment #0, so not more to say here ;-)

Asking Ben for review because of the common buildsysmbols step, and gozer because of the CC*-specific client.py checkout steps.

Comment 2

[[e-mail] (:bhearsum)]

Comment on attachment 417254 [details] [diff] [review]
make all those timeouts be 1h

>diff --git a/process/factory.py b/process/factory.py
>--- a/process/factory.py
>+++ b/process/factory.py
>@@ -984,17 +984,18 @@ class MercurialBuildFactory(MozillaBuild
>         )
> 
>     def addUploadSymbolsStep(self):
>         self.addStep(ShellCommand,
>          name='make_uploadsymbols',
>          command=['make', 'uploadsymbols'],
>          env=self.env,
>          workdir='build/%s' % self.objdir,
>-         haltOnFailure=True
>+         haltOnFailure=True,
>+         timeout=60*60 # try an hour for the moment
>         )

I think this is the wrong step to lengthen the timeout on...

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

There's output there, it just doesn't get handled properly on windows. I'm curious if "python -u" (or setting PYTHONUNBUFFERED=1) fixes the problem.

Comment 4

[Robert Kaiser]

(In reply to comment #3)
> There's output there, it just doesn't get handled properly on windows. I'm
> curious if "python -u" (or setting PYTHONUNBUFFERED=1) fixes the problem.

From an IRC conversation I had with ted one time, actually, stdout in the buildsymbols step is used by the scripts internally somehow and redirected. If buildbot counts stderr output against breaking the timeout as well, we could go with adding something there, probably.

In any case, we should push that to a different bug - even more as I just discovered that we already have an hour of timeout on that step, even without my patch ;-)

Comment 5

[Robert Kaiser]

Attachment: v2 - leave out the buildsymbols timeout, it's already 1h

I probably got confused when porting my patch, as buildsymbols already is set to 1h of timeout - the one on uploadsymbols is unneeded, of course ;-)

Comment 6

[Robert Kaiser]

Ben, can you land this together with bug 534456 tomorrow?

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 418003 [details] [diff] [review]
v2 - leave out the buildsymbols timeout, it's already 1h

changeset:   553:3f6a6acc21a0

Comment 8

[Robert Kaiser]

Thanks, Ben.
It sticked, SeaMonkey is running on the unpatched tip of buildbotcustom, so this is fixed.