801461 - Set a timeout on the http request when importing build data

Status: RESOLVED FIXED

(Tree Management Graveyard :: TBPL, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

During the Friday network flakiness in SCL3 the cron job calling import-buildbot-data.py got hung in a state for several hours. After the network recovered the job was still hung and IT's oncall was needed to kill the process off to get the build information flowing in to tbpl's db again.

Adding a timeout to the request for builds-4hrs.js.gz would be one way to work around this. If it's running on Python 2.6 then urllib2.urlopen can be passed a timeout argument at 
http://hg.mozilla.org/users/mstange_themasta.com/tinderboxpushlog/file/default/dataimport/import-buildbot-data.py#l155
I don't know how long the transfer takes but it might be worth setting a timeout like 60 seconds in case it's slow in some circumstances we don't want to give up too soon.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

We hit this again today. http://heyman.info/2010/apr/22/python-urllib2-timeout-issue/ may be useful.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

And again today - bug 821232. What is a sensible timeout value ?

Comment 3

[Ed Morley [:emorley]]

(In reply to Nick Thomas [:nthomas] from comment #2)
> And again today - bug 821232. What is a sensible timeout value ?

The file at the moment is 1.4MB, which takes me 2-3s locally to download.

30-60s should allow plenty of wiggle room.

Comment 4

[Ed Morley [:emorley]]

Oops and now bug 827443.

I have a WIP in my mq from before Christmas - will try and find some time for it this week.

Comment 5

[Ed Morley [:emorley]]

Attachment: Patch v1

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 701793 [details] [diff] [review]
Patch v1

Seems fine to me.

Comment 7

[Ed Morley [:emorley]]

https://hg.mozilla.org/webtools/tbpl/rev/1c6ee7b76fed

Comment 8

[Ed Morley [:emorley]]

In production.