453457 - HgPoller tends to get stuck (never finish the current poll) when hgweb is down

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Benjamin Smedberg]

Last night I pushed a change to hg.mozilla.org/try that never got built:

http://hg.mozilla.org/try/pushloghtml?tipsonly=1

And in fact jimb appears to have done a push also since then which wasn't built either.

Comment 1

[[e-mail] (:bhearsum)]

There was some downtime on hg.m.o last night, I think. I'll have a look through the logs.

Comment 2

[[e-mail] (:bhearsum)]

Looks like HgPoller got hung up. I'll restart the master and it should pick up your builds. Getting a lot of ' Not polling <HgPoller for http://hg.mozilla.orgtry> because last poll is still working' in the log - do we need to pass the getPage calls a timeout or something?

Comment 3

[[e-mail] (:bhearsum)]

Ah, unfortunately you're going to have to push again.

Comment 4

[[e-mail] (:bhearsum)]

Here's where we got hung up:
2008-08-31 20:02:51-0700 [-] Polling Hg server at http://hg.mozilla.org/try/index.cgi/pushlog?tipsonly=1

This poll never finished, and never timed out.

Aravind, do you know if hg.m.o went down around that time?

Comment 5

[[e-mail] (:bhearsum)]

Fixing up the summary + severity. Whenever this happens it forces us to restart the affected masters.

Comment 6

[Benjamin Smedberg]

Patch in bug 452102

Comment 7

[Benjamin Smedberg]

Fix landed in CVS. Pushing back to bhearsum to update and restart the tryserver master.

Comment 8

[[e-mail] (:bhearsum)]

Alright, I've updated hgpoller on production-master, staging-master, qm-moz2-unittest01, and sm-try-master and reconfig'ed the Buildbot masters on them. We shouldn't have any more problems with HgPoller dying.

Thanks for fixing that Benjamin.