774862 - should fix our polling code to handle the case where hg.mozilla.org resets

Status: RESOLVED DUPLICATE of bug 1289514

(Release Engineering :: General, defect, P3)

Description

[Kim Moir [:kmoir] ET]

For example, in bug 774799

/builds/buildbot/build_scheduler/master/twistd.log.5:2012-07-17 11:37:20-0700 [HTTPPageGetter,client] http://hg.mozilla.org/try has been reset 

occurred and this caused duplicate try builds which is a waste of our machine capacity.

Comment 1

[[e-mail] (:bhearsum)]

Not a release-specific problem, moving to Automation (General).

Comment 2

[Chris AtLee [:catlee]]

Becoming more urgent since try seems to respond with 500 errors pretty regularly these days.

Comment 3

[Chris AtLee [:catlee]]

See also bug 770811

Comment 4

[Chris AtLee [:catlee]]

Our polling code is checking specifically for "unknown revision" in the error message here, so it's not just any old ISE that will trigger this.

Comment 5

[Ed Morley [:emorley]]

This seems to be happening on non-Try too:
https://tbpl.mozilla.org/?tree=Mozilla-Inbound&rev=3b72e384d1bf

Comment 6

[Chris Cooper [:coop] (he/him)]

Adding the [buildduty] flag since buildduty is going to care a lot about the doubling of build load due to this.

Comment 7

[Chris AtLee [:catlee]]

Attachment: disable resetting

let's disable the resetting code for now. repo resets are pretty infrequent, so we can do them manually until this bug gets fixed for real.

Comment 8

[Ed Morley [:emorley]]

We had another batch of double builds on try last night, eg:
https://tbpl.mozilla.org/?tree=Try&rev=eaf06d352463

Was this fix in place at that point? Looking at the timeframes I would have thought so, but I may be messing up the timezones or something :-(

Comment 9

[Chris AtLee [:catlee]]

This was checked in, but not yet deployed. I'll do that this morning.

Comment 10

[Rail Aliiev [:rail]]

In production

Comment 11

[Chris AtLee [:catlee]]

If we care to re-visit this, we need to re-enable the commented-out code and make it more resilient to intermittent 500 errors from hg.mozilla.org.

We're in good shape for now. The next time a project branch or try gets reset, we'll need to reset the pollers manually by restarting the build scheduler master.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Bug 833555 is an example of where it would be good to have this code back, since none of us remembered to restart the build scheduler for bug 816300.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Updated the note at https://wiki.mozilla.org/ReleaseEngineering/DisposableProjectBranches#Book_one_of_our_fabulous_.22disposable.22_project_branches to help when builds don't start (when the poller is still looking for changes since an old revision).

Comment 15

[Panos Astithas (he/him) [:past] (please ni?)]

Please reconfigure the build scheduler. I've made the first push to the Gum twig after a reset and I don't see any builds starting.

Comment 16

[Panos Astithas (he/him) [:past] (please ni?)]

Please ignore comment 15, the second push started building just fine.

Comment 17

[Ed Morley [:emorley]]

This is doable now, with bug 1114843 / bug 1065771 - we don't need to rely on the HTTP response code.
This also overlaps with bug 1104374, since the ideal solution here likely relies on switching to using push IDs not changesets.

See also:
https://mozilla-version-control-tools.readthedocs.org/en/latest/hgmo/pushlog.html#writing-agents-that-consume-pushlog-data

Comment 18

[Gregory Szorc [:gps]]

I'm fixing this as part of scope bloat in bug 1289514.