472333 - DependentL10n scheduler is lost after a master reconfig

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Chris Cooper [:coop] (he/him)]

Whenever the buildbot master is reconfig-ed, the DependentL10n scheduler decouples from the nightly scheduler and we stop getting l10n builds. 

We need to figure out a more robust way to start this scheduler.

Comment 1

[Axel Hecht [:Pike]]

Can you replicate this with a dummy setup? I.e., is this a buildbot bug?

Comment 2

[[e-mail] (:bhearsum)]

Yeah, it's sortof a Buildbot bug. Because of the way our configuration is organized Buildbot recreates all of the Schedulers on a reconfig. This causes the Dependent Scheduler to lose its upstream IIRC.

Comment 3

[Chris Cooper [:coop] (he/him)]

FWIW, Axel filed a similar buildbot ticket upstream a while ago:

http://buildbot.net/trac/ticket/35

Given the longevity of the ticket, I don't expect it to be fixed upstream any time soon.

Comment 4

[Axel Hecht [:Pike]]

According to that bug (oops, who filed that, me?), we can be hit by this for triggerable schedulers, too.

I'd love to see a test case, if this is biting strongly enough, I'd ponder tackling it upstream.

Comment 5

[Chris Cooper [:coop] (he/him)]

To get us back to the state where we can at least reconfig the master for other changes, we're going to switch from a dependent scheduler to a nightly scheduler for l10n. We'll give it an offset time from the en-US nightly so that we'll have a good chance of being as up-to-date as possible.

Not ideal, but better than the current situation I think.

Comment 6

[Chris Cooper [:coop] (he/him)]

Switching to a nightly scheduler seems to be working fine in staging. 

I'll have a patch ready shortly. I'm also going to schedule two rounds of nightlies/day for l10n to give us some better coverage until we get build-on-change.

Comment 7

[Chris Cooper [:coop] (he/him)]

Attachment: Switch from dependent to nightly scheduler for l10n

Comment 8

[Armen [:armenzg]]

Comment on attachment 356536 [details] [diff] [review]
Switch from dependent to nightly scheduler for l10n

The only comment is if you could change "tip" for "default"

Comment 9

[Chris Cooper [:coop] (he/him)]

I've scheduled downtime for tomorrow morning.

Comment 10

[Chris Cooper [:coop] (he/him)]

Now working in production.