704200 - android dep builds permared after bug 701864; sometimes causing nightlies not to trigger

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

Because we require a working set of dep builds to trigger nightlies, these burning Android dep builds can cause us not to. In some cases, we end up triggering nightlies solely based on l10n changes, but this isn't always the case, such as bug 704122. https://bugzilla.mozilla.org/show_bug.cgi?id=704122#c7 has a better description of the algorithm used, but we should really fix this. Some options:
1) turn off the permared builds
2) disable android nightlies completely
3) stop looking at android dep build status when decided whether to trigger nightlies or not

Comment 1

[Aki Sasaki (not active)]

I intentionally turned these on because we don't know mobile's timing and I didn't want to have us be a blocker to that. This bug changes that story quite a bit.

* If we turn off the 'android' platform (dep or nightly) we need to keep it on birch.

* If we only turn off 'android' nightlies, will the other nightlies trigger? This would be the best solution.
** set android enable_nightly to False
** set android enable_nightly for Birch to True

* Not sure how ugly (3) will be.

* Any way we resolve this, we should leave 'linux-android' and 'android-xul' alone.

Comment 2

[[e-mail] (:bhearsum)]

(In reply to Aki Sasaki [:aki] (back nov 28) from comment #1)
> I intentionally turned these on because we don't know mobile's timing and I
> didn't want to have us be a blocker to that. This bug changes that story
> quite a bit.
> 
> * If we turn off the 'android' platform (dep or nightly) we need to keep it
> on birch.

Yup. Can we drop android from the default list of platforms here: http://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/config.py#l70. It's already in birch's list (http://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/project_branches.py#l191), so we shouldn't have to do anything else except back out this change when we're ready.

> * If we only turn off 'android' nightlies, will the other nightlies trigger?

I don't think so. My reading of the code says that we look at *all* of the dep builders, not just the ones with nightlies enabled.

> This would be the best solution.
> ** set android enable_nightly to False
> ** set android enable_nightly for Birch to True

Therefore, this won't help.

> * Any way we resolve this, we should leave 'linux-android' and 'android-xul'
> alone.

Makes sense to me.

Comment 3

[Phil Ringnalda (:philor)]

Much as I don't like this solution, if mobile is really going to land something to stop the permared tomorrow, then we only have one more nightly that can fail to trigger, and it will only fail to trigger if there are no l10n changes all day today. There's still the risk (which we've been living with since last week) that we'll build a nightly on a busted rev, since using the l10n fallback puts us back in the old-skool "whatever's tip at 3am ships if it happens to compile" style, but maybe living with that risk is less painful than fixing this so that a single nightly can get a properly chosen rev.

Comment 4

[[e-mail] (:bhearsum)]

That's a good point. I'm going to get a patch ready, but I'll hold off landing anything until we have a better idea when these builds are going to stop burning.

Comment 5

[Mark Finkle (:mfinkle) (use needinfo?)]

Mobile is working hard to get native builds working on m-c, but we won't be ready in the next week or so. Let's disable the native builds on m-c until mobile gives releng a thumbs up.

Comment 6

[[e-mail] (:bhearsum)]

Attachment: disable android builder

Not sure if Aki is going to be around to review this or not, but this should get us going again. I changed the default for the "android" platform to disabled, which turns it off for everything except birch, which has it explicitly enabled: http://hg.mozilla.org/build/buildbot-configs/file/tip/mozilla/project_branches.py#l191.

My before and after diff of config.py output looked sane, I'll attach it here as well.

Comment 7

[[e-mail] (:bhearsum)]

Attachment: diff of config.py output before/after patch

Comment 8

[[e-mail] (:bhearsum)]

This patch is landed in production. The permared builders are now off everywhere, so nightlies should be back to normal.