969542 - Make B2G device builds periodic

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris AtLee [:catlee]]

We spend a lot of time building b2g device builds per push just to make sure they compile. Let's try running them periodically instead. This should save a bunch of time on the infra.

Emulator builds should still be done per push. This should catch most bustage from JB/ICS changes.

Comment 1

[Chris AtLee [:catlee]]

Attachment: enable periodic device builds on b2g-inbound, m-c

Comment 2

[Chris AtLee [:catlee]]

Attachment: periodic build support

Comment 3

[Ryan VanderMeulen [:RyanVM]]

Can we please run these on mozilla-inbound and fx-team now too? We're still going to be way under the load we used to be, and bug 966257 is a real example of bustage that hit on inbound first.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #3)
> Can we please run these on mozilla-inbound and fx-team now too? We're still
> going to be way under the load we used to be, and bug 966257 is a real
> example of bustage that hit on inbound first.

By "these" I mean all device image builds, FWIW (would also apply to m-c). Also, I don't see any changes to b2g-inbound in the first patch. Is that right?

Comment 5

[Aki Sasaki (not active)]

This bug may conflict with Clint's ask in bug 969767.

Comment 6

[Ryan VanderMeulen [:RyanVM]]

Maybe we could just store the nightlies that will still run like normal?

Comment 7

[Aki Sasaki (not active)]

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #6)
> Maybe we could just store the nightlies that will still run like normal?

Who is this in response to?
It won't solve bug 969767 if that's what you're asking.  That bug specifically wants all per-checkin builds saved for 12 weeks for bisecting purposes; we won't get per-checkin builds if we shift to periodic.

Comment 8

[Chris AtLee [:catlee]]

12 weeks of periodic builds may be enough?

Comment 9

[Chris AtLee [:catlee]]

(In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #4)
> (In reply to Ryan VanderMeulen [:RyanVM UTC-5] from comment #3)
> > Can we please run these on mozilla-inbound and fx-team now too? We're still
> > going to be way under the load we used to be, and bug 966257 is a real
> > example of bustage that hit on inbound first.

Yeah, we can do that. I want to make sure this works properly first!

> By "these" I mean all device image builds, FWIW (would also apply to m-c).
> Also, I don't see any changes to b2g-inbound in the first patch. Is that
> right?

This changes the default behaviour to be periodic builds instead of per-push. The rest of the patch is undoing that default for all the !trunk branches.

Comment 10

[cmtalbert]

(In reply to Aki Sasaki [:aki] from comment #5)
> This bug may conflict with Clint's ask in bug 969767.

Yes it does. Specifically, we need per-push builds on b2g-inbound so that we can use tinderbox-builds to do quicker manual bisections without building b2g over and over again on our laptops. If we could work out an API where a tester could request specific device builds from old pushes then we could solve both problems - we could run our builds periodically and still support our bisection needs (that old bisect-in-the-cloud idea of mine ;-) ). 

But for the moment, the only option we have is brute force, unfortunately. So, one consideration I could make is that we only need the builds for the devices that we are testing on for a particular release. Right now, that's the hamachi builds. Maybe we can make a compromise on that so that our testing team is supported with per push builds of the primary device they are using for a given release. And we can set that on a per release basis.

I still think the ideal solution here is a bisect in the cloud solution where my local bisection script uses APIs to talk to the automation and then we only do these builds on an "on demand or periodic" basis.

Comment 11

[Aki Sasaki (not active)]

(In reply to Clint Talbert ( :ctalbert ) from comment #10)
> (In reply to Aki Sasaki [:aki] from comment #5)
> > This bug may conflict with Clint's ask in bug 969767.
> 
> Yes it does. Specifically, we need per-push builds on b2g-inbound so that we
> can use tinderbox-builds to do quicker manual bisections without building
> b2g over and over again on our laptops. If we could work out an API where a
> tester could request specific device builds from old pushes then we could
> solve both problems - we could run our builds periodically and still support
> our bisection needs (that old bisect-in-the-cloud idea of mine ;-) ). 
> 
> But for the moment, the only option we have is brute force, unfortunately.
> So, one consideration I could make is that we only need the builds for the
> devices that we are testing on for a particular release. Right now, that's
> the hamachi builds. Maybe we can make a compromise on that so that our
> testing team is supported with per push builds of the primary device they
> are using for a given release. And we can set that on a per release basis.

Ok. This means the currently attached patches need changing to allow for per-checkin builds for at least hamachi{,-eng}? on b2g-inbound, and then we need to allow for pvtbuilds uploads of hamachi.

> I still think the ideal solution here is a bisect in the cloud solution
> where my local bisection script uses APIs to talk to the automation and then
> we only do these builds on an "on demand or periodic" basis.

Yeah.  I think that requires in-tree manifests, which we only recently have, otherwise we're building against tip of gaia/gonk even when bisecting.  After that we *may* have enough hooks in buildbot land to be able to do this without massive changes ?  (Not 100% sure)

Comment 12

[Chris AtLee [:catlee]]

Attachment: enable periodic device builds on b2g-inbound, m-c

same as before, except leaves hamachi per-push

Comment 13

[Armen [:armenzg]]

Live in production (not getting CCed).