1042553 - Provide Builds for every Gecko commit for b2ghaystack

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: Performance, defect, P1)

Description

[Mason Chang [Inactive] [:mchang]]

Modify b2ghaystack to get builds for every gecko commit. This means either (1) find out if we are building gecko builds for every gecko commit, or (2) build our own single machine that given two regressions and the mozilla-central source, builds the necessary files to bisect.

Comment 1

[Mason Chang [Inactive] [:mchang]]

Hi Dave, think you could find out if we already have builds somewhere? Otherwise, I can start ordering a machine to do this for us. Thanks!

Comment 2

[Dave Huseby [:huseby]]

I just asked davehunt on IRC.

Comment 3

[Dave Huseby [:huseby]]

We're going to need to put together our own build cluster.  As far as Dave Hunt knows, we only have gecko builds per b2g-inbound commit.  Typically, those contain 60-150 gecko commits.  So we'll need enough machines to make 3-7 gecko builds in parallel.  Since there is resistance to having more flame devices in the rack, I think we should try to get by with only 3 devices to do the bisection, which means we only need enough build machines to make 3 gecko builds in parallel to feed the bisection.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

how long does a test take on a flame?  

I assume it is less time than a build takes, so you would need more build machines than flames.  If you only do a build once you have results from your bisection, then it will take longer, but you won't need as many build machines :)

Comment 5

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Dave Huseby [:huseby] from comment #3)
> We're going to need to put together our own build cluster.  As far as Dave
> Hunt knows, we only have gecko builds per b2g-inbound commit.  Typically,
> those contain 60-150 gecko commits.  So we'll need enough machines to make
> 3-7 gecko builds in parallel.  Since there is resistance to having more
> flame devices in the rack, I think we should try to get by with only 3
> devices to do the bisection, which means we only need enough build machines
> to make 3 gecko builds in parallel to feed the bisection.

I'm curious to know where the figure of 60-150 gecko commits comes from? Looking at the b2g-inbound pushlog (http://hg.mozilla.org/integration/b2g-inbound/pushloghtml) I see most have considerably less commits than that.

(In reply to Joel Maher (:jmaher) from comment #4)
> how long does a test take on a flame?

I think the bisects are likely to be on individual apps that have shown regressions. For b2gperf a single app would take ~10 minutes to test.

Comment 6

[Mason Chang [Inactive] [:mchang]]

So I think we have two routes here:

1) Build locally and enable b2ghaystack, given a mozilla-central directory and gecko commit, to just build and flash locally.
2) Create a custom job on Jenkins and fetch the build from jenkins when it's done. However, jgriffin said this might be an issue since we have to wait for Jenkin's to queue and run the job. If we could prioritize our custom job so that we could get a build back within an hour, that might be doable. Even then though, on my macbook pro, I'm able to get a full clobber build in ~10 minutes with ccache, so I'm leaning more towards (1).

Comment 7

[Jonathan Griffin (:jgriffin)]

Just to clarify, in 2) we'd be requesting missing builds from buildbot, not Jenkins, but b2ghaystack would still have to wait for them.

A full clobber device build in 10 minutes is impressive!

Comment 8

[Dave Huseby [:huseby]]

I can confirm.  With my distcc+ccache build cluster, I was getting 10-15 minute full clobber builds.  http://bit.ly/1fI3kd8

I'm interested mostly in whatever is the easiest way.

Dave,

what do you think would be easier to make b2ghaystack do?  1) drive local builds given a repo and revision and run b2gperf directly, or 2) request missing builds from buildbot?

Comment 9

[Dave Hunt [:davehunt] [he/him] ⌚BST]

This is really outside of my expertise. When b2ghaystack was written it was intended for bisecting within available build rather than triggering additional builds. Given the suggestions in comment 8 I would probably lean towards requesting builds from buildbot so that we have consistency in the build environments, but like I say, I'm out of my depth here.

Comment 10

[Mason Chang [Inactive] [:mchang]]

Do we have an easy ability / API to request builds from buildbot and get a notification programmatically? Or would we have to do that ourselves as well.

Comment 11

[Jonathan Griffin (:jgriffin)]

There is an API for requesting the builds.  There isn't an API per se for getting a notification, but we could rely on pulse or builds-4hr.js (the same thing that Treeherder uses).

I'm still not sure how much of a real problem this is.  The number of skipped builds is normally small, and so the effort of building out this capability may exceed its actual usefulness.

I'd rather get b2ghaystack using existing builds, see how big of a problem this is in real practice, and then tackle this if seems to warrant the effort.

Comment 12

[Jonathan Griffin (:jgriffin)]

Let's discuss this at next Friday's B2G Perf sync up and come to some conclusion there.

Comment 13

[Mason Chang [Inactive] [:mchang]]

From the b2g automation + fxos perf team meeting, we decided that due to the difficulty in getting buildbot and b2ghaystack to work together, and that we don't know that this is a real problem, we should do two things. First, we will log when a gecko commit causes a regression and how much bisection duty it required so that we have metrics to see how big of a problem this is. Second, if it is a small problem, and b2g-inbound only misses 3-4 gecko commits at a time, we can just bisect locally as it should be fast at that point. We can further automate this process by modifying b2g haystack to use a local build instead of requesting a build.

Comment 14

[BMO Automation]

Firefox OS is not being worked on