Closed Bug 943485 Opened 11 years ago Closed 9 years ago

mozregression should support b2g

Categories

(Testing :: mozregression, defect)

Product:
Testing
Component:
mozregression
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: BenWa, Unassigned)

References

Details

+++ This bug was initially created as a clone of Bug #935183 +++

Now that mozregression can support inbound builds we should move to support b2g regression. Currently the tools for b2g regressions are lagging very far behind and prevent us from doing a swift backout (or atleast identify the regression cause) when things go wrong. This should improve our development velocity.

Assumption 1: From what I understand conceptually b2g builds can be broken down into 'gaia', 'gecko', '3rd party libraries' parts. In general these are independent except when they are not. I've seen a few rare issues where two of these components needed to be updated at the same time (Like bug 913593). I don't have a great solution for that problem but maybe for now we can have mozregression ignore this and still find most regression. We ca operate on either 'gaia' or 'gecko' regressions.

Assumption 2: The 'gaia' and 'b2g' build artifacts are phone neutral.

If the above assumption holds then we should be able to retain the 'gaia' and 'gecko' build artifacts in S3 and extend mozregression to flash them to the phone.

Problem is I only see us retaining desktop builds of b2g.

Can someone weight in here?
Flags: needinfo?(gal)
We don't maintain separate gaia and b2g build artifacts; instead, we only create build artifacts for particular environments.  That is, we retain build artifacts for e.g., hamachi devices, and separate build artifacts for b2g desktop builds.  Both types of artifacts contain gecko+gaia (+gonk in the case of device builds).

We can bisect over gaia independently of this, by cloning the gaia repo and generating profiles at desired commits.  A gaia "build" doesn't involve compilation; it's just creating and optimizing a gecko profile, and the process only takes a few seconds.

Gecko is harder, since we don't create or retain phone-neutral gecko build artifacts, although we _could_ do this.

"Supporting b2g" really means different things depending on what type of build(s) we're supporting...we could add support for b2g desktop builds in a pretty straightforward fashion.   Adding device support is harder because of the above, and because we can't retain device build artifacts in S3, due to the inclusion of proprietary libs.

We could either add the ability to perform device-specific bisection, using the limited number of builds that we save internally in an LDAP-protected location, or we could ask rel-eng to start producing and retaining phone-neutral gecko builds just for this purpose.  The latter would come with some cost (since we wouldn't be using those builds for anything other than this tool), and it isn't clear to me whether the ROI would warrant it.
I'm not interested in performing regressions on proprietary libraries which makes this easier to solve.

(In reply to Jonathan Griffin (:jgriffin) from comment #1)
> A gaia "build" doesn't involve
> compilation; it's just creating and optimizing a gecko profile, and the
> process only takes a few seconds.

Sounds like this bug should be morph to 'gecko b2g' then. Those are the regressions I'm most interested in anyways.

> The latter would come
> with some cost (since we wouldn't be using those builds for anything other
> than this tool), and it isn't clear to me whether the ROI would warrant it.

We are already producing them however? Then we need to spend a bit more machine time uploading the builds to the ftp/S3 and pay the storage cost. I know that once this is employed we can deal with regressions more effectively. Gecko b2g regressions are currently painful, at least to me. I'd like to know how other b2g developers/QA feel on this matter?

For cost retaining desktop builds was in the order of $20,000/year for 5 years of changeset builds. I'd imagine this cost to be lower on b2g since gecko is device neutral (only 1 platform):
https://bugzilla.mozilla.org/show_bug.cgi?id=765258#c0
(In reply to comment #2)
> For cost retaining desktop builds was in the order of $20,000/year for 5 years
> of changeset builds. I'd imagine this cost to be lower on b2g since gecko is
> device neutral (only 1 platform):
> https://bugzilla.mozilla.org/show_bug.cgi?id=765258#c0

Hmm, is that really true with things like MOZ_B2G_RIL?
(In reply to :Ehsan Akhgari (needinfo? me!) from comment #3)
> (In reply to comment #2)
> > For cost retaining desktop builds was in the order of $20,000/year for 5 years
> > of changeset builds. I'd imagine this cost to be lower on b2g since gecko is
> > device neutral (only 1 platform):
> > https://bugzilla.mozilla.org/show_bug.cgi?id=765258#c0
> 
> Hmm, is that really true with things like MOZ_B2G_RIL?

Possibly not; I'm not sure how many build flags affect B2G gecko builds, or if there's such a thing as a standard gecko build that would work on all devices.
That's really unfortunate then =\.
We should continue to have one uniform gecko for any gonk version.
Flags: needinfo?(gal)
The (new to us) info that landing a patch on b2g-inbound, rather than mozilla-inbound, means that we keep device builds off that patch, is just amazing and probably lowers a bit the urgency (though not importance) of this bug.
Moving this to the mozregression module.

Things have probably changed since this bug was filed (I know we have fairly frequently updated full device tinderbox builds for the Tarako at least now, probably soon to include the Flame), we should probably revisit fixing this bug keeping that in mind.
Component: General → mozregression
Note that we now have b2ghaystack for tracking down perf regressions, sort of related. See bug 943611 and http://blargon7.com/2014/03/hunting-for-performance-regressions-in-firefox-os/
Blocks: 1018101
I'd like to take this bug, but I'm not sure I understand very well.

I see this as a refactoring of some parts of mozregression, mainly the function inboundfinder.getInboundRevisions [1], which is very similar to what b2ghaystack does in [2]. There's also the function utils.urlLinks [3] that can be updated to match the one used in b2ghaystack [4].

Tell me if this is enough for this bug. This way we can then work on bug 1018101.

Before that we may wait for bug 1018096 as it could involve a lot of changes to mozregression codebase.

[1] https://github.com/mozilla/mozregression/blob/master/mozregression/inboundfinder.py#L31

[2] https://github.com/mozilla/b2ghaystack/blob/master/b2ghaystack/b2ghaystack.py#L45

[3] https://github.com/mozilla/mozregression/blob/master/mozregression/utils.py#L66

[4] https://github.com/mozilla/b2ghaystack/blob/master/b2ghaystack/b2ghaystack.py#L34
Flags: needinfo?(wlachance)
It would probably make sense to start with making mozregression work with B2G Desktop builds, as it should be is easier to get working and doesn't require a device. Once we work out the kinks there, we can look into getting on device support working. 

I believe B2G Desktop is about to be replaced with something called "Mulet" (https://groups.google.com/d/msg/mozilla.dev.b2g/2WUg10993OM/VSceN9i2w2EJ). I am not sure about the details of how far this is along, :jgriffin can you say more, especially in the context of mozregression? Are we anywhere close to producing inbound/nightly builds for Mulet?

Regardless, mozregression is already pretty extensible as far as add support for new types of applications is -- I would start by looking in: 

https://github.com/mozilla/mozregression/blob/master/mozregression/runnightly.py
https://github.com/mozilla/mozregression/blob/master/mozregression/runinbound.py

Once we have more information on what type of build we want to work against, I can give more information on which repositories to look for builds in, etc.

I agree that we should commit bug 1018096 first.
Flags: needinfo?(wlachance) → needinfo?(jgriffin)
We're probably a week or two away from Mulet builds in inbound.  I think you could start by getting this to work with b2gdesktop builds; switching that to Mulet is probably only a matter of replacing a few constants.
Flags: needinfo?(jgriffin)
William, can you give me more info for b2g desktop builds integration, as you proposed in comment #12 ?

I suppose I will have to add B2gNightly and B2GInbound classes - And probably tweak the get_inbound_revisions method [1] to support finding valid b2g builds (maybe I can use the logic used in b2ghaystack for that [2]).

Also, If you have any information to help me testing this, I sure will appreciate ! Dave already helped me to test b2ghaystack by giving me some informations in [3], and maybe I can reuse it for this bug.

[1] https://github.com/mozilla/mozregression/blob/master/mozregression/inboundfinder.py#L37

[2] https://github.com/mozilla/b2ghaystack/blob/master/b2ghaystack/b2ghaystack.py#L45

[3] https://bugzilla.mozilla.org/show_bug.cgi?id=1003793#c3
Flags: needinfo?(wlachance)
(In reply to j.parkouss from comment #14)
> William, can you give me more info for b2g desktop builds integration, as
> you proposed in comment #12 ?
> 
> I suppose I will have to add B2gNightly and B2GInbound classes - And
> probably tweak the get_inbound_revisions method [1] to support finding valid
> b2g builds (maybe I can use the logic used in b2ghaystack for that [2]).

Yes, exactly. We can do this in two phases if you like: nightlies, then inbound. 

So currently b2gdesktop builds are available at:

http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly

(grab the ones marked mozilla-central)

Inbound builds are at:

http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/

I suspect the hard(er) part will be actually getting the b2g builds to start with the correct profile. I recommend that you use mozrunner as we do with regular desktop builds. We have some example code of this in marionette, unfortunately it's not completely straightforward but hopefully it gives a place to start:

http://hg.mozilla.org/mozilla-central/file/78245b8d422d/testing/marionette/client/marionette/marionette.py#l473
http://hg.mozilla.org/mozilla-central/file/78245b8d422d/testing/marionette/client/marionette/geckoinstance.py

> Also, If you have any information to help me testing this, I sure will
> appreciate ! Dave already helped me to test b2ghaystack by giving me some
> informations in [3], and maybe I can reuse it for this bug.

Yes, well, the good thing here compared to b2ghaystack is that we can use publicly available b2gdesktop builds to test.
Flags: needinfo?(wlachance)
Ok then, I will work on this. :)
Hey William,

I started to work on nightlies - you can see it here:

https://github.com/parkouss/mozregression/commit/f74de2228478368017c7250aedaf4dc59513c11f

It seems to works pretty well and I think I'm on the right way, but I'd like you to confirm or infirm. :)

I have to admit that I never used b2g. Here is a trace of script execution:

> moznightly --app b2g -d 2014-06-04
>
> Downloading build from: http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/2014/06/2014-06-04-16-02-02-mozilla-central/fxos-simulator-2.0-linux-x86_64.xpi
> ===== Downloaded 100% =====
> Installing nightly
> Starting nightly
> console.error: fxos_2_0_simulator:
>   Error opening input stream (invalid filename?)
> console.error: fxos_2_0_simulator:
>   Error opening input stream (invalid filename?)

Am I missing something ? I can see that the addon is present in the launched browser.
Flags: needinfo?(wlachance)
(In reply to j.parkouss from comment #17)
> Hey William,
> 
> I started to work on nightlies - you can see it here:
> 
> https://github.com/parkouss/mozregression/commit/
> f74de2228478368017c7250aedaf4dc59513c11f
> 
> It seems to works pretty well and I think I'm on the right way, but I'd like
> you to confirm or infirm. :)
> 
> I have to admit that I never used b2g. Here is a trace of script execution:
> 
> > moznightly --app b2g -d 2014-06-04
> >
> > Downloading build from: http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/2014/06/2014-06-04-16-02-02-mozilla-central/fxos-simulator-2.0-linux-x86_64.xpi
> > ===== Downloaded 100% =====
> > Installing nightly
> > Starting nightly
> > console.error: fxos_2_0_simulator:
> >   Error opening input stream (invalid filename?)
> > console.error: fxos_2_0_simulator:
> >   Error opening input stream (invalid filename?)
> 
> Am I missing something ? I can see that the addon is present in the launched
> browser.

Hey Julian, glad to hear you're making progress on this, though it appears you're currently downloading the wrong file. You don't want the simulator (which is a Firefox extension), but the standalone desktop build. For example: http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/2014/06/2014-06-04-16-02-02-mozilla-central/b2g-32.0a1.multi.mac64.dmg for Mac.
I should have asked before...

Thank you Dave, that's helping ;)
Flags: needinfo?(wlachance)
This seems a lot better:

https://github.com/parkouss/mozregression/compare/mozilla:master...master

Now, I have a real standalone desktop window that is running. :)

I don't understand the profile problem discussed in comment #15. Can anyone help me on this ?
Hmm.. I'm not sure either. For automation we allow users to specify a profile path, and if they do then a copy of that profile is created and used. We also set a preference that's rather specific to our automation. I don't think either of these would be a problem for regression hunting in the desktop build though.
Flags: needinfo?(wlachance)
Yeah, I wouldn't worry about the specifics of profile handling in those examples. I just wanted to make it clear you can use mozrunner to launch instances of b2g desktop.
Flags: needinfo?(wlachance)
Oh, cool, I didn't see that you had something working! Could you rebase your changes / pull request against the current master so it's more clear what was modified? Something like "git pull --rebase https://github.com/mozilla/mozregression.git master" then follow the instructions here:

http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html

Once you've squashed everything, you can push everything back to your own fork (probably with the --force option)

(In reply to j.parkouss from comment #20)
> This seems a lot better:
> 
> https://github.com/parkouss/mozregression/compare/mozilla:master...master
> 
> Now, I have a real standalone desktop window that is running. :)
> 
> I don't understand the profile problem discussed in comment #15. Can anyone
> help me on this ?
(In reply to William Lachance (:wlach) from comment #23)
> Oh, cool, I didn't see that you had something working! Could you rebase your
> changes / pull request against the current master so it's more clear what
> was modified? Something like "git pull --rebase
> https://github.com/mozilla/mozregression.git master" then follow the
> instructions here:
> 
> http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html
> 
> Once you've squashed everything, you can push everything back to your own
> fork (probably with the --force option)

Thank you for these informations about git usage. It helped me a lot - a new pull request is done.

I will then work on inbounds builds, if everything is fine for you. :)
(In reply to j.parkouss from comment #24)
> (In reply to William Lachance (:wlach) from comment #23)
> > Oh, cool, I didn't see that you had something working! Could you rebase your
> > changes / pull request against the current master so it's more clear what
> > was modified? Something like "git pull --rebase
> > https://github.com/mozilla/mozregression.git master" then follow the
> > instructions here:
> > 
> > http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html
> > 
> > Once you've squashed everything, you can push everything back to your own
> > fork (probably with the --force option)
> 
> Thank you for these informations about git usage. It helped me a lot - a new
> pull request is done.
> 
> I will then work on inbounds builds, if everything is fine for you. :)

Yes, this looks good! I was surprised that it required so little modification.

I had a few minor requests for modification. We could put this in as-is but I imagine inbound support would only be a little more work so maybe we might as well do it at the same time?

Thanks again for this, it should be very useful for QA and other people working on FirefoxOS.
(In reply to William Lachance (:wlach) from comment #25)
> I had a few minor requests for modification. We could put this in as-is but
> I imagine inbound support would only be a little more work so maybe we might
> as well do it at the same time?

Sure. I'm not far away, but I have one problem. The url you gave me for inbound builds (
http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/
) does not seems to contains any b2g installer.

For example,
http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/b2g-inbound-linux64/1386253813/
only contains firefox-28.0a1.en-US.linux-x86_64.tar.bz2 and it is a firefox installer, not a b2g one.

Also, I took json push log revisions from:
http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/b2g-inbound-linux64/
instead of:
http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64/

Is that right ?
Flags: needinfo?(wlachance)
You should be able to find the inbound builds for B2G desktop at:
http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/

I'm not sure if these should also be listed at http://inbound-archive.pub.build.mozilla.org/ though...
Well, if use http://ftp.mozilla.org/pub/mozilla.org/b2g/tinderbox-builds/ as stated in comment #28 instead of 
http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/ as stated in comment #15, i got something working.

I updated the pull request - William, can you give it a look ?

If all is well, I'd like to refactor a bit the code to let b2ghaystack reuse some logic. And also maybe speedup the retrieving of inbounds builds like we have done in https://bugzilla.mozilla.org/show_bug.cgi?id=1003793. Maybe we can open another bug to discuss and implement this ?
Hey! So I checked out your changes and things look mostly good. The most immediate problem is that we're not bisecting inbound builds by default, I guess because b2g is not marked as supporting inbound here:

https://github.com/mozilla/mozregression/blob/master/mozregression/regression.py#L163

I think if you fix that it should "just work". You can verify that by doing something like:

mozregression -n b2g -g 2014-06-07 -b 2014-06-08

and see if it starts checking against inbound builds (and gives a reasonable regression range at the end). :) Once you're done, please update the pull request, let me know, and I can merge this in.

--

Not strictly blocking, but I realized that this won't be quite as useful as it could be as we're currently not tracking gaia revision information in desktop builds, or printing them out. I filed bug 1022885 to track that, since this bug is ultimately really about getting b2g device builds going in mozregression.

--

As far as next steps are concerned, both your ideas sound good. Feel free to open up new bugs to track them. :) Also, related to the above is bug 1022720 (make mozregression use mozversion), which would be a useful win and would help pave the way to adding gaia revision information once that's possible.
Flags: needinfo?(wlachance)
(In reply to William Lachance (:wlach) from comment #30)
> Hey! So I checked out your changes and things look mostly good. The most
> immediate problem is that we're not bisecting inbound builds by default, I
> guess because b2g is not marked as supporting inbound here:
> 
> https://github.com/mozilla/mozregression/blob/master/mozregression/
> regression.py#L163
> 
> I think if you fix that it should "just work". You can verify that by doing
> something like:
> 
> mozregression -n b2g -g 2014-06-07 -b 2014-06-08

Woops, I missed that. :) It's corrected. I tested it and all seems to goes well.

I'd like to work on bug 1022720 and then on bug 1022885 to be able to report gaia revision information in desktop builds.
(In reply to William Lachance (:wlach) from comment #32)
> I committed j.parkouss's work to date:
> 
> https://github.com/mozilla/mozregression/commit/
> 62ba14b42934fb1dbea9fe6c8d9bb5e8850abd5b
> https://github.com/mozilla/mozregression/commit/
> 6d91cb45ee746366ae9375b2c4a6b562ee440700
> 
> Still waiting for gaia revision info. And support for device builds.

Hi william,

Does "support for device builds" means finding build information with build urls and use authentication like this is done in b2ghaystack [1] ?

I made a pull request [2] that will help to support this. I didn't attach the pull request url as a patch review here as I'm not sure that it is related to this bug yet.

I'd like to work on gaia revision info, but I did not understand exactly what you want. Could you explain it to me in details, if you're interested for me to work on this ?

[1] https://github.com/mozilla/b2ghaystack/blob/master/b2ghaystack/b2ghaystack.py#L45

[2] https://github.com/mozilla/mozregression/pull/106
Hey! Sorry about the late response here.

(In reply to Julien Pagès from comment #33)
> (In reply to William Lachance (:wlach) from comment #32)
> > I committed j.parkouss's work to date:
> > 
> > https://github.com/mozilla/mozregression/commit/
> > 62ba14b42934fb1dbea9fe6c8d9bb5e8850abd5b
> > https://github.com/mozilla/mozregression/commit/
> > 6d91cb45ee746366ae9375b2c4a6b562ee440700
> > 
> > Still waiting for gaia revision info. And support for device builds.
> 
> Hi william,
> 
> Does "support for device builds" means finding build information with build
> urls and use authentication like this is done in b2ghaystack [1] ?

Yep. Sadly this is currently authentication-only, although I understand that we are looking to make it possible to flash gaia and gecko (which is probably 90% of what we want for mozregression)

> I made a pull request [2] that will help to support this. I didn't attach
> the pull request url as a patch review here as I'm not sure that it is
> related to this bug yet.

Cool, I spun off a new bug (bug 1030314) to track this.

> I'd like to work on gaia revision info, but I did not understand exactly
> what you want. Could you explain it to me in details, if you're interested
> for me to work on this ?

Unfortunately currently we aren't able to get gaia revision data from the desktop builds because of bug 1022864. When this is implemented (or we make mozregression support flame device builds, which do have this information), all we need to do is track the gaia revision information using mozversion and print it out as we do with gecko revision information. So for example we print something like the following after bisecting all the nightlies:

https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=13be61241671&tochange=4876594bacaa

In addition, I'd *also* like to print something like this:

https://github.com/mozilla-b2g/gaia/compare/efdaa4e41d39c515fb0ab88cdd4375c63767d83c...38cd96b0cd191b21f6e8744e62392843e8415f68

where efdaa4e41d39c515fb0ab88cdd4375c63767d83c and 38cd96b0cd191b21f6e8744e62392843e8415f68 are corresponding gaia commits to the gecko range. This should really be tracked in bug 1022885 though, I'll post this there.
If I understand well, we have done everything we needed here, except for what is described by bug 1022885.

Since it is already tracked maybe we can close this bug ?
Flags: needinfo?(wlachance)
Agreed, let's close this and track followups in bug 1022885 and elsewhere.
Status: NEW → RESOLVED
Closed: 9 years ago
Flags: needinfo?(wlachance)
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.