935183 - mozregression should support using inbound builds

Status: RESOLVED FIXED

(Testing :: mozregression, defect)

Description

[William Lachance (:wlach)]

With the increasing number of patches landing every day, the usefulness of mozregression is less than it used to be. We recently added the archiving of the last 6 months worth of desktop inbound builds on S3. If we used those inside mozregression, we could get a much tighter regression range.

From my understanding, we have to be a bit careful using the inbound builds during bisection as the build time may not correspond to the date of the revision. 

We should probably also start just by using nightlies, and only switch to inbound once we've narrowed the regression range to within a day.

It may also be useful to do predictive downloading of intermediary builds while a current one is being unpacked/tested to reduce turnaround time given that we're probably going to be testing more builds once this lands.

Comment 1

[William Lachance (:wlach)]

Hey Benoit, could you provide a few pieces of information on:

1. Where exactly are these inbound builds archived?
2. Do you have any thoughts on how we could get around the fact that the build date might not correspond to the revision ordering? Do we need to cross-correlate builds with the mercurial tree or something?

Comment 2

[Jonathan Griffin (:jgriffin)]

It looks like the builds are available at: 

http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/

Comment 3

[Benoit Girard (:BenWa)]

(In reply to William Lachance (:wlach) from comment #1)
> Hey Benoit, could you provide a few pieces of information on:
> 
> 1. Where exactly are these inbound builds archived?
Bug 765258 should have all the relevant information hanging off it.
> 2. Do you have any thoughts on how we could get around the fact that the
> build date might not correspond to the revision ordering? Do we need to
> cross-correlate builds with the mercurial tree or something?

There's a few ways we can do (in order from most to least favorite):
1) Have to inbound archive maintain symlink that are in order and build the index as we populate the archive.
2) Maintain an index outside
3) Fetch all the builds for the relevant date range (and possibly more since some will be outside the range because of job queues), parse the json and lookup the push data against hg.

Comment 5

[Jonathan Griffin (:jgriffin)]

I think any of these solutions probably requires us to look up push data against hg, it's just a matter of where that logic lives.  Probably the most practical solution for the short term is #3, although I agree #1 or #2 make more sense long-term.

Comment 6

[Milan Sreckovic [:milan] (needinfo for best results)]

 I would definitely start with #3.  It may not be that bad, but even if it is, it would still be an improvement over what we have today.  A quick improvement may be maintaining the index mentioned in #2 locally on a computer you're using.  It'd still be "slow" the first time, but it could improve if you have a lot of regressions in the similar calendar span...  probably too much to hope for.

Comment 7

[Gregory Szorc [:gps]]

Can tree herder expose all this information through an easy-to-query API?

My gecko-dev hg extension (https://hg.mozilla.org/users/gszorc_mozilla.com/hgext-gecko-dev) maintains a SQLite database with pushlog data. It synchronizes pushlog when you `hg pull`. I've been wanting to supplement the database with build results. I've been waiting on the data to be exposed in a more friendly format than per-day or 4 hour gzipped JSON files (which are too large to fetch and store).

We're essentially missing a simple "look up build metadata for a single push" API. Build that and so many problems go away. I assume tree herder is building this...

Comment 8

[Gregory Szorc [:gps]]

Forgot to mention: the code for my hg extension to track pushes is written as a standalone module and doesn't require hg modules. https://hg.mozilla.org/users/gszorc_mozilla.com/hgext-gecko-dev/file/218c65fe5456/mozautomation/changetracker.py

db = ChangeTracker('db.db')
db.load_pushlog('central')
# pushlog data is loaded from mozilla-central and is available for query

You may need to write some additional APIs for querying, as the existing ones are pretty minimal.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I would like to mention again (what I already did on the duped bug) that mozdownload is most likely the best place to get this feature implemented in. mozregression should really make use of that tool. I see it contra-productive to start working on new features again for different tools, while we could easily depend on other packages.

For this specific feature we have:
https://github.com/mozilla/mozdownload/issues/173

Comment 10

[William Lachance (:wlach)]

(In reply to Henrik Skupin (:whimboo) from comment #9)
> I would like to mention again (what I already did on the duped bug) that
> mozdownload is most likely the best place to get this feature implemented
> in. mozregression should really make use of that tool. I see it
> contra-productive to start working on new features again for different
> tools, while we could easily depend on other packages.
> 
> For this specific feature we have:
> https://github.com/mozilla/mozdownload/issues/173

I sympathize with the sentiment, but I suspect making mozregression use mozdownload would probably take at least a few days of work and I don't think we should gate a quick developer win on that happening. They're both written in python so I think it should be relatively straightforward to port this feature from mozregression to mozdownload once it's implemented.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Might work too. :) So I would wait with any work on mozdownload then.

Comment 12

[William Lachance (:wlach)]

I'm going to take this. I will try for strategy (3) at first.

Comment 13

[William Lachance (:wlach)]

(In reply to Gregory Szorc [:gps] from comment #7)
> Can tree herder expose all this information through an easy-to-query API?
> 
> My gecko-dev hg extension
> (https://hg.mozilla.org/users/gszorc_mozilla.com/hgext-gecko-dev) maintains
> a SQLite database with pushlog data. It synchronizes pushlog when you `hg
> pull`. I've been wanting to supplement the database with build results. I've
> been waiting on the data to be exposed in a more friendly format than
> per-day or 4 hour gzipped JSON files (which are too large to fetch and
> store).
> 
> We're essentially missing a simple "look up build metadata for a single
> push" API. Build that and so many problems go away. I assume tree herder is
> building this...

I have no idea if this is on their agenda. I agree it would be useful. Perhaps it would be useful to file a bug (CCing :jeads, :camd, :mdoglio) to see and/or persuade.

Comment 14

[William Lachance (:wlach)]

Ok, I have a prototype up at:

http://github.com/wlach/mozregression

Installation goes something like:

git checkout https://github.com/wlach/mozregression.git
cd mozregression
virtualenv .
source bin/activate
pip install -e .

Then you should be able to use mozregression as normal, but it'll fall back to inbound after it's narrowed the nightly range down.

This still needs some cleanup and probably review before it's ready to go. I'm also worried that mozcommitbuilder doesn't seem to be working with it, though I wonder if that might be due to mozcommitbuilder being broken. Will look into that before pushing to master.

Meanwhile, feedback and bug reports welcome. I think it should be useful as-is.

Comment 15

[Benoit Girard (:BenWa)]

Thanks. I'll take a look.

Comment 16

[Milan Sreckovic [:milan] (needinfo for best results)]

(In reply to William Lachance (:wlach) from comment #14)
> Ok, I have a prototype up at:
> 
> http://github.com/wlach/mozregression
> 
> Installation goes something like:
> 
> git checkout https://github.com/wlach/mozregression.git
> cd mozregression
> virtualenv .
> source bin/activate
> pip install -e .
> 
> Then you should be able to use mozregression as normal, but it'll fall back
> to inbound after it's narrowed the nightly range down.

This is very cool, can't wait to try it.  I don't have all the packages necessary to do it from these instructions (and you may need 'git clone' above), so Benoit, let me know how it goes.

Comment 17

[Benoit Girard (:BenWa)]

I had to install pip and virtualenv. I'm not familiar with python dependencies but we should make sure the process is straight forward for contributors who don't have these installed.

Comment 18

[Benoit Girard (:BenWa)]

Attachment: Exception when FTP server is down

The FTP server is down for maintenance. Might be good to translate this exception into an error message for contributors.

Comment 19

[Gregory Szorc [:gps]]

If you have m-c, you have pip and virtualenv. You can install virtualenv from python/virtualenv if you have the tree available. And, it's likely more recent than the system's version because we've upgraded our virtualenv to ensure we can support Python wheel packages.

Comment 20

[Benoit Girard (:BenWa)]

(In reply to Gregory Szorc [:gps] from comment #19)
> If you have m-c, you have pip and virtualenv. You can install virtualenv
> from python/virtualenv if you have the tree available. And, it's likely more
> recent than the system's version because we've upgraded our virtualenv to
> ensure we can support Python wheel packages.

Right but the goal is to make mozregression standalone so that more users can contribute regression windows.

Comment 21

[William Lachance (:wlach)]

(In reply to Benoit Girard (:BenWa) from comment #20)
> (In reply to Gregory Szorc [:gps] from comment #19)
> > If you have m-c, you have pip and virtualenv. You can install virtualenv
> > from python/virtualenv if you have the tree available. And, it's likely more
> > recent than the system's version because we've upgraded our virtualenv to
> > ensure we can support Python wheel packages.
> 
> Right but the goal is to make mozregression standalone so that more users
> can contribute regression windows.

For that use case, it should be possible to wrap mozregression up into a standalone executable and put a graphical front end on it. I think that's out of scope for this bug though. :)

Comment 22

[Milan Sreckovic [:milan] (needinfo for best results)]

So, is the installation procedure for the new and enhanced mozregression going to remain about the same?  These are today's instructions:

Windows

    install MozillaBuild
    Run C:\mozilla-build\start-l10n.bat. This will give you a command-line prompt.
    Type this into the command-line and press Enter:

    easy_install mozregression

Mac
Run these commands in the Terminal:

curl -O http://peak.telecommunity.com/dist/ez_setup.py
sudo python ez_setup.py
sudo easy_install mozregression

Ubuntu
Run these commands in the Terminal:

sudo apt-get install python-pip
sudo pip install mozregression

Would that still work?

Comment 23

[Benoit Girard (:BenWa)]

It would be nice if we supported background downloads. It would mean fetching two builds (bad, good). Of course we need to abort is we were still fetching the first build when we got a response and needed the second. Is that asking for too much?

Comment 24

[William Lachance (:wlach)]

(In reply to Milan Sreckovic [:milan] from comment #22)
> So, is the installation procedure for the new and enhanced mozregression
> going to remain about the same?  These are today's instructions:
> ...

Yes, once we've uploaded my version of mozregression to pypi (later this week?) these instructions should work.

Comment 25

[William Lachance (:wlach)]

(In reply to Benoit Girard (:BenWa) from comment #23)
> It would be nice if we supported background downloads. It would mean
> fetching two builds (bad, good). Of course we need to abort is we were still
> fetching the first build when we got a response and needed the second. Is
> that asking for too much?

I don't think this should be terribly difficult. I will try to get to this either later today or this week at worst.

Comment 26

[Benoit Girard (:BenWa)]

Attachment: log

I just completed a run and it defaults to compiling the builds. I see a few inbound builds for those date at this URL but not nearly enough so I'm not sure why that is:

http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/b2g-inbound-macosx64/1381602569/

Comment 27

[William Lachance (:wlach)]

(In reply to Benoit Girard (:BenWa) from comment #26)
> Created attachment 8334803 [details]
> log
> 
> I just completed a run and it defaults to compiling the builds. I see a few
> inbound builds for those date at this URL but not nearly enough so I'm not
> sure why that is:
> 
> http://inbound-archive.pub.build.mozilla.org/pub/mozilla.org/firefox/
> tinderbox-builds/b2g-inbound-macosx64/1381602569/

So this log suggests that you're not running my version of mozregression. You need to install it per my instructions above, since we haven't yet uploaded the new version to pypi "pip install" won't work.

Comment 28

[William Lachance (:wlach)]

(In reply to William Lachance (:wlach) from comment #25)
> (In reply to Benoit Girard (:BenWa) from comment #23)
> > It would be nice if we supported background downloads. It would mean
> > fetching two builds (bad, good). Of course we need to abort is we were still
> > fetching the first build when we got a response and needed the second. Is
> > that asking for too much?
> 
> I don't think this should be terribly difficult. I will try to get to this
> either later today or this week at worst.

Ok, after looking into it a bit this is actually harder than I thought it would be. Not really difficult but probably a solid day or so's worth of effort and refactoring. I don't think we should block inbound support on this landing. Someone had actually already filed the feature request here:

https://github.com/mozilla/mozregression/issues/61

Comment 29

[Benoit Girard (:BenWa)]

Ok no problem.

Comment 30

[William Lachance (:wlach)]

Ok, the work to support inbound builds has landed in the main repo and we did a new release to pypi, so the instructions above in comment 22 should get you a version of mozregression that will test inbound builds as well. Please test it out and report any problems in the issues section.

One caveat is that Fennec doesn't have inbound support yet. I filed a github issue for this here:

https://github.com/mozilla/mozregression/issues/74

Comment 31

[William Lachance (:wlach)]

(resolving this as fixed, reopen if there are issues)

Comment 32

[Milan Sreckovic [:milan] (needinfo for best results)]

This is going to make a lot of difference, thanks!

Comment 33

[William Lachance (:wlach)]

Found some issues, fixed them here: https://github.com/mozilla/mozregression/pull/75

Will close when addressed and we've pushed a new version to pypi.

Comment 34

[William Lachance (:wlach)]

Ok, fixed the dependant issue, uploaded new version to pypi. I think we should now be good!