1232873 - [B2G] gecko/gaia swapping option

Status: RESOLVED INVALID

(Testing :: mozregression, defect)

Description

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Option to swap the gecko/gaia for the last two downloads:
1) This might be trickier.  We need to download the last two gecko/gaia for the device
2) once we download those we need to flash the gecko for one and the gaia for the other and flash the device with those.
3) then we need to do the opposite gecko/gaia flashing

We probably need an option to retry the flashing.
shell scripts for flashing the gecko and gaia can be found : https://github.com/Mozilla-TWQA/B2G-flash-tool/blob/master/shallow_flash.sh 

( I have my own version : https://github.com/nhirata/fullflash; not sure which one is better )

Comment 1

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I forgot to include the results needed.  We basically need to know which result is good and bad.

ie gecko good and gaia bad reproducing the issue or if gecko bad and gaia good reproduces the issue.
Which ever reproduces the issue with the bad is the side we have to check further.

Comment 2

[Julien Pagès (:parkouss)]

Ok, I'll first work on being able to implement that logic, without flashing for now.

So basically download the four files, ask to test the first good/bad couple then the other one. Depending on the result, continue the bisection for gecko or gaia only.

Comment 3

[Julien Pagès (:parkouss)]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #0)
> Option to swap the gecko/gaia for the last two downloads:
> 1) This might be trickier.  We need to download the last two gecko/gaia for
> the device

Shouldn't we do that for the first two downloads ? I thought this was required to first know if we need to bisect gaia or gecko then.

Also, I am thinking that gecko and gaia versions flashed on the phones should be compatible - that would mean we can't flash gaia (or gecko) only if we are not sure that the versions are compatible. Is that right ? If yes, do you have an idea to handle that ?

Comment 4

[Johan Lorenzo [:jlorenzo] - PTO - Back April 13th]

Discussed on IRC. Answer below, for reference.

(In reply to Julien Pagès (:parkouss) from comment #3)
> Shouldn't we do that for the first two downloads ? I thought this was
> required to first know if we need to bisect gaia or gecko then.
We need first to know what is the first broken build and the last working one. A build (like contained in aries.zip) contains a given couple (Gecko, Gaia). 

We have to download again Gecko and Gaia, because "aries.zip" contains the Android partitions files. These files are like .iso, we need to extract them in order to get the actual content of gecko and gaia. These steps are doable, but it has been easier to get Gecko and Gaia separately from taskcluster.

> Also, I am thinking that gecko and gaia versions flashed on the phones
> should be compatible - that would mean we can't flash gaia (or gecko) only
> if we are not sure that the versions are compatible. Is that right ? If yes,
> do you have an idea to handle that ?

Once we know the first broken and the last working, gecko and gaia are (almost) always compatible. That's actually the reason why we first find the first broken build. It helps limiting the risk of incompatibilities.

Comment 5

[Julien Pagès (:parkouss)]

Cool, thanks for the explanations!

So, I propose a simple workflow/usage for now:

1. find good and bad (using full package, like aries.zip): we can use the --launch option for that, ie:

mozregression --launch 2015-10-01 --app b2g-aries --repo m-c

The idea is to try multiple dates with this command (one command for one date) to find the first good and bad builds.
Note that it is broken on the current release, I just fixed it on master.

2. Once we have the range, mozregression will download and test the 4 build files like described above. Once we know if gaia or gecko is broken, we continue the bisection with only the interesting file (gaia.zip or b2g-android-arm.tar.gz). This will be a single command, eg:

mozregression --good 2015-10-01 --bad 2015-11-02 --app b2g-aries --repo m-c

Comment 6

[Julien Pagès (:parkouss)]

Attachment: gecko / gaia swapping

With those patches, running |mozregression --good 2015-09-10 --bad 2015-10-01 --repo mozilla-central --app b2g-flame| will download the 4 gaia/gecko files at first, and ask you to test two times, respectively for the couples (good gecko, bad gaia) and (good gaia, bad gecko).

Once tested, and depending on the result, mozregression will continue a "standard" bisection either with the gaia file or with the gecko file.

No automatic flashing for now - but at least mozregression should be downloading the right files.

Naoki, Johan, it would be good if one of you could actually give this a try - so you can confirm that this is what you need. Thanks!

Comment 7

[Julien Pagès (:parkouss)]

Does this looks good to you ? See comment 5 and comment 6, and if you have the time give it a try. Thanks :)

Comment 8

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I guess I can give it a try.  I'll take a look next week unless johan or someone from Kevin's team beats me to it.

We don't usually do the gecko/gaia swap in the beginning because the gaia could break from too old of a gecko or too new of a gecko because the API might not be in the gecko layer for gaia.  The thought may work for a smaller regression window, having said that if the regression windows is a couple of months back then it might not work well.

We reserve the swapping at the end.

Also to note, if we determine it's a gaia issue we can just use the git bisect at that point and build the gaia.  Building gaia would be a lot faster than trying to download it all the time I think.  The gecko becomes more of an issue.

Comment 9

[Julien Pagès (:parkouss)]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #8)
> I guess I can give it a try.  I'll take a look next week unless johan or
> someone from Kevin's team beats me to it.

Well don't give it a try yet, since it is not quite what is needed. :) I prefer not to spend our time in testing something that is not what we want, so first I need to perfectly understand the need.

> We don't usually do the gecko/gaia swap in the beginning because the gaia
> could break from too old of a gecko or too new of a gecko because the API
> might not be in the gecko layer for gaia.  The thought may work for a
> smaller regression window, having said that if the regression windows is a
> couple of months back then it might not work well.
> 
> We reserve the swapping at the end.

Oh, I guess I misread you somehow. So the idea is to download and flash the full file ('aries.zip' for aries) for each build in the bisection and only when we have found the smallest possible regression window we do the swap to know whether it is a gaia or a gecko regression. Right ?

> Also to note, if we determine it's a gaia issue we can just use the git
> bisect at that point and build the gaia.  Building gaia would be a lot
> faster than trying to download it all the time I think.  The gecko becomes
> more of an issue.

Hm, I'm not sure I understand here. If we found a gaia issue, then we are done with mozregression bisection anyway. (mozregression has no more files that can be tested since we do that check at the end). So there is no possible downloads that mozregression can make - but we can print something to say that the gaia range can be reduced by using git bisect.

Comment 10

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

> Well don't give it a try yet, since it is not quite what is needed. :) I
> prefer not to spend our time in testing something that is not what we want,
> so first I need to perfectly understand the need.

Ok.

> Oh, I guess I misread you somehow. So the idea is to download and flash the
> full file ('aries.zip' for aries) for each build in the bisection and only
> when we have found the smallest possible regression window we do the swap to
> know whether it is a gaia or a gecko regression. Right ?

That is correct.  We narrow down the smallest possible regression before we do the swap.
Otherwise we might run into other issues (ie API missing and the bug reproduces or doesn't reproduce in both)

> Hm, I'm not sure I understand here. If we found a gaia issue, then we are
> done with mozregression bisection anyway. (mozregression has no more files
> that can be tested since we do that check at the end). So there is no
> possible downloads that mozregression can make - but we can print something
> to say that the gaia range can be reduced by using git bisect.

Taskcluster builds are based on HG gecko pulls.  There could be multiple gaia in between, in which case doing a git bisect on gaia might be easier.  I think the main thing we're concerned about is gecko regressions for mozregression.  I don't think you need to worry about gaia regressions so much.

Comment 11

[Johan Lorenzo [:jlorenzo] - PTO - Back April 13th]

Clearing NI per comment 9: No need to test it out, yet.

Comment 12

[Julien Pagès (:parkouss)]

Attachment: example log

I reworked my patches, and I think this is quite good now.

I attached an example of a run. So at the end of the bisection, mozregression will help you to test whether gaia or gecko is broken (by downloading files and ask to flash manually for now).

I took the opportunity to change the output of downloaded files with taskcluster, and now you will see:

> Download INFO Downloading build from https://tools.taskcluster.net/task-inspector/#FGBSRkB7SFa25XKnfl-gPw/0, using file private/build/aries.zip

instead of the real build url. I believe it is more convenient, and also it probably was a security hole to print the real url with the full bewit parameter.

So when one of you guys have some time, please give it some quick try, so we can land this and improve other things, such as calling scripts to flash automatically. If you are using virtualenv, here is how you can test:

git clone https://github.com/parkouss/mozregression
virtualenv -p /usr/bin/python2 venv
source venv/bin/activate
cd mozregression
git checkout b2g-mode
pip install -e .

# now, you can run mozregression, e.g

mozregression -g 2016-01-05 -b 2016-01-08 --app=b2g-aries

Comment 13

[Julien Pagès (:parkouss)]

Attachment: correct log

Yay, sorry I uploaded a not up to date log. This one is better.

Comment 14

[Johan Lorenzo [:jlorenzo] - PTO - Back April 13th]

Sorry, it's already been a week. I'm not sure if I'll have enough time to test it. NI'ing nhirata for back up.

Comment 15

[Jayme Mercado [:JMercado] (Inactive 3/4/2016)]

Attachment: console output mozregress.txt

I used the terminal commands from comment 12 and ran the mozregression tool for a window I had previously ran for bug 1239702.  

The tool failed to find the correct window after going to fx-team inbound instead of mozilla-inbound and only then attempted to perform swaps before giving an error.

When performing a manual regression we perform swaps at two points.  The first and most important time we perform swaps is after finding the last working/first broken in central.  We use this to determine where to go next: if it is a gaia issue we go to b2g inbound or if it is a gecko issue we use the pushlog to determine whether to go.  Often there is only one inbound in the gecko pushlog, but if there is more than one it gets complicated as we must determine which inbound actually had the cause by going into each and determining which occurred first.

We then do a second set of swaps when we find the last work/first broken again in whatever inbound was chosen.  These swaps are more of a safeguard against finding the wrong last/first pair or having incorrectly identified the broken portion in the earlier swaps.  If the swaps don't both show the same part as broken we know something went wrong with our window.

Comment 16

[Jayme Mercado [:JMercado] (Inactive 3/4/2016)]

One other thing that is not a bug in the tool but slowed me down some was that the tool was downloading the user builds instead of the debug builds.  So root and adb were off by default so I would have to enable it before reflashing.

Comment 17

[Julien Pagès (:parkouss)]

Thanks for giving it a try.

(In reply to Jayme Mercado [:JMercado] from comment #15)
> Created attachment 8712410 [details]
> console output mozregress.txt
> 
> I used the terminal commands from comment 12 and ran the mozregression tool
> for a window I had previously ran for bug 1239702.  
> 
> The tool failed to find the correct window after going to fx-team inbound
> instead of mozilla-inbound and only then attempted to perform swaps before
> giving an error.

Hm, looking at your log mozregression switches to fx-team because when you narrowed down on mozilla-central the latest bad push was a merge originated from fx-team:

https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=fe2802097d8613deb9f0d89b6ed876e314c886d0&tochange=091d6785608297bab2cf36b1b6a2d9a388b82a01

This is how mozregression works, when a merge is detected it continue the bisection on the branch where the branch comes from. Looking at the rest of your comment, I guess we don't want to apply that rule for b2g.

I am not sure why you got a Not authorized error at the end (and only at the end).

> When performing a manual regression we perform swaps at two points.  The
> first and most important time we perform swaps is after finding the last
> working/first broken in central.

Oh. I missed that part of information: first swap done after bisection *in central*. Ok, got it.

> We use this to determine where to go next:
> if it is a gaia issue we go to b2g inbound or if it is a gecko issue we use
> the pushlog to determine whether to go.  Often there is only one inbound in
> the gecko pushlog, but if there is more than one it gets complicated as we
> must determine which inbound actually had the cause by going into each and
> determining which occurred first.

Hm, maybe you could provide me an example, or point me to some documentation ? I am not sure I understand the story for the pushlog and gecko...

> We then do a second set of swaps when we find the last work/first broken
> again in whatever inbound was chosen.  These swaps are more of a safeguard
> against finding the wrong last/first pair or having incorrectly identified
> the broken portion in the earlier swaps.  If the swaps don't both show the
> same part as broken we know something went wrong with our window.

ok.

(In reply to Jayme Mercado [:JMercado] from comment #16)
> One other thing that is not a bug in the tool but slowed me down some was
> that the tool was downloading the user builds instead of the debug builds. 
> So root and adb were off by default so I would have to enable it before
> reflashing.

Oh, you should be able to get them with:

> mozregression -g 2015-12-15 -b 2015-12-25 --app=b2g-flame -B debug

Comment 18

[Johan Lorenzo [:jlorenzo] - PTO - Back April 13th]

Thanks for the help Jayme!

Comment 19

[Jayme Mercado [:JMercado] (Inactive 3/4/2016)]

I was attempting to find a window which I could use as a reference so this one is a bit old but it will show what I mean hopefully.  In this window the issue is determined to be a gecko change after the swaps are performed.  In the pushlog there are commits from two inbounds (fx-team and b2g-inbound) as well as some pushes straight to Central.

How I would go about attempting to determine which inbound had the cause is this (Note I don't have the exact build IDs for this step but will give examples that should make sense, if it doesn't please let me know):  The first broken build in central is 20140529103003 as seen in the window below.  My first step is to install the first build created in each inbound inbound AFTER that build.  If that build does NOT reproduce the issue that inbound can be eliminated from the inbound options.

For example if b2g-inbound had builds at 20140529100520 and 20140529103405 and fx-team had builds made 20140529080330 and 20140529105943 you would want the later one for both but not the earlier one for this check.  If 20140529103405 (b2g-inbound) reproduced the issue but 20140529105943 (fx-team) did not, you would know that the correct inbound is b2g-inbound.  If the situation was reversed it would be fx-team.  If NEITHER reproduced the cause is likely one of the pushes that went straight to central.  Finally, if both reproduce is when the most work needs to be done.

In the case where multiple inbounds pass the above test the window just has to be performed in the remaining inbounds.  A rule of thumb we have is to do this in the following order:  Mozilla inbound then, B2g-Inbound, then Fx-team or other inbounds should they appear.  Most windows that have both mozilla and b2g inbound that pass the above come from Mozilla-inbound.  Windows do come from fx-team inbound but are not as common as the other two inbounds.  

This is the most troubling part: If the pushlog for an inbound shows a merge from either central or another inbound it is likely the wrong inbound.  Extremely rarely all of the inbounds will have cross merges, if that is the case the only recourse is to parse through the logs manually.  I know of no process that will always pick the correct cause in that final situation

I know this isn't the easiest method to turn into a tool, but I hope it helps.  Let me know if there's any more questions.

Central Regression Window:

Last working Environmental Variables:
Device: Flame
BuildID: 20140529013001
Gaia: 4142df90c71abdc1e3a87cd37dff1a22d5e38b34
Gecko: 1e712b724d17
Version: 32.0a1

First Broken Environmental Variables:
Device: Flame
BuildID: 20140529103003
Gaia: b669dd2cc321f37cebc7081a79b968cac36b4200
Gecko: a3848fbadb16
Version: 32.0a1

Last working gaia / First broken gecko - issue occurs - indicates a gecko issue
Gaia: 4142df90c71abdc1e3a87cd37dff1a22d5e38b34
Gecko: a3848fbadb16

First broken gaia / Last working gekko - issue does not occur
Gaia: b669dd2cc321f37cebc7081a79b968cac36b4200
Gecko: 1e712b724d17

Central Pushlog: http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=1e712b724d17&tochange=a3848fbadb16

Comment 20

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

whoops.  looks like Jayme handled it.  clearing my ni.

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+1]]

B2g is gone.