1431403 - Fail to open the application on bisection with Fennec builds

Status: VERIFIED FIXED

(Testing :: mozregression, defect)

Description

[Oana Horvath [:ohorvath]]

STR:
1. Open the mozregression GUI or the WIN terminal.
2. Start a new bisection, choose application: fennec, repository: mozilla-central / beta/ release.
3. Pick the last known good and bad build dates. E.g. 2018-01-03 / 2018-01-17.
4. Run.

Actual results:
The app is installed, but we cannot give the verdict after testing: if the build is good or bad. 
In the GUI The build appears as skipped.
I've encountered the following errors:
GUI: https://www.screencast.com/t/wpKhrgFxK8
Terminal: 
https://www.screencast.com/t/cupsndg3O4ZM
https://www.screencast.com/t/V23TeQ8yB

Comment 1

[Ioana Chiorean]

Moving the importance to blocker as this is blocking the QA work.

Comment 2

[William Lachance (:wlach)]

This sort of thing has cropped up before, I suspect that the problem is with mozdevice, which does a bunch of stuff which is desirable in automation but not here. We should probably just copy the bits of code that do the things we want and forget the rest. In any case, I'll take a look.

Comment 3

[William Lachance (:wlach)]

Hi Ioana, this is actually working fine for me on my pixel 2, minus a few warning messages. For now let's try debugging this with the command line version. What model phone are you using? Can you connect to it via adb (i.e. does `adb shell` work for you?)

Comment 4

[Oana Horvath [:ohorvath]]

We use a lot of different devices to which we can connect to via adb. But for e.g. for taking these screenshots, I've used a Sony Xperia Z5 Premium (Android 6.0.1) & a Samsung Galaxy Tab 3 (Android 7.0).

Comment 5

[William Lachance (:wlach)]

(In reply to Oana Horvath from comment #4)
> We use a lot of different devices to which we can connect to via adb. But
> for e.g. for taking these screenshots, I've used a Sony Xperia Z5 Premium
> (Android 6.0.1) & a Samsung Galaxy Tab 3 (Android 7.0).

Ok, can you connect to these devices from adb and install a downloaded copy of fennec (adb install -r) on them in the same environment as the one you're using to run mozregression?

Comment 6

[Ioana Chiorean]

As Oana is in PTO tomorrow, maybe Bogdan or Sorina can check this?

Comment 7

[Oana Horvath [:ohorvath]]

adb install works on both of them, using a Nightly build, as long as it's a fresh install or an upgrade: 
https://www.screencast.com/t/gTfiWpX0c
https://www.screencast.com/t/XiUpBe4s

Can't do adb install -r with older versions to replace the current(/latest) version: "Failed to install fennec-59.0a1.multi.android-arm.apk: Failure [INSTALL_FAILED_VERSION_DOWNGRADE]".

Comment 8

[Oana Horvath [:ohorvath]]

(In reply to Oana Horvath from comment #7)
> adb install works on both of them, using a Nightly build:
> https://www.screencast.com/t/gTfiWpX0c
> https://www.screencast.com/t/XiUpBe4s

Sorry, for the downgrade I forgot to add "-d".

Comment 9

[William Lachance (:wlach)]

(In reply to Oana Horvath from comment #8)
> (In reply to Oana Horvath from comment #7)
> > adb install works on both of them, using a Nightly build:
> > https://www.screencast.com/t/gTfiWpX0c
> > https://www.screencast.com/t/XiUpBe4s
> 
> Sorry, for the downgrade I forgot to add "-d".

Been playing a bit with this locally, I'm still confused at what might be happening, as mozregression should be giving more diagnostic information when this type of error occurs.

Things to try:
1. Make extra sure the phone's screen is on and unlocked when running mozregression (there is a developer option to keep the screen on, I recommend enabling it during testing)
2. If that doesn't help, can you try running the commandline version of mozregression with "-d" for more debugging information?

Comment 10

[Bogdan Surd, QA [:bsurd, NI]]

Attachment: mozregression.jpg

Hello,

I've run the tool using the command line and have attached the image file with the error that I get attached.

I'm getting the same error regardless of the device that I use.

Comment 11

[Bogdan Surd, QA [:bsurd, NI]]

Attachment: mozregression.txt

Comment 12

[William Lachance (:wlach)]

Thank you Bogdan, that is quite helpful. Or at least it helps us get to the next stage of our investigation.

Unfortunately it looks like we're not logging exception information in mozregression, which makes it quite difficult to see what is actually happening here.

I've been debugging a bunch locally and it seems like if you pass "exc_info=True" to our log handler (as we do here: https://github.com/mozilla/mozregression/blob/master/mozregression/launchers.py#L69), it doesn't *actually* log the full traceback as you'd expect. Looking internally at the mozlog source code, it seems like that data just gets stored in a variable called "stack" (https://searchfox.org/mozilla-central/rev/4611b9541894e90a421debb57ddbbcff55c2f369/testing/mozbase/mozlog/mozlog/structuredlog.py#496) which doesn't bubble up anywhere with the log formatter/handler setup we have in mozregression (https://github.com/mozilla/mozregression/blob/master/mozregression/log.py#L40).

It looks like in mach we manually add the traceback data to the log output (https://searchfox.org/mozilla-central/rev/4611b9541894e90a421debb57ddbbcff55c2f369/testing/mozbase/mozlog/mozlog/formatters/machformatter.py#333). Let's do that here:

https://github.com/mozilla/mozregression/pull/481

Comment 13

[William Lachance (:wlach)]

Hi, I just uploaded a version of mozregression with the logging fixes to pypi (2.3.23). Could you try running it again with that version and debugging enabled? Not sure who to needinfo so pinging Ioana again. Thanks!

Comment 14

[Bogdan Surd, QA [:bsurd, NI]]

Attachment: log.txt

Here is the log with the new version of mozregression installed.
I updated mozregression using pip install --upgrade mozregression. Please let me know if a clean install would have been better or if it's fine updating moz using the command line.

Also for any further assistance with this issue please NI me.

Comment 15

[William Lachance (:wlach)]

Ah excellent, that log shows that the problem is in pushing the profile. It looks like there was a fix to pushing stuff in mozdevice which didn't yet make it to a release on pypi-- maybe if we're lucky just releasing a new version of mozdevice and making mozregression depend on it will fix the problem.

I just released mozregression 2.3.24 with that change. Can you upgrade your copy and try again? Fingers crossed.

(NI'ing ioana again, but feel free to reassign to the right person)

Comment 16

[Bogdan Surd, QA [:bsurd, NI]]

Attachment: log.txt

Hello,

Upgraded to 2.3.24 but I'm still getting some errors. Here is the log file from the upgraded version.

Comment 17

[William Lachance (:wlach)]

I suspect the issue is with running mozregression on windows. I'll investigate a little deeper soon (tomorrow hopefully).

Comment 18

[William Lachance (:wlach)]

(In reply to William Lachance (:wlach) (use needinfo!) from comment #17)
> I suspect the issue is with running mozregression on windows. I'll
> investigate a little deeper soon (tomorrow hopefully).

Ok so I *think* the issue is that we're using sys.path.normpath to specify the remote path, which doesn't work on Windows:

https://searchfox.org/mozilla-central/source/testing/mozbase/mozdevice/mozdevice/adb.py#1731

If we change that to posixpath.normpath I think we should be good. I will try tomorrow when I have access to a Windows machine (unfortunately I can't seem to get adb working from my virtualbox instance here at work).

Comment 19

[William Lachance (:wlach)]

This should be resolved once we have merged bug 1434664, released mozdevice 0.52, and release a new version of mozregression depending on that. Hopefully this will happen soon. Thanks for your patience everyone.

Comment 20

[William Lachance (:wlach)]

Hi Ioana, I think (hope?) the issue is now fixed with the latest version of mozregression. If you (or someone else) could verify that I would appreciate it. :)

Comment 21

[Bogdan Surd, QA [:bsurd, NI]]

Verified it using a Nexus 5 and Nexus 6, the GUI as well as the command line both work fine now. Thank you!