1465145 - bisecting API 11 fennec on API15 Android devices isn't working

Status: UNCONFIRMED

(Testing :: mozregression, defect)

Description

[Djfe]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0
Build ID: 20180528220216



Actual results:

platform: Windows-10-10.0.17134
python: 2.7.14 FROZEN (32bit)
mozregui: 0.9.22
mozregression: 2.3.27
message: IndexError: list index out of range
traceback:   File ".\mozregui\bisection.py", line 82, in check_merge
  File "..\mozregression\bisector.py", line 263, in handle_merge
  File "..\mozregression\build_range.py", line 94, in __getitem__

2018-05-29T11:16:27: DEBUG : ADBHost: {'_logger': <mozlog.structuredlog.StructuredLogger object at 0x04480690>, '_adb_version': '', '_adb_host': None, '_adb_port': None, '_timeout': 300, '_polling_interval': 0.1, '_adb_path': 'adb', '_verbose': False}
2018-05-29T11:17:27: INFO : TaskCluster only keeps builds for one year. Using 2017-05-29 11:17:27.786000 instead of 2015-12-03.
2018-05-29T11:17:27: INFO : TaskCluster only keeps builds for one year. Using 2017-05-29 11:17:27.786000 instead of 2015-12-04.
2018-05-29T11:17:27: DEBUG : Using url: https://hg.mozilla.org/mozilla-central/json-pushes?startdate=2017-05-29&enddate=2017-05-30
2018-05-29T11:17:28: INFO : Using fc4d115e549ae73dc220bdeb486de0bd47bacedc (pushed on 2017-05-29 08:18:15) for date 2017-05-29 11:17:27.786000
2018-05-29T11:17:28: INFO : Using 34ac1a5d6576d6775491c8a882710a1520551da6 (pushed on 2017-05-29 15:18:05) for date 2017-05-29 11:17:27.786000
2018-05-29T11:17:28: DEBUG : using taskcluster route 'gecko.v2.mozilla-central.revision.fc4d115e549ae73dc220bdeb486de0bd47bacedc.mobile.android-api-15-debug'
2018-05-29T11:17:28: DEBUG : using taskcluster route 'gecko.v2.mozilla-central.revision.cce4d83d2b99ffedbd67a2f40ce26e53e9ae27ab.mobile.android-api-15-debug'
2018-05-29T11:17:28: DEBUG : using taskcluster route 'gecko.v2.mozilla-central.revision.34ac1a5d6576d6775491c8a882710a1520551da6.mobile.android-api-15-debug'
2018-05-29T11:17:35: DEBUG : using alternate debug route 'gecko.v2.mozilla-central.revision.fc4d115e549ae73dc220bdeb486de0bd47bacedc.mobile.android-api-15-dbg'
2018-05-29T11:17:35: DEBUG : using alternate debug route 'gecko.v2.mozilla-central.revision.cce4d83d2b99ffedbd67a2f40ce26e53e9ae27ab.mobile.android-api-15-dbg'
2018-05-29T11:17:41: WARNING : Skipping build fc4d115e549a: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.fc4d115e549ae73dc220bdeb486de0bd47bacedc.mobile.android-api-15-debug'
2018-05-29T11:17:42: WARNING : Skipping build cce4d83d2b99: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.cce4d83d2b99ffedbd67a2f40ce26e53e9ae27ab.mobile.android-api-15-debug'
2018-05-29T11:17:42: DEBUG : Starting merge handling...


Expected results:

I'm unable to search the commit that introduced a regression between 2015-12-03 and 2015-12-04
(I used 5th of december here, because of another bug)

Seems like Mozregression assumes, that the available API version needs to be smaller/equal to the version online, while it is the other way around.
bigger/equal is the way to go.

API 15 devices should just get API 11 from back then and it would work fine.

I already did manual bisecting that's why I know, that 2015-12-04 introduced the bug. And I know, that these apk files work on my android device just fine.

Regards,
Djfe

Comment 1

[Djfe]

I just realized that my Oneplus 3T is api level 24 (7.0) and not 15 (4.0)

Comment 2

[William Lachance (:wlach)]

Djfe: What are the exact parameters you're giving to mozregression? mozregression doesn't really care about api versions except to figure out what's available on certain dates.

As the output suggests, we can't use taskcluster to get builds for something that old, but we should at least be able to do a nightly regression analysis.

Comment 3

[Djfe]

Hi William,
thanks for replying :)

I rebooted and now I can't reproduce the following error any longer:
traceback:   File ".\mozregui\bisection.py", line 82, in check_merge
  File "..\mozregression\bisector.py", line 263, in handle_merge
  File "..\mozregression\build_range.py", line 94, in __getitem__

I think, I still used the same config as before.

It seems to work now, except that taskcluster doesn't have the nightlies any longer, but that is intended after all.

I tdoesn't have enough data to bisect
https://i.imgur.com/1rpeS0Q.png

Log:
2018-05-30T15:26:20: DEBUG : ADBHost: {'_logger': <mozlog.structuredlog.StructuredLogger object at 0x039F8690>, '_adb_version': '', '_adb_host': None, '_adb_port': None, '_timeout': 300, '_polling_interval': 0.1, '_adb_path': 'adb', '_verbose': False}
2018-05-30T15:26:31: INFO : TaskCluster only keeps builds for one year. Using 2017-05-30 15:26:31.199000 instead of 2015-12-03.
2018-05-30T15:26:31: INFO : TaskCluster only keeps builds for one year. Using 2017-05-30 15:26:31.199000 instead of 2015-12-04.
2018-05-30T15:26:31: DEBUG : Using url: https://hg.mozilla.org/mozilla-central/json-pushes?startdate=2017-05-30&enddate=2017-05-31
2018-05-30T15:26:32: INFO : Using 0c712d76d598ec92a7adaaf7180fd4e559f9abc0 (pushed on 2017-05-30 09:14:52) for date 2017-05-30 15:26:31.199000
2018-05-30T15:26:32: INFO : Using fbe0e3f31233cc7846a2168c613452a1d7147cc2 (pushed on 2017-05-30 20:09:20) for date 2017-05-30 15:26:31.199000
2018-05-30T15:26:32: DEBUG : using taskcluster route 'gecko.v2.mozilla-central.revision.0c712d76d598ec92a7adaaf7180fd4e559f9abc0.mobile.android-api-9-debug'
2018-05-30T15:26:32: DEBUG : using taskcluster route 'gecko.v2.mozilla-central.revision.fbe0e3f31233cc7846a2168c613452a1d7147cc2.mobile.android-api-9-debug'
2018-05-30T15:26:32: DEBUG : using taskcluster route 'gecko.v2.mozilla-central.revision.c3cd8c58fa87d571874f2bcc34ec0864ac7acd5f.mobile.android-api-9-debug'
2018-05-30T15:26:39: DEBUG : using alternate debug route 'gecko.v2.mozilla-central.revision.c3cd8c58fa87d571874f2bcc34ec0864ac7acd5f.mobile.android-api-9-dbg'
2018-05-30T15:26:39: DEBUG : using alternate debug route 'gecko.v2.mozilla-central.revision.fbe0e3f31233cc7846a2168c613452a1d7147cc2.mobile.android-api-9-dbg'
2018-05-30T15:26:39: DEBUG : using alternate debug route 'gecko.v2.mozilla-central.revision.0c712d76d598ec92a7adaaf7180fd4e559f9abc0.mobile.android-api-9-dbg'
2018-05-30T15:26:45: WARNING : Skipping build fbe0e3f31233: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.fbe0e3f31233cc7846a2168c613452a1d7147cc2.mobile.android-api-9-debug'
2018-05-30T15:26:45: WARNING : Skipping build 0c712d76d598: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.0c712d76d598ec92a7adaaf7180fd4e559f9abc0.mobile.android-api-9-debug'
2018-05-30T15:26:45: WARNING : Skipping build c3cd8c58fa87: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.c3cd8c58fa87d571874f2bcc34ec0864ac7acd5f.mobile.android-api-9-debug'
2018-05-30T15:26:45: DEBUG : using taskcluster route 'gecko.v2.mozilla-central.revision.286f71223256cbb3a769432fd860f563c4886e81.mobile.android-api-9-debug'
2018-05-30T15:26:45: DEBUG : using taskcluster route 'gecko.v2.mozilla-central.revision.ae492e1619671ca2b5f55369b70d97c7da67dd72.mobile.android-api-9-debug'
2018-05-30T15:26:45: DEBUG : using taskcluster route 'gecko.v2.mozilla-central.revision.692e277e2b9f3bb36d6d67e1166d135228f2d851.mobile.android-api-9-debug'
2018-05-30T15:26:52: DEBUG : using alternate debug route 'gecko.v2.mozilla-central.revision.286f71223256cbb3a769432fd860f563c4886e81.mobile.android-api-9-dbg'
2018-05-30T15:26:52: DEBUG : using alternate debug route 'gecko.v2.mozilla-central.revision.692e277e2b9f3bb36d6d67e1166d135228f2d851.mobile.android-api-9-dbg'
2018-05-30T15:26:52: DEBUG : using alternate debug route 'gecko.v2.mozilla-central.revision.ae492e1619671ca2b5f55369b70d97c7da67dd72.mobile.android-api-9-dbg'
2018-05-30T15:26:58: WARNING : Skipping build 286f71223256: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.286f71223256cbb3a769432fd860f563c4886e81.mobile.android-api-9-debug'
2018-05-30T15:26:58: WARNING : Skipping build ae492e161967: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.ae492e1619671ca2b5f55369b70d97c7da67dd72.mobile.android-api-9-debug'
2018-05-30T15:26:59: WARNING : Skipping build 692e277e2b9f: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.692e277e2b9f3bb36d6d67e1166d135228f2d851.mobile.android-api-9-debug'
2018-05-30T15:26:59: DEBUG : using taskcluster route 'gecko.v2.mozilla-central.revision.268cfff98df1cc643df7f79dab467043fd1837f1.mobile.android-api-9-debug'
2018-05-30T15:26:59: DEBUG : using taskcluster route 'gecko.v2.mozilla-central.revision.e15fb840661215dc3928c3bf1fea0ca0379511f9.mobile.android-api-9-debug'
2018-05-30T15:26:59: DEBUG : using taskcluster route 'gecko.v2.mozilla-central.revision.39d5cc0fda5e16c49a59d29d4ca186a5534cc88b.mobile.android-api-9-debug'
2018-05-30T15:27:05: DEBUG : using alternate debug route 'gecko.v2.mozilla-central.revision.39d5cc0fda5e16c49a59d29d4ca186a5534cc88b.mobile.android-api-9-dbg'
2018-05-30T15:27:06: DEBUG : using alternate debug route 'gecko.v2.mozilla-central.revision.268cfff98df1cc643df7f79dab467043fd1837f1.mobile.android-api-9-dbg'
2018-05-30T15:27:06: DEBUG : using alternate debug route 'gecko.v2.mozilla-central.revision.e15fb840661215dc3928c3bf1fea0ca0379511f9.mobile.android-api-9-dbg'
2018-05-30T15:27:11: WARNING : Skipping build 39d5cc0fda5e: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.39d5cc0fda5e16c49a59d29d4ca186a5534cc88b.mobile.android-api-9-debug'
2018-05-30T15:27:12: WARNING : Skipping build 268cfff98df1: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.268cfff98df1cc643df7f79dab467043fd1837f1.mobile.android-api-9-debug'
2018-05-30T15:27:13: WARNING : Skipping build e15fb8406612: Unable to find build info using the taskcluster route 'gecko.v2.mozilla-central.revision.e15fb840661215dc3928c3bf1fea0ca0379511f9.mobile.android-api-9-debug'
2018-05-30T15:29:33: INFO : Stopped


I guess, I need to use "git bisect" now to be able to find the commit, that introduced the regression, right?
https://wiki.mozilla.org/Firefox_OS/Performance/Bisecting_Regressions

https://github.com/mozilla/gecko-dev/commits/master@%7B2015-12-04T03:02:08Z%7D/mobile

My current idea is:
git bisect start c40454e094856a75a4bc33f1683bce1efeb59751 5c14ea4c04f88d5e5d5a2c15ca7252649f0e3569
and compile and run in each step
https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Simple_Firefox_for_Android_build

Or should I rather use mercurial instead?

Comment 4

[Djfe]

Oh, and I tried fennec and fennec-2.3 (mozilla-central) and neither worked.

In-case you still want the config, I used:
https://i.imgur.com/M0dju9r.png
https://i.imgur.com/q2SpBZS.png
https://i.imgur.com/GAFAh7M.png

Comment 5

[William Lachance (:wlach)]

(In reply to Djfe from comment #3)
> Hi William,
> thanks for replying :)
> 
> I rebooted and now I can't reproduce the following error any longer:
> traceback:   File ".\mozregui\bisection.py", line 82, in check_merge
>   File "..\mozregression\bisector.py", line 263, in handle_merge
>   File "..\mozregression\build_range.py", line 94, in __getitem__
> 
> I think, I still used the same config as before.
> 
> It seems to work now, except that taskcluster doesn't have the nightlies any
> longer, but that is intended after all.
> ...

We should be able to bisect nightlies without using taskcluster. What happens if you don't specify `mozilla-central` as the branch name?
 
> I guess, I need to use "git bisect" now to be able to find the commit, that
> introduced the regression, right?
> https://wiki.mozilla.org/Firefox_OS/Performance/Bisecting_Regressions

That link is for firefox os, not firefox.

> https://github.com/mozilla/gecko-dev/commits/master@%7B2015-12-04T03:02:
> 08Z%7D/mobile
> 
> My current idea is:
> git bisect start c40454e094856a75a4bc33f1683bce1efeb59751
> 5c14ea4c04f88d5e5d5a2c15ca7252649f0e3569
> and compile and run in each step
> https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/
> Build_Instructions/Simple_Firefox_for_Android_build
> 
> Or should I rather use mercurial instead?

This is probably beyond my knowledge to really help out with. I would ask about this on #developers or #ateam (but maybe try to get mozregression working first)

Comment 6

[Djfe]

(In reply to William Lachance (:wlach) (use needinfo!) from comment #5)
> We should be able to bisect nightlies without using taskcluster. What
> happens if you don't specify `mozilla-central` as the branch name?

fennec-2.3
"Unable to find enough data to bisect."
https://pastebin.com/cFUVJFMA

fennec
works and downloads nightlies and runs them on my phone, but the wrong version: 55 instead of 45
https://pastebin.com/rBeLxqXn

> That link is for firefox os, not firefox.
I'm aware. But it explains git's bisect fairly well.

> This is probably beyond my knowledge to really help out with. I would ask
> about this on #developers or #ateam (but maybe try to get mozregression
> working first)

ok :)