1536013 - mozregression does not have an option to test aarch64 builds on Qualcomm processor laptops

Status: VERIFIED FIXED

(Testing :: mozregression, defect)

Description

[Daniel Bodea [:danibodea]]

Note

• When we find an issue specific to the aarch64 build and we attempt to find its regressor, we noticed that mozregression will only get 32-bit (non-aarch64) builds instead of getting the aarch64 (Qualcomm processor specific) builds.
• The terminal version cannot be installed due to its Qualcomm architecture.

Affected versions

• any

Affected platforms

• running on Lenovo Yoga C630 (Qualcomm processor laptop)

Steps to reproduce

1. Lunch mozregression on any range on a Qualcomm processor laptop.

Expected result

• aarch64 build versions should run.

Actual result

• 32-bit version builds are run.

Additional notes

• ARM64/aarch64 specific issues CANNOT be regressed.

Comment 1

[Ian Moody [:Kwan] (UTC+1)]

Hey Bodea, I have a preliminary pull request attempting to add this, but I don't have a Windows aarch64 device to test it on. Would you be willing to?
There is a build available from the CI (direct exe link)

Although, I'm a little puzzled/concerned why you're getting 32bit builds rather than 64bit. As far as I can tell it should try to download 64bit builds (though I'm far from certain, I'm still new to this area of the code). Perhaps mozregression is running in a way (32-bit mode?) that makes Python think it is running on 32bit arm hardware rather than aarch64? If that's the case this won't fix it.

An interesting test would be to, if possible, hop into the python REPL on the command line and try

import platform
platform.uname()

and seeing what that returns.

Comment 2

[Daniel Bodea [:danibodea]]

I have performed the requested action on my test machine and these are my results:
"C:\Users\svuser>python
Python 3.7.2 (tags/v3.7.2:9a3ffc0492, Dec 23 2018, 22:20:52) [MSC v.1916 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.

import platform
platform.uname()
uname_result(system='Windows', node='SVROLP00304', release='10', version='10.0.17134', machine='ARM64', processor='ARMv8 (64-bit) Family 8 Model 803 Revision 70C, Qualcomm Technologies Inc')"

I have to mention that this issue is ARM64 specific; it occurs only on devices with Qualcomm processors like Lenovo Yoga C630. These laptops normally use 32-bit applications so a specific Firefox build was created for them (aarch64 builds). This is an exception from the rule, it should get arch64 builds in instead of the 32-bit normal build (or even the 64-bit normal build).

Both the old GUI app and the new one you post in comment 1 have the only choice of 32 Bits in the Basic Configuration screen and 32-bit version builds are being opened by them.

In conclusion, the issue is not fixed. Feel free to NI me if more testing is needed.

Thank you for your contribution!

Comment 3

[Ian Moody [:Kwan] (UTC+1)]

Ah, so processor is ARMv8, not AArch64. Hmm, it might be that mozinfo should be updated to handle this first, so it returns something friendlier.

Could you do please another experiment in the python REPL? Will have to install mozinfo via pip to do some of it if it isn't already available.

import platform
platform.architecture()
import mozinfo
mozinfo.bits
mozinfo.processor

Comment 4

[Ian Moody [:Kwan] (UTC+1)]

Oh, and actually, as a sanity check, could you please run those tests in python2 as well? And re-run the platform.uname() one in it? mozregression is still a python2 app for now, so best to check there aren't different results.

Comment 5

[Daniel Bodea [:danibodea]]

I cannot install Python 2 on Lenovo Yoga because it does not have a 32-bit version (or I could not find one). It requires the 32-bit version.

Comment 6

[Daniel Bodea [:danibodea]]

Information displayed when running mozinfo on Lenovo Yoga:
"
C:\Users\svuser>mozinfo
os: win
processor: ARM64
version: 10.0.17134
os_version: 10.0
bits: 32
has_sandbox: True
webrender: False
automation: False
service_pack:
"

Information displayed when the commands in comment 3 are being ran:
"
>>> import platform
>>> platform.architecture()
('32bit', 'WindowsPE')
>>> import mozinfo
>>> mozinfo.bits
32
>>> mozinfo.processor
'ARM64'
"

Comment 7

[Ian Moody [:Kwan] (UTC+1)]

(In reply to Bodea Daniel [:danibodea] from comment #5)

I cannot install Python 2 on Lenovo Yoga because it does not have a 32-bit version (or I could not find one).

This page seems to include a download for a 32 bit msi installer for Python2.7, though it's probably not necessary to check now.

(In reply to Bodea Daniel [:danibodea] from comment #6)

mozinfo.bits
32
mozinfo.processor
'ARM64'

Okay, so this is what I was afraid of, bits being 32. I need to think about this a little more.

Okay, it might not be as bad as I feared. Could you try this new build please? (direct exe link) This would hopefully be better than my previous approach, since this will let you still do win32 builds, whereas before I was just locking it to ARM64 only.

Comment 8

[Daniel Bodea [:danibodea]]

Firstly, I managed to install Python 2 on Lenovo Yoga with the link provided. For some reason, the last time I tried, I remember an error about the architecture was displayed. Must have been some human error of mine.

Then I ran the commands in comment 1 and comment 3 to see if anythng differs from Python 3 to Python 2. Fortunately it seems to display the same information:
"
C:\Users\svuser>python
Python 2.7.16 (v2.7.16:413a49145e, Mar 4 2019, 01:30:55) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.architecture()
('32bit', 'WindowsPE')
>>> import mozinfo
>>> mozinfo.bits
32
>>> mozinfo.processor
'ARM64'
>>> platform.uname()
('Windows', 'SVROLP00304', '10', '10.0.17134', 'ARM64', 'ARMv8 (64-bit) Family 8 Model 803 Revision 70C, Qualcomm Technologies Inc')
"

Furthermore, I installed the terminal version of mozregression to see if it works any differently, but it did not. The 32-bit builds were launched also.

Lastly, I installed the new build from comment 7. Using this GUI mozregression, it appears that now, I can choose to get 64-bit apps, but an error is displayed when the build is launched:
"An error occurred while startingthe process, so the build will be skipped. Error message: Unable to start the application."
I suspect that the non-aarch64 build is being fetched, instead of the aarch64 build.

It appears that the issue is still not fixed with this build.

Comment 9

[Ian Moody [:Kwan] (UTC+1)]

Hmm, it might be that I've only changed how it fetches inbound builds from taskcluster, but not packaged nightlies. Can you please try running a single build with the branch set to mozilla-inbound and picking a single changeset, for instance d15d511d7fed?

Comment 10

[Ian Moody [:Kwan] (UTC+1)]

Hah, hilariously, I think I just discovered that mozregressing jsshell on Window x86-64 has been broken since we started producing aarch64 builds.

Comment 11

[Ian Moody [:Kwan] (UTC+1)]

New build, hopefully now also working for nightlies (direct exe link)

If I hack-edit mozinfo in my Windows VM to lie and say I'm running a 32 bit Python on ARM64, mozregression-gui now downloads a folder ending in -aarch64, and complains about being "Unable to start the application", so fingers crossed :D

Comment 12

[Daniel Bodea [:danibodea]]

It appears that this last build will download and launch aarch64 builds.

I have attempted to find a regression for an aarch64 specific issue, but the process could not be finished because of the impossibility of determining whether some builds are good or bad. Could not confirm whether it bisects correctly, but so far, it appears to be operating correctly, launching the correct builds and most importantly, the correct build type.
I could not find any major issue relating the functionality of mozregression on Qualcomm processor device.

I will consider this issue fixed with this last mozregression build. If further issues are being observed, new issues will be logged. Awaiting the correct bug status.

Thank you!

Comment 13

[Ian Moody [:Kwan] (UTC+1)]

Hey again, just a couple more tests I'd like you to run please before I merge this, just to make sure it's doing the right thing under the actual hardware (I did my hack test again and got the right builds, but I need to confirm mozinfo is actually returning the right value to mozregression on ARM64)

So if you could download this build (direct exe link) and run two tests:

1. Run a single build with bits set to 64, and pick a date at the start of March, say the 1st, so you don't have to worry about shippable
2. Run a single build with bits set to 32, again from say 2nd March.

Hopefully they both run and are the expected architecture. Another way of confirming is checking the file names in %USerProfile%\.mozilla\mozregression\persist, that the last two downloaded end with win64-aarch64.zip and win32.zip.

Thank you so much for all this testing and debugging, you've been incredibly helpful!

Comment 14

[Ian Moody [:Kwan] (UTC+1)]

Attachment: [GitHub PR] Bug 1536013 - Select win64-aarch64 builds when appropriate.

Comment 15

[Daniel Bodea [:danibodea]]

Hello!
I have verified it by running single builds as requested in comment 13. Furthermore, I have confirmed that the correct build architecture was downloaded/ran by opening the "about:buildconfig" page and confirming that the target is the appropriate one and by going to the folder where the builds are being downloaded to confirm they have the right name.
In conclusion, mozregression GUI does work as expected; downloads the aarch64 build type when selecting 64bit and the non-aarch64 32bt build when selecting as such, on a Qualcomm architecture device like Lenovo Yoga C630.

Can you answer some questions about usability, please?

1. How do I know I download the right mozregression GUI app version? Where should I get the right build from?
2. Will we be able to use the mozregression terminal version for Qualcomm devices?

Thank you too!

Comment 16

[Ian Moody [:Kwan] (UTC+1)]

(In reply to Bodea Daniel [:danibodea] from comment #15)

Hello!
I have verified it by running single builds as requested in comment 13. Furthermore, I have confirmed that the correct build architecture was downloaded/ran by opening the "about:buildconfig" page and confirming that the target is the appropriate one and by going to the folder where the builds are being downloaded to confirm they have the right name.
In conclusion, mozregression GUI does work as expected; downloads the aarch64 build type when selecting 64bit and the non-aarch64 32bt build when selecting as such, on a Qualcomm architecture device like Lenovo Yoga C630.

Excellent, thank you again! I'll merge the changes into master then.

(In reply to Bodea Daniel [:danibodea] from comment #15)

Can you answer some questions about usability, please?

1. How do I know I download the right mozregression GUI app version? Where should I get the right build from?
2. Will we be able to use the mozregression terminal version for Qualcomm devices?

Thank you too!

1. At the moment only the continuous integration (CI) version I linked to will support aarch64 (it's much like the autoland and mozilla-inbound builds mozregression tests against in not being an actual release yet). But hopefully within the next few days or a week or so there will be a new proper release that'll have both this and bug 1532412 in to fix bisecting around the opt->pgo->shippable build transition.
2. Yes the terminal version should work too. If it doesn't once it's released just let me know :)

In fact, if you need a more usable GUI build, bug 1532412 has its own CI version linked that also includes the fix for this, so should be practically usable until we can do a proper release.

Comment 17

[Ian Moody [:Kwan] (UTC+1)]

Released in CLI 2.3.40 and GUI 0.9.38

Comment 18

[Daniel Bodea [:danibodea]]

I have updated the terminal version of mozregression and verified that aarch64 builds are being downloaded and launched correctly as well as 32 bit build types. I will consider this bug fixed and verified. Thank you!