1711366 - firefox-90 branch is broken in third_party/highway on armv7 + neon

Status: RESOLVED FIXED

(Core :: Graphics: ImageLib, defect)

Description

[tt_1]

Attachment: firefox-90-nightly-broken-arm.log.gz

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0

Steps to reproduce:

checked out latest firefox nightly sources and compiled them

Actual results:

the build fails in third_party/highway with numerous errors just like this:

19:53.10 /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-90.0/work/firefox-90.0/third_party/highway/hwy/ops/arm_neon-inl.h:876:326: error: no matching function for call to 'hwy::N_NEON::Mask128<float, 4>::Mask128(uint32x4_t)'
19:53.11 /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-90.0/work/firefox-90.0/third_party/highway/hwy/ops/arm_neon-inl.h:876:326: error: no matching function for call to 'hwy::N_NEON::Mask128<float, 2>::Mask128(uint32x2_t)'
19:53.15 /usr/armv7a-unknown-linux-gnueabihf/tmp/portage/www-client/firefox-90.0/work/firefox-90.0/third_party/highway/hwy/ops/arm_neon-inl.h:876:326: error: no matching function for call to 'hwy::N_NEON::Mask128<float, 1>::Mask128(uint32x2_t)'

the full build log is attached

Expected results:

I expect the compile to run through without issues

Comment 1

[tt_1]

upstream issue: https://github.com/google/highway/issues/192

Comment 2

[tt_1]

current workaround is to pass --disable-jxl during configure

Comment 3

[Kagami Rosylight [:saschanaz] (they/them)]

Hi :tt_1, could you check what happens if you run ./mach vendor highway and build again?

Comment 4

[tt_1]

I don't think this is needed, as upstream already acknowledged its a neon specific issue & that they will come with a patch for it soon.

In any case, I wonder what that would actually do? My package manager only uses ./mach configure, build and install

Comment 5

[Kagami Rosylight [:saschanaz] (they/them)]

Ah okay, never mind then, https://github.com/google/highway/issues/188#issuecomment-841791614 indeed says it's still an issue there.

FYI, ./mach vendor highway updates highway with the latest commit from the upstream.

Comment 6

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1707590

Comment 7

[tt_1]

I'm not very much familiar with the hg source code tree, but have checked out everything locally via this guide: https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/firefoxworkflow.html

however, ./mach vendor highway doesn't work for me: highway: Failed to find manifest: highway

I have some hope it has been fixed for armv7a-neon as of today in https://github.com/google/highway/commit/4a47570daf29bfff0577541f8a0c0c342bf363cd

maybe you can upload the split patch against the latest hg tree, if you're more familiar with the workflow?

Comment 8

[Kagami Rosylight [:saschanaz] (they/them)]

Oops, it's ./mach vendor media/highway/moz.yaml, I always confuse myself with this, sorry 😱

It should be straightforward if you already checked out, could you try the new command? Thanks!

Comment 9

[tt_1]

thats not working either:

./mach vendor media/highway/moz.yaml
Error running mach:

['vendor', 'media/highway/moz.yaml']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file vendor| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

FileNotFoundError: [Errno 2] No such file or directory: 'third_party/jpeg-xl/third_party/highway'

File "/home/firefox/python/mozbuild/mozbuild/vendor/mach_commands.py", line 75, in vendor
manifest = load_moz_yaml(library)
File "/home/firefox/python/mozbuild/mozbuild/vendor/moz_yaml.py", line 293, in load_moz_yaml
schema_additional(filename, manifest, require_license_file=require_license_file)
File "/home/firefox/python/mozbuild/mozbuild/vendor/moz_yaml.py", line 440, in _schema_1_additional
files = [f.lower() for f in os.listdir(vendor_directory)]

Comment 10

[Kagami Rosylight [:saschanaz] (they/them)]

Oh no! Could you open media/highway/moz.yaml, modify vendor-directory: third_party/jpeg-xl/third_party/highway into vendor-directory: third_party/highway, and try again? Sorry for your inconvenience, I forgot to update it when moving the directory.

Comment 11

[tt_1]

boi, this is cranky:

/mach vendor media/highway/moz.yaml
0:00.93 Glean initialized
0:01.06 No RLB symbol found. Not trying to flush the RLB dispatcher.
0:01.97 You have uncommitted changes to the following files:

media/highway/moz.yaml

Please commit or stash these changes before vendoring, or re-run with --ignore-modified.

0:01.97 You have uncommitted changes to the following files:

media/highway/moz.yaml

Please commit or stash these changes before vendoring, or re-run with --ignore-modified.

--> I decided to use --ignore-modified, rather than learning how to commit with hg

/home/firefox $ ./mach vendor media/highway/moz.yaml --ignore-modified
0:01.00 Latest commit is 4a47570daf29bfff0577541f8a0c0c342bf363cd from 2021-05-17T14:03:49Z
0:01.00 Fetching code archive from https://github.com/google/highway/archive/4a47570daf29bfff0577541f8a0c0c342bf363cd.tar.gz
0:01.69 rm -rf third_party/highway
0:01.69 Unpacking upstream files from third_party/highway.
0:01.72 Removing unnecessary files.
0:01.72 Removing: ['third_party/highway/g3doc/']
0:01.72 Updating moz.yaml.
Found release: commit ca1a57c342cd815053abfcffa29b44eaead4f20b (2021-04-15T17:54:35Z).

Found revision: ca1a57c342cd815053abfcffa29b44eaead4f20b

0:01.72 Updating files
0:01.72 Registering changes with version control.
0:02.42 Updating moz.build files
0:04.42 Could not add third_party/highway/hwy/contrib/image/image.cc to the appropriate moz.build file
0:04.42 Could not add third_party/highway/hwy/contrib/image/image.h to the appropriate moz.build file
0:04.42 Could not add third_party/highway/hwy/contrib/image/image_test.cc to the appropriate moz.build file
0:04.43 Could not add third_party/highway/hwy/contrib/math/math-inl.h to the appropriate moz.build file
0:04.43 Could not add third_party/highway/hwy/contrib/math/math_test.cc to the appropriate moz.build file
0:04.43 Could not add third_party/highway/hwy/ops/arm_sve-inl.h to the appropriate moz.build file
0:04.43 Could not remove third_party/highway/Makefile from the appropriate moz.build file
0:04.43 Could not remove third_party/highway/contrib/image/image.cc from the appropriate moz.build file
0:04.43 Could not remove third_party/highway/contrib/image/image.h from the appropriate moz.build file
0:04.43 Could not remove third_party/highway/contrib/image/image_test.cc from the appropriate moz.build file
0:04.43 Could not remove third_party/highway/contrib/math/math-inl.h from the appropriate moz.build file
0:04.43 Could not remove third_party/highway/contrib/math/math_test.cc from the appropriate moz.build file
0:04.43 This is a deficiency in ./mach vendor and should be reported to the Updatebot maintainers.

am I done now?

Comment 12

[Kagami Rosylight [:saschanaz] (they/them)]

That doesn't seem good, I'll take a look myself and upload the patch when it's done, so that you can confirm that it builds on your machine. Thanks and again sorry for the inconvenience.

Comment 13

[tt_1]

this commit has fixes for armv7 and aarch64 added: https://github.com/google/highway/commit/376a400463f7e79e2e79e34ad67225e9397df54f

standalone with cmake does now work for both of them, so I'd suggest to pull in an update from this commit, or a more recent head of git if available at the moment you make the decision.

Comment 14

[tt_1]

version 0.12.1 is in the making, it will most likely be released soon: https://github.com/google/highway/issues/192#issuecomment-843027995

Comment 15

[Rares Doghi, Desktop QA]

I think this issue belongs to the Firefox Build System > General or maybe the Core > Imagelib ? since its related to the initial implementation of JXL

Comment 16

[Kagami Rosylight [:saschanaz] (they/them)]

Attachment: Bug 1711366 - Update highway to 0.12.1 r=aosmond

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d5ae3221209c
Update highway to 0.12.1 r=aosmond

Comment 18

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/d5ae3221209c

Comment 20

[Dan Horák]

confirming that at least ppc64le builds again, s390x fails for with an unrelated issue

Comment 21

[Kagami Rosylight [:saschanaz] (they/them)]

s390x fails for with an unrelated issue

Could you report that as a separate bug? Thanks!

Comment 22

[Dan Horák]

(In reply to Kagami :saschanaz from comment #21)

s390x fails for with an unrelated issue

Could you report that as a separate bug? Thanks!

Sure, I haven't finished my investigation yet :-) Looks as something NSPR related so far ...