1633873 - android-fat-aar-artifact target without --enable-application=mobile/android should fail with helpful error

Status: NEW

(GeckoView :: General, enhancement, P5)

Description

[Georg Koppen]

I've been trying to create fat .aar files locally to pick up changes for Tor Browser and be able later on to base the mobile Tor Browser on the upcoming Fenix.

It turns out this is broken and it is not obvious why that's the case. Here is what the documentation says about running this locally (https://searchfox.org/mozilla-central/source/taskcluster/docs/kinds.rst#23):

If you want to run this task locally, you need to specify these environment variable:

• MOZ_ANDROID_FAT_AAR_ARCHITECTURES: must be a comma-separated list of architecture.
  Eg: "armeabi-v7a,arm64-v8a,x86,x86_64".
• each of MOZ_ANDROID_FAT_AAR_ARM64_V8A, MOZ_ANDROID_FAT_AAR_ARMEABI_V7A,
  MOZ_ANDROID_FAT_AAR_X86, MOZ_ANDROID_FAT_AAR_X86_64 must be a path relative to
  MOZ_FETCHES_DIR.

I tried that both with our own arch-dependent .aar files and the ones Mozilla creates but both fail. To repro take just two .aar files and do

export MOZ_ANDROID_FAT_AAR_ARCHITECTURES=arm64-v8a,x86
export MOZ_ANDROID_FAT_AAR_ARM64_V8A=*arm64-v8a*.aar
export MOZ_ANDROID_FAT_AAR_X86=*x86-*.aar
export MOZ_FETCHES_DIR=/path/to/your/dir/with/the/aar/files

and then (assuming you have all that done in a mozilla repo):

./mach configure --disable-compile-environment
./mach build android-fat-aar-artifact

The result is something like

"Disallowed: Inputs missing expected architecture-specific input: arm64-v8a/greprefs.js"

So, jlorenzo helped me debugging that today (thanks again!) and it turns out while missing_arch_prefs (https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/action/fat_aar.py#51) is {'defaults/pref/arm64-v8a/geckoview-prefs.js', 'arm64-v8a/greprefs.js'} path (https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/action/fat_aar.py#54) for greprefs.js and geckoview-prefs.js is assets/arm64-v8a/greprefs.js and assets/defaults/pref/arm64-v8a/geckoview-prefs.js respectively. Note the additional "assets" which the UnpackFinder gives back here and which makes sure the path is not discarded from the missing_arch_prefs (https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/action/fat_aar.py#79).

A try build for creating the fat .aar file, however, is shown that on CI there is no additional "assets" in the path (see the log of one of my try builds: https://firefoxci.taskcluster-artifacts.net/c-RHgbOgS5OHGYjsgIxskQ/0/public/logs/live_backing.log with "FAT AAR DEBUG" statements)

It's not obvious which of the additional settings in the try build is causing the omission of the "assets" path part by the UnpackFinder and making the build successful, but either way I think it's kind of non-intuitive to run into such path issues when extracting omni.ja files and thus worth fixing.

Comment 1

[Nick Alexander :nalexander [he/him]]

I haven't been able to track this all the way down, but something is rotten in the state of Denmark: there's a Py2/Py3 incompatibility that's crept in. (Also, Py3 is unbelievably slow copying these massive files, so slow I can't be bothered to let it finish, but one thing at a time here.)

Py3:

 0:05.73 /usr/bin/make -j12 android-fat-aar-artifact
 0:05.75 /Applications/Xcode.app/Contents/Developer/usr/bin/make recurse_android-fat-aar-artifact
 0:05.76 /Users/nalexander/Mozilla/objdirs/objdir-droid/_virtualenvs/init_py3/bin/python -m mozbuild.action.fat_aar  --armeabi-v7a /Users/nalexander/Mozilla/gecko/fetches/geckoview-default-armeabi-v7a-77.0.20200428145439.aar --arm64-v8a /Users/nalexander/Mozilla/gecko/fetches/geckoview-default-arm64-v8a-77.0.20200428145439.aar --distdir /Users/nalexander/Mozilla/objdirs/objdir-droid/dist/fat-aar
 0:07.45 Allowed: Path "AndroidManifest.xml" has architecture-specific versions:
 0:07.45   arm64-v8a -> 273f09c445030d940cd2668423dfc0e28f04d592
 0:07.45   armeabi-v7a -> 77cc220a6b32c3dedbb4b62b1bd80c235836d444
 0:07.45 Allowed: Path "chrome/toolkit/content/global/buildconfig.html" has architecture-specific versions:
 0:07.45   arm64-v8a -> d896ecc75d28142ca9839539eafc91c7f6599fd3
 0:07.45   armeabi-v7a -> d78a39550dc90dc7f6fd15f763cb43e19a1a1d6f
 0:07.45 Allowed: Path "classes.jar!/org/mozilla/geckoview/BuildConfig.class" has architecture-specific versions:
 0:07.45   arm64-v8a -> 08e11811af0eef75fdbad18312dde89bfd8f6c87
 0:07.45   armeabi-v7a -> d2e1c0c5986a1a8aff7a4d2c94d163ce0fa7606c

Py2:

nalexander@roboto ~/M/gecko> ./mach python -m mozbuild.action.fat_aar  --armeabi-v7a /Users/nalexander/Mozilla/gecko/fetches/geckoview-default-armeabi-v7a-77.0.20200428145439.aar --arm64-v8a /Users/nalexander/Mozilla/gecko/fetches/geckoview-default-arm64-v8a-77.0.20200428145439.aar --distdir /Users/nalexander/Mozilla/objdirs/objdir-droid/dist/fat-aar
New python executable in /Users/nalexander/Mozilla/gecko/obj-x86_64-apple-darwin18.7.0/_virtualenvs/init/bin/python2.7
Also creating executable in /Users/nalexander/Mozilla/gecko/obj-x86_64-apple-darwin18.7.0/_virtualenvs/init/bin/python
Installing setuptools, pip, wheel...
done.
running build_ext
copying build/lib.macosx-10.14-x86_64-2.7/psutil/_psutil_osx.so -> psutil
copying build/lib.macosx-10.14-x86_64-2.7/psutil/_psutil_posix.so -> psutil

Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
Allowed: Path "AndroidManifest.xml" has architecture-specific versions:
  arm64-v8a -> 273f09c445030d940cd2668423dfc0e28f04d592
  armeabi-v7a -> 77cc220a6b32c3dedbb4b62b1bd80c235836d444
Allowed: Path "classes.jar!/org/mozilla/geckoview/BuildConfig.class" has architecture-specific versions:
  arm64-v8a -> 08e11811af0eef75fdbad18312dde89bfd8f6c87
  armeabi-v7a -> d2e1c0c5986a1a8aff7a4d2c94d163ce0fa7606c
Disallowed: Path "assets/chrome/toolkit/content/global/buildconfig.html" has architecture-specific versions:
  arm64-v8a -> d896ecc75d28142ca9839539eafc91c7f6599fd3
  armeabi-v7a -> d78a39550dc90dc7f6fd15f763cb43e19a1a1d6f
Disallowed: Inputs missing expected architecture-specific input: arm64-v8a/greprefs.js
Disallowed: Inputs missing expected architecture-specific input: armeabi-v7a/greprefs.js
Disallowed: Inputs missing expected architecture-specific input: defaults/pref/arm64-v8a/geckoview-prefs.js
Disallowed: Inputs missing expected architecture-specific input: defaults/pref/armeabi-v7a/geckoview-prefs.js

Why this succeeds in automation I don't understand, but we need to dig into whatever changed to get to the bottom of this.

Comment 2

[Johan Lorenzo [:jlorenzo]]

Thank you for finding this out, Nick! I think it actually explains what happens in automation \o/

If I understand this line[1] correctly:

[task 2020-04-29T03:40:31.398Z] 03:40:31     INFO -  /builds/worker/workspace/obj-build/_virtualenvs/init_py3/bin/python -m mozbuild.action.fat_aar  --armeabi-v7a /builds/worker/fetches/geckoview-default-armeabi-v7a-77.0.20200429030514.aar --arm64-v8a /builds/worker/fetches/geckoview-default-arm64-v8a-77.0.20200429030514.aar --x86 /builds/worker/fetches/geckoview-default-x86-77.0.20200429030514.aar --x86-64 /builds/worker/fetches/geckoview-default-x86_64-77.0.20200429030514.aar --distdir /builds/worker/workspace/obj-build/dist/fat-aar

automation is running Python 3, per init_py3. To me, that explains why it's working there.

[1] https://firefox-ci-tc.services.mozilla.com/tasks/cP9-YfJZSVqkFtCqtoTJKQ/runs/0/logs/https%3A%2F%2Ffirefox-ci-tc.services.mozilla.com%2Fapi%2Fqueue%2Fv1%2Ftask%2FcP9-YfJZSVqkFtCqtoTJKQ%2Fruns%2F0%2Fartifacts%2Fpublic%2Flogs%2Flive.log#L1392

Comment 3

[Georg Koppen]

(In reply to Johan Lorenzo [:jlorenzo] from comment #2)

Thank you for finding this out, Nick! I think it actually explains what happens in automation \o/

If I understand this line[1] correctly:

[task 2020-04-29T03:40:31.398Z] 03:40:31     INFO -  /builds/worker/workspace/obj-build/_virtualenvs/init_py3/bin/python -m mozbuild.action.fat_aar  --armeabi-v7a /builds/worker/fetches/geckoview-default-armeabi-v7a-77.0.20200429030514.aar --arm64-v8a /builds/worker/fetches/geckoview-default-arm64-v8a-77.0.20200429030514.aar --x86 /builds/worker/fetches/geckoview-default-x86-77.0.20200429030514.aar --x86-64 /builds/worker/fetches/geckoview-default-x86_64-77.0.20200429030514.aar --distdir /builds/worker/workspace/obj-build/dist/fat-aar

automation is running Python 3, per init_py3. To me, that explains why it's working there.

That's not the full story at least. The following is the result after configuring with --disable-compile-environment and is using Python 3, too:

/usr/bin/make -j4 android-fat-aar-artifact
/usr/bin/make recurse_android-fat-aar-artifact
/home/thomas/Arbeit/Tor/tor-browser/obj-x86_64-pc-linux-gnu/_virtualenvs/init_py3/bin/python -m mozbuild.action.fat_aar  --armeabi-v7a //home/thomas/Arbeit/Tor/tor-browser/artifacts/geckoview-nightly-armeabi-v7a-76.0.20200427010101.aar --arm64-v8a //home/thomas/Arbeit/Tor/tor-browser/artifacts/geckoview-nightly-arm64-v8a-76.0.20200427010101.aar   --distdir /home/thomas/Arbeit/Tor/tor-browser/obj-x86_64-pc-linux-gnu/dist/fat-aar
Allowed: Path "AndroidManifest.xml" has architecture-specific versions:
  arm64-v8a -> 71ba81e1e30418fc9b4bc82840fc7f5e437f07f3
  armeabi-v7a -> c5589f4eab042c16278f7f30c6194f995188ed07
Allowed: Path "classes.jar!/org/mozilla/gecko/util/HardwareUtils.class" has architecture-specific versions:
  arm64-v8a -> b20b005c5b7c473aff24ce4747f4589e6fd69698
  armeabi-v7a -> c045eae4f64a96575cfcbdf5cfdcc27602371aa5
Allowed: Path "classes.jar!/org/mozilla/geckoview/BuildConfig.class" has architecture-specific versions:
  arm64-v8a -> e99b3174e3568c0abba3923d8cb53cbbad663a75
  armeabi-v7a -> 33adde90a37b34cc42f12aba817e859c6de7e2a9
Disallowed: Path "assets/chrome/toolkit/content/global/buildconfig.html" has architecture-specific versions:
  arm64-v8a -> 720d00a66db1444969c6c97e78cc02ca4c0bd1e9
  armeabi-v7a -> 42a3370aa7efca138a544a677ffd4567f5c4a4a9
Disallowed: Inputs missing expected architecture-specific input: arm64-v8a/greprefs.js
Disallowed: Inputs missing expected architecture-specific input: armeabi-v7a/greprefs.js
Disallowed: Inputs missing expected architecture-specific input: defaults/pref/arm64-v8a/geckoview-prefs.js
Disallowed: Inputs missing expected architecture-specific input: defaults/pref/armeabi-v7a/geckoview-prefs.js

jlorenzo and I looked a bit more at this issue today and we both can get passed that bug if we use ac_add_options --enable-application=mobile/android and then ./mach configure and ./mach build android-fat-aar-artifact, all other things being still the same.

Comment 4

[Georg Koppen]

(In reply to Georg Koppen from comment #3)

(In reply to Johan Lorenzo [:jlorenzo] from comment #2)

Thank you for finding this out, Nick! I think it actually explains what happens in automation \o/

If I understand this line[1] correctly:

[task 2020-04-29T03:40:31.398Z] 03:40:31     INFO -  /builds/worker/workspace/obj-build/_virtualenvs/init_py3/bin/python -m mozbuild.action.fat_aar  --armeabi-v7a /builds/worker/fetches/geckoview-default-armeabi-v7a-77.0.20200429030514.aar --arm64-v8a /builds/worker/fetches/geckoview-default-arm64-v8a-77.0.20200429030514.aar --x86 /builds/worker/fetches/geckoview-default-x86-77.0.20200429030514.aar --x86-64 /builds/worker/fetches/geckoview-default-x86_64-77.0.20200429030514.aar --distdir /builds/worker/workspace/obj-build/dist/fat-aar

automation is running Python 3, per init_py3. To me, that explains why it's working there.

That's not the full story at least. The following is the result after configuring with --disable-compile-environment and is using Python 3, too:

Just to make the point clear, the problem is not --disable-compile-environment but the lack of --enable-application=mobile/android. With the latter added the android-fat-aar-artifact target succeeds. I'll update the summary again.

Comment 5

[Nick Alexander :nalexander [he/him]]

(In reply to Georg Koppen from comment #4)

(In reply to Georg Koppen from comment #3)

(In reply to Johan Lorenzo [:jlorenzo] from comment #2)

Thank you for finding this out, Nick! I think it actually explains what happens in automation \o/

If I understand this line[1] correctly:

[task 2020-04-29T03:40:31.398Z] 03:40:31     INFO -  /builds/worker/workspace/obj-build/_virtualenvs/init_py3/bin/python -m mozbuild.action.fat_aar  --armeabi-v7a /builds/worker/fetches/geckoview-default-armeabi-v7a-77.0.20200429030514.aar --arm64-v8a /builds/worker/fetches/geckoview-default-arm64-v8a-77.0.20200429030514.aar --x86 /builds/worker/fetches/geckoview-default-x86-77.0.20200429030514.aar --x86-64 /builds/worker/fetches/geckoview-default-x86_64-77.0.20200429030514.aar --distdir /builds/worker/workspace/obj-build/dist/fat-aar

automation is running Python 3, per init_py3. To me, that explains why it's working there.

That's not the full story at least. The following is the result after configuring with --disable-compile-environment and is using Python 3, too:

Just to make the point clear, the problem is not --disable-compile-environment but the lack of --enable-application=mobile/android. With the latter added the android-fat-aar-artifact target succeeds. I'll update the summary again.

Ah, I see. Producing a fat AAR is, indeed, a type of build -- we use the Gradle build machinery to assemble the AAR, after all -- and therefore the application must be correct. But it's misleading that one can even try with the wrong application. I'd take a patch to have the fat_aar.py script bail if the build configuration is nonsensical, or to not define the relevant Makefile targets unless the application is correct. And I guess the docs could mention that requirement.

Otherwise it seems like the functionality is working as intended.

Comment 6

[James Willcox (:snorp) ([e-mail]) (he/him)]

Nick, do we have any work to do here? It sounds like this is resolved by adding --enable-application, which seems reasonable to me.

Comment 7

[Nick Alexander :nalexander [he/him]]

(In reply to James Willcox (:snorp) ([e-mail]) (he/him) from comment #6)

Nick, do we have any work to do here? It sounds like this is resolved by adding --enable-application, which seems reasonable to me.

I'd take any of the patches I suggest in #c5, but nothing sounds urgent.