Publish a universal (fat) GV AAR with native libraries for multiple architectures

ASSIGNED
Assigned to

Status

enhancement
P1
normal
ASSIGNED
4 months ago
22 hours ago

People

(Reporter: nalexander, Assigned: nalexander, NeedInfo)

Tracking

(Depends on 2 bugs, Blocks 2 bugs)

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [geckoview:fenix:p2])

Attachments

(3 attachments, 1 obsolete attachment)

A big pain point for developers is with multiple artifacts per architecture, especially as GV apps add multiple dependencies with native code (e.g. Rust components). Publishing a fat AAR alongside the architecture dependent ones would help with this.

Bug 1508976 is the first part of this sequence: getting the initial Bgv (build geckoview fat AAR) job running in automation. This tracks getting a Ngv (Nightly build geckoview fat AAR) job running and getting the corresponding GV beetmover task working, so that we publish the fat AAR to maven.mozilla.org.

I'm not quite sure how to track the downstream dependencies for consuming this, but here's:

We should also disable a bunch more jobs:

  • Ngvs (there's nothing to sign here, no APKS are relevant)
  • SymNgv (the symbols have already been uploaded by the sub jobs)
  • UgsNgv (the sources have already been uploaded by the sub jobs)

I'll try to arrange this in the next iteration. Still worth reviewing (and bumping beetmoverscript!) in parallel.

FYI Nick: This patch still needs to be deployed once merged.

Flags: needinfo?(jlorenzo)
Attachment #9039088 - Flags: review?(mhentges)
Attachment #9039088 - Flags: review?(mhentges) → review+
See Also: → 1530757
Posted file GitHub Pull Request
Attachment #9047207 - Flags: review?(aki)
Attachment #9047207 - Flags: review?(aki) → review+

This is legit busted 'cuz it's hard to test: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7de0ab8c22167d0d5170edf3dbdf976ebf0ef232&selectedJob=230919822. I could force the Maven structure to not have -try but that feels questionable...

I'll push updated patches and maybe somebody from releng can get this across the line?

(In reply to Nick Alexander :nalexander [he/him] from comment #4)

We should also disable a bunch more jobs:

  • Ngvs (there's nothing to sign here, no APKS are relevant)

I think this is handled.

  • SymNgv (the symbols have already been uploaded by the sub jobs)

It's asserted that Nightly jobs have enable-full-crashsymbols in https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/transforms/upload_symbols.py#33, so I'm loathe to go against that.

  • UgsNgv (the sources have already been uploaded by the sub jobs)

I'll try to arrange this in the next iteration. Still worth reviewing (and bumping beetmoverscript!) in parallel.

I see no good way to avoid this: https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/transforms/upload_generated_sources.py and https://searchfox.org/mozilla-central/source/taskcluster/ci/upload-generated-sources/kind.yml don't have a flag. I guess we could use not-for-build-platform.

Thoughts, releng folks?

[geckoview:fenix:p2] because this is important for Fenix but technically not a release blocker.

Whiteboard: [geckoview:fenix:p1] → [geckoview:fenix:p2]

jlorenzo: making sure this doesn't drift too far down your queue.

Flags: needinfo?(jlorenzo)

Thank you for the reminder. It slipped my mind, yesterday :)

Flags: needinfo?(jlorenzo)

Comment 13

3 months ago
Pushed by nalexander@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c47b37ac1775
Publish GeckoView multi-architecture fat AAR Nightly. r=jlorenzo
https://hg.mozilla.org/integration/autoland/rev/7a6be593b0be
Post: Clean up Android TC artifacts. r=jlorenzo
Depends on: 1532737

Sadly, this ran into reality: greprefs.js is non-trivially different between armv7 and aarch64 now:

nalexander@roboto ~/M/gecko> diff -U3 /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/fat_aar/armeabi-v7a/omnijar/greprefs.js /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/fat_aar/arm64-v8a/omnijar/greprefs.js
--- /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/fat_aar/armeabi-v7a/omnijar/greprefs.js	2019-03-05 12:35:23.000000000 -0800
+++ /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/fat_aar/arm64-v8a/omnijar/greprefs.js	2019-03-05 12:35:24.000000000 -0800
@@ -1487,8 +1487,8 @@
 pref("javascript.options.baselinejit",      true);
 //Duplicated in JitOptions - ensure both match.
 pref("javascript.options.baselinejit.threshold", 10);
-//@line 1465 "$SRCDIR/modules/libpref/init/all.js"
-pref("javascript.options.ion",              true);
+//@line 1463 "$SRCDIR/modules/libpref/init/all.js"
+pref("javascript.options.ion",              false);
 //@line 1467 "$SRCDIR/modules/libpref/init/all.js"
 //Duplicated in JitOptions - ensure both match.
 pref("javascript.options.ion.threshold",    1000);

Such differences are only likely to increase as we invest more into tuning our Android engine settings across a range of target devices.

snorp: somehow someway we need to have some (all?) of these settings be per architecture. This could look like:

  1. moving greprefs to an architecture specific location in the omnijar and then gluing all the greprefs.js files into the fat AAR omnijar

  2. adding functions to greprefs.js files that are determined at runtime (i.e, isAndroidAarch64())

or ... there are lots of options. Initial thoughts?

snorp: NI for comment above ^

Flags: needinfo?(snorp)

Ughhhhhh, what a pain. I should've known we can't have nice things.

I guess my preference would be to have a completely separate greprefs.js for each arch in the fat omnijar. We'd need to then add some stuff to look for the arch-specific greprefs.js first.

Flags: needinfo?(snorp)
Depends on: 1533051
Attachment #9047269 - Attachment is obsolete: true

jlorenzo: time for re-review. Check out https://maven-default.stage.mozaws.net/?prefix=maven2/org/mozilla/geckoview/geckoview-nightly-try/ which looks good to me; see also my latest try build for more details: https://treeherder.mozilla.org/#/jobs?repo=try&revision=0da82a8f6e9cbf4c3b48b107ab845771118a3655. (It's possible that will fail due to code restructuring that shouldn't impact the outputs when issues are ironed out; I can link earlier try builds if you need those outputs.)

Flags: needinfo?(jlorenzo)

(In reply to Nick Alexander :nalexander [he/him] from comment #17)

jlorenzo: time for re-review. Check out https://maven-default.stage.mozaws.net/?prefix=maven2/org/mozilla/geckoview/geckoview-nightly-try/ which looks good to me; see also my latest try build for more details: https://treeherder.mozilla.org/#/jobs?repo=try&
revision=0da82a8f6e9cbf4c3b48b107ab845771118a3655

It did fail for trivial reasons but https://treeherder.mozilla.org/#/jobs?repo=try&revision=529bcfd26a9a653665d4c4dc83a4d5d2a4d81623 is green.

You need to log in before you can comment on or make changes to this bug.