1498715 - Use faster XZ compression preset for Android assets/ libraries for local developers

Status: RESOLVED WONTFIX

(Firefox Build System :: Android Studio and Gradle Integration, enhancement)

Description

[Nick Alexander :nalexander [he/him]]

XZ compression is slow.  SO SLOW!  Local developers XZ compress libxul.so every push to device (during |mach package|).  Let's do that faster when we're building locally.  In places where it matters (i.e., perf testing library unpacking and first run experiences), MOZILLA_OFFICIAL and --enable-release will still be needed (i.e., will still work).

This is follow-up to Bug 1418464, which avoided XZ compressing twice per |mach package|.

Comment 1

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1498715 - Use faster XZ compression preset for Android assets/ libraries for local developers. r?esawin

Before:

xz-compressing dist/fennec/assets/x86/libxul.so with "xz -zkf dist/fennec/assets/x86/libxul.so --threads=1 --x86 --lzma2=dict=8MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0"...
xz-compressing dist/fennec/assets/x86/libxul.so with "xz -zkf dist/fennec/assets/x86/libxul.so --threads=1 --x86 --lzma2=dict=8MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0"... DONE (69.13s)

$ ls -al dist/fennec/assets/x86/libxul.so
-rwxr-xr-x  1 nalexander  staff  31286204 12 Oct 14:06 dist/fennec/assets/x86/libxul.so

After:

xz-compressing dist/fennec/assets/x86/libxul.so with "xz -zkf dist/fennec/assets/x86/libxul.so --threads=1 --x86 --lzma2=dict=8MiB,lc=3,lp=0,pb=2,mode=fast,nice=273,mf=hc4,depth=24"...
xz-compressing dist/fennec/assets/x86/libxul.so with "xz -zkf dist/fennec/assets/x86/libxul.so --threads=1 --x86 --lzma2=dict=8MiB,lc=3,lp=0,pb=2,mode=fast,nice=273,mf=hc4,depth=24"... DONE (19.52s)

$ ls -al dist/fennec/assets/x86/libxul.so
-rwxr-xr-x  1 nalexander  staff  34877984 12 Oct 14:00 dist/fennec/assets/x86/libxul.so

All timings on a late 2013 MacBook Pro (i.e., a very slow device).

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f9cf33c52739
Use faster XZ compression preset for Android assets/ libraries for local developers. r=esawin

Comment 3

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/f9cf33c52739

Comment 4

[Daniel Varga [:dvarga]]

Backed out changeset f9cf33c52739 (Bug 1498715) on request by nalexander for local developer failures. a=backout

Backout link: https://hg.mozilla.org/mozilla-central/rev/8d6a97e2a3570e5ed13d404aabcaab41ee53c813

Comment 5

[Nick Alexander :nalexander [he/him]]

This has been tripping up local developers, although we don't yet understand why.  We've seen:

success on macOS/XZ 5.2.4 (nalexander)
failure on Linux/XZ 5.1.0alpha (kats)
failure on macOS/XZ unknown

Bug 1380195 instituted the current situation, which explains why we're not using `--preset=...`.  (Annoying, that.)

kats shared a failing .so with me.  I'll try to instrument the XZ loader and figure out what's happening.

Comment 6

[Nick Alexander :nalexander [he/him]]

OK -- this is actually a Fennec vs geckoview_example issue, and that is because the packaging for those two is different.  Our custom `package_fennec_apk.py` script takes pains to always STORE assets/**/*.so.  Things that consume GeckoView do not take such pains, and have always been susceptible to unreadable assets/ libraries, 'cuz they might be DEFLATED.  Until now, however, we've always XZ compressed in such a way that nothing has, in fact, been DEFLATED.  The faster local developer settings allow things to DEFLATE instead, and that's the issue.

Now, we can use:
```
diff --git a/mobile/android/geckoview_example/build.gradle b/mobile/android/geckoview_example/build.gradle
--- a/mobile/android/geckoview_example/build.gradle
+++ b/mobile/android/geckoview_example/build.gradle
@@ -17,6 +17,10 @@ android {
         versionName "1.0"
     }
 
+    aaptOptions {
+        noCompress '.so'
+    }
+
     compileOptions {
         sourceCompatibility JavaVersion.VERSION_1_8
         targetCompatibility JavaVersion.VERSION_1_8
```
to get around this, with the downside of _also_ not compressing libs/ libraries.  It's not possible to be more fine-grained... cuz the Android-Gradle plugin just checks suffixes, not something richer (like regexs or globs).  That's probably fine.

But `aaptOptions` isn't a library-level thing; it's an App-level thing.  So it's pretty important that we do one or more of:

a) never ship non-maximally XZ compressed assets/ libraries (current)
b) accept DEFLATED assets/ libraries (future)
c) do something radically different around library compression (Bug 1486524)
d) force libraries to use `aaptOptions` (bad).

I think it's fine to fix GVE for now and reland this; upstream the `aaptOptions` for impacted folks who work cross-repo (FxR); and move on with our lives.

Comment 7

[Nick Alexander :nalexander [he/him]]

rbarker: would it be okay to add the `aaptOptions` block for FxR?  This change should really speed up your local development cycle, since it's roughly 1/3 the time spent compressing...

Comment 8

[Nick Alexander :nalexander [he/him]]

kats: could you reland (`hg graft -f -r f9cf33c52739`), apply the inline patch above, and verify that this fixes the issue you were witnessing with GVE?  Many thanks!

Comment 9

[Randall Barker [:rbarker]]

Our apk size jumped 31M:

-rw-r--r--  1 rbarker  staff    48M Oct 17 16:10 FirefoxReality-1.0.1-22901610-noapi-arm-debug.apk // before
-rw-r--r--  1 rbarker  staff    79M Oct 17 16:11 FirefoxReality-1.0.1-22901611-noapi-arm-debug.apk // after

Comment 10

[Randall Barker [:rbarker]]

Additionally, the generated APK doesn't work. I just get a black screen.

Comment 11

[Nick Alexander :nalexander [he/him]]

(In reply to Randall Barker [:rbarker] from comment #9)
> Our apk size jumped 31M:
> 
> -rw-r--r--  1 rbarker  staff    48M Oct 17 16:10
> FirefoxReality-1.0.1-22901610-noapi-arm-debug.apk // before
> -rw-r--r--  1 rbarker  staff    79M Oct 17 16:11
> FirefoxReality-1.0.1-22901611-noapi-arm-debug.apk // after

Ouch.  Push them somewhere for comparision?  I bet you have a bunch of non-XZ libs that are not getting compressed anymore.

Comment 12

[Nick Alexander :nalexander [he/him]]

(In reply to Nick Alexander :nalexander [he/him] from comment #11)
> (In reply to Randall Barker [:rbarker] from comment #9)
> > Our apk size jumped 31M:
> > 
> > -rw-r--r--  1 rbarker  staff    48M Oct 17 16:10
> > FirefoxReality-1.0.1-22901610-noapi-arm-debug.apk // before
> > -rw-r--r--  1 rbarker  staff    79M Oct 17 16:11
> > FirefoxReality-1.0.1-22901611-noapi-arm-debug.apk // after
> 
> Ouch.  Push them somewhere for comparision?  I bet you have a bunch of
> non-XZ libs that are not getting compressed anymore.

Or at least `zip -l -v ...` the contents for me.

Comment 13

[Randall Barker [:rbarker]]

>> diff ~/compress.txt ~/nocompress.txt 
1c1
< Archive:  ./build/outputs/apk/noapiArm/debug/FirefoxReality-1.0.1-22901646-noapi-arm-debug.apk
---
> Archive:  ./build/outputs/apk/noapiArm/debug/FirefoxReality-1.0.1-22901645-noapi-arm-debug.apk
4,7c4,7
<     9340  Defl:N     2313  75% 00-00-1980 00:00 0684a0d4  AndroidManifest.xml
<     1211  Defl:N     1064  12% 00-00-1980 00:00 f0260bb1  META-INF/CERT.RSA
<    72897  Defl:N    29591  59% 00-00-1980 00:00 aaf5b500  META-INF/CERT.SF
<    72835  Defl:N    28913  60% 00-00-1980 00:00 ccc6308f  META-INF/MANIFEST.MF
---
>     9340  Defl:N     2313  75% 00-00-1980 00:00 b55cc339  AndroidManifest.xml
>     1211  Defl:N     1064  12% 00-00-1980 00:00 10391b49  META-INF/CERT.RSA
>    72897  Defl:N    29588  59% 00-00-1980 00:00 2aea8915  META-INF/CERT.SF
>    72835  Defl:N    28919  60% 00-00-1980 00:00 bf3eafe2  META-INF/MANIFEST.MF
30,37c30,37
<   255972  Defl:N   156368  39% 00-00-1980 00:00 c249d4fe  assets/armeabi-v7a/libfreebl3.so
<    26792  Defl:N    13642  49% 00-00-1980 00:00 55ca3f9c  assets/armeabi-v7a/liblgpllibs.so
<   133476  Defl:N    62192  53% 00-00-1980 00:00 5b09515a  assets/armeabi-v7a/libmozavcodec.so
<   150008  Defl:N    71514  52% 00-00-1980 00:00 5448605e  assets/armeabi-v7a/libmozavutil.so
<  1226412  Defl:N   761639  38% 00-00-1980 00:00 e6d95f8f  assets/armeabi-v7a/libnss3.so
<   348452  Defl:N   177323  49% 00-00-1980 00:00 71371447  assets/armeabi-v7a/libnssckbi.so
<   146480  Defl:N    82598  44% 00-00-1980 00:00 7e90b434  assets/armeabi-v7a/libsoftokn3.so
< 59336044  Defl:N 30129821  49% 00-00-1980 00:00 c34bc6a6  assets/armeabi-v7a/libxul.so
---
>   255972  Stored   255972   0% 00-00-1980 00:00 c249d4fe  assets/armeabi-v7a/libfreebl3.so
>    26792  Stored    26792   0% 00-00-1980 00:00 55ca3f9c  assets/armeabi-v7a/liblgpllibs.so
>   133476  Stored   133476   0% 00-00-1980 00:00 5b09515a  assets/armeabi-v7a/libmozavcodec.so
>   150008  Stored   150008   0% 00-00-1980 00:00 5448605e  assets/armeabi-v7a/libmozavutil.so
>  1226412  Stored  1226412   0% 00-00-1980 00:00 e6d95f8f  assets/armeabi-v7a/libnss3.so
>   348452  Stored   348452   0% 00-00-1980 00:00 71371447  assets/armeabi-v7a/libnssckbi.so
>   146480  Stored   146480   0% 00-00-1980 00:00 7e90b434  assets/armeabi-v7a/libsoftokn3.so
> 59336044  Stored 59336044   0% 00-00-1980 00:00 c34bc6a6  assets/armeabi-v7a/libxul.so
68c68
<   553188  Defl:N   200148  64% 00-00-1980 00:00 4d62c13e  classes.dex
---
>   553188  Defl:N   200147  64% 00-00-1980 00:00 18051fda  classes.dex
249,257c249,257
<   571240  Defl:N   212909  63% 00-00-1980 00:00 8254b2b4  lib/armeabi-v7a/libc++_shared.so
<  1483660  Defl:N   785046  47% 00-00-1980 00:00 2485f601  lib/armeabi-v7a/libgvr.so
<   609864  Defl:N   377036  38% 00-00-1980 00:00 223346ff  lib/armeabi-v7a/libgvr_audio.so
<  1284708  Defl:N   509642  60% 00-00-1980 00:00 50deec2b  lib/armeabi-v7a/libmozglue.so
<   919512  Defl:N   400544  56% 00-00-1980 00:00 66f791e7  lib/armeabi-v7a/libnative-lib.so
<   230868  Defl:N   160221  31% 00-00-1980 00:00 fbc0db0c  lib/armeabi-v7a/libopus.so
<     9632  Defl:N     5414  44% 00-00-1980 00:00 919812b5  lib/armeabi-v7a/libopusjni.so
<    14368  Defl:N     7341  49% 00-00-1980 00:00 69d8633e  lib/armeabi-v7a/libplugin-container.so
<    38508  Defl:N    22840  41% 00-00-1980 00:00 1823315b  lib/armeabi-v7a/libwebrtc_jni.so
---
>   571240  Stored   571240   0% 00-00-1980 00:00 8254b2b4  lib/armeabi-v7a/libc++_shared.so
>  1483660  Stored  1483660   0% 00-00-1980 00:00 2485f601  lib/armeabi-v7a/libgvr.so
>   609864  Stored   609864   0% 00-00-1980 00:00 223346ff  lib/armeabi-v7a/libgvr_audio.so
>  1284708  Stored  1284708   0% 00-00-1980 00:00 50deec2b  lib/armeabi-v7a/libmozglue.so
>   919512  Stored   919512   0% 00-00-1980 00:00 66f791e7  lib/armeabi-v7a/libnative-lib.so
>   230868  Stored   230868   0% 00-00-1980 00:00 fbc0db0c  lib/armeabi-v7a/libopus.so
>     9632  Stored     9632   0% 00-00-1980 00:00 919812b5  lib/armeabi-v7a/libopusjni.so
>    14368  Stored    14368   0% 00-00-1980 00:00 69d8633e  lib/armeabi-v7a/libplugin-container.so
>    38508  Stored    38508   0% 00-00-1980 00:00 1823315b  lib/armeabi-v7a/libwebrtc_jni.so
640c640
< 88921594         49839376  44%                            635 files
---
> 88921594         82689284   7%                            635 files

Comment 14

[Kartikaya Gupta (email:[e-mail])]

The patch from comment 6 fixes it for me, GVE startup no longer crashes.

Comment 15

[Nick Alexander :nalexander [he/him]]

(In reply to Randall Barker [:rbarker] from comment #13)
> >> diff ~/compress.txt ~/nocompress.txt 
> 1c1
> < Archive: 
> ./build/outputs/apk/noapiArm/debug/FirefoxReality-1.0.1-22901646-noapi-arm-
> debug.apk
> ---
> > Archive:  ./build/outputs/apk/noapiArm/debug/FirefoxReality-1.0.1-22901645-noapi-arm-debug.apk
> 4,7c4,7
> <     9340  Defl:N     2313  75% 00-00-1980 00:00 0684a0d4 
> AndroidManifest.xml
> <     1211  Defl:N     1064  12% 00-00-1980 00:00 f0260bb1  META-INF/CERT.RSA
> <    72897  Defl:N    29591  59% 00-00-1980 00:00 aaf5b500  META-INF/CERT.SF
> <    72835  Defl:N    28913  60% 00-00-1980 00:00 ccc6308f 
> META-INF/MANIFEST.MF
> ---
> >     9340  Defl:N     2313  75% 00-00-1980 00:00 b55cc339  AndroidManifest.xml
> >     1211  Defl:N     1064  12% 00-00-1980 00:00 10391b49  META-INF/CERT.RSA
> >    72897  Defl:N    29588  59% 00-00-1980 00:00 2aea8915  META-INF/CERT.SF
> >    72835  Defl:N    28919  60% 00-00-1980 00:00 bf3eafe2  META-INF/MANIFEST.MF
> 30,37c30,37
> <   255972  Defl:N   156368  39% 00-00-1980 00:00 c249d4fe 
> assets/armeabi-v7a/libfreebl3.so
> <    26792  Defl:N    13642  49% 00-00-1980 00:00 55ca3f9c 
> assets/armeabi-v7a/liblgpllibs.so
> <   133476  Defl:N    62192  53% 00-00-1980 00:00 5b09515a 
> assets/armeabi-v7a/libmozavcodec.so
> <   150008  Defl:N    71514  52% 00-00-1980 00:00 5448605e 
> assets/armeabi-v7a/libmozavutil.so
> <  1226412  Defl:N   761639  38% 00-00-1980 00:00 e6d95f8f 
> assets/armeabi-v7a/libnss3.so
> <   348452  Defl:N   177323  49% 00-00-1980 00:00 71371447 
> assets/armeabi-v7a/libnssckbi.so
> <   146480  Defl:N    82598  44% 00-00-1980 00:00 7e90b434 
> assets/armeabi-v7a/libsoftokn3.so
> < 59336044  Defl:N 30129821  49% 00-00-1980 00:00 c34bc6a6 
> assets/armeabi-v7a/libxul.so
> ---
> >   255972  Stored   255972   0% 00-00-1980 00:00 c249d4fe  assets/armeabi-v7a/libfreebl3.so
> >    26792  Stored    26792   0% 00-00-1980 00:00 55ca3f9c  assets/armeabi-v7a/liblgpllibs.so
> >   133476  Stored   133476   0% 00-00-1980 00:00 5b09515a  assets/armeabi-v7a/libmozavcodec.so
> >   150008  Stored   150008   0% 00-00-1980 00:00 5448605e  assets/armeabi-v7a/libmozavutil.so
> >  1226412  Stored  1226412   0% 00-00-1980 00:00 e6d95f8f  assets/armeabi-v7a/libnss3.so
> >   348452  Stored   348452   0% 00-00-1980 00:00 71371447  assets/armeabi-v7a/libnssckbi.so
> >   146480  Stored   146480   0% 00-00-1980 00:00 7e90b434  assets/armeabi-v7a/libsoftokn3.so
> > 59336044  Stored 59336044   0% 00-00-1980 00:00 c34bc6a6  assets/armeabi-v7a/libxul.so
> 68c68
> <   553188  Defl:N   200148  64% 00-00-1980 00:00 4d62c13e  classes.dex
> ---
> >   553188  Defl:N   200147  64% 00-00-1980 00:00 18051fda  classes.dex
> 249,257c249,257
> <   571240  Defl:N   212909  63% 00-00-1980 00:00 8254b2b4 
> lib/armeabi-v7a/libc++_shared.so
> <  1483660  Defl:N   785046  47% 00-00-1980 00:00 2485f601 
> lib/armeabi-v7a/libgvr.so
> <   609864  Defl:N   377036  38% 00-00-1980 00:00 223346ff 
> lib/armeabi-v7a/libgvr_audio.so
> <  1284708  Defl:N   509642  60% 00-00-1980 00:00 50deec2b 
> lib/armeabi-v7a/libmozglue.so
> <   919512  Defl:N   400544  56% 00-00-1980 00:00 66f791e7 
> lib/armeabi-v7a/libnative-lib.so
> <   230868  Defl:N   160221  31% 00-00-1980 00:00 fbc0db0c 
> lib/armeabi-v7a/libopus.so
> <     9632  Defl:N     5414  44% 00-00-1980 00:00 919812b5 
> lib/armeabi-v7a/libopusjni.so
> <    14368  Defl:N     7341  49% 00-00-1980 00:00 69d8633e 
> lib/armeabi-v7a/libplugin-container.so
> <    38508  Defl:N    22840  41% 00-00-1980 00:00 1823315b 
> lib/armeabi-v7a/libwebrtc_jni.so
> ---
> >   571240  Stored   571240   0% 00-00-1980 00:00 8254b2b4  lib/armeabi-v7a/libc++_shared.so
> >  1483660  Stored  1483660   0% 00-00-1980 00:00 2485f601  lib/armeabi-v7a/libgvr.so
> >   609864  Stored   609864   0% 00-00-1980 00:00 223346ff  lib/armeabi-v7a/libgvr_audio.so
> >  1284708  Stored  1284708   0% 00-00-1980 00:00 50deec2b  lib/armeabi-v7a/libmozglue.so
> >   919512  Stored   919512   0% 00-00-1980 00:00 66f791e7  lib/armeabi-v7a/libnative-lib.so
> >   230868  Stored   230868   0% 00-00-1980 00:00 fbc0db0c  lib/armeabi-v7a/libopus.so
> >     9632  Stored     9632   0% 00-00-1980 00:00 919812b5  lib/armeabi-v7a/libopusjni.so
> >    14368  Stored    14368   0% 00-00-1980 00:00 69d8633e  lib/armeabi-v7a/libplugin-container.so
> >    38508  Stored    38508   0% 00-00-1980 00:00 1823315b  lib/armeabi-v7a/libwebrtc_jni.so
> 640c640
> < 88921594         49839376  44%                            635 files
> ---
> > 88921594         82689284   7%                            635 files

Could you repeat the procedure, with and without the new patch in https://phabricator.services.mozilla.com/D8614?  You'll want to copy the restricted block from mobile/android/geckoview_example/build.gradle to GVR as well.  I expect the black screen was a problem loading mozglue or another lib/ library that we don't want stored.

I'm very surprised that you see such a big change in libxul.so itself -- I see only a few Mb difference (~10%); you see almost 30 Mb (~50%).  If that persists we should dig deeper?

Thanks!

Comment 16

[Nick Alexander :nalexander [he/him]]

We're going to stop compressing libs entirely in https://bugzilla.mozilla.org/show_bug.cgi?id=1486524 instead.