1367768 - Gradle build fails with DexIndexOverflowException, consider enabling jumbo mode.

Status: RESOLVED FIXED

(Firefox Build System :: Android Studio and Gradle Integration, defect)

Description

[Jan Henning [:JanH]]

Attachment: My mozconfig

Building from today's mozilla-central, my local build fails with
> 10:59.68 Execution failed for task ':app:transformClassesWithDexForOfficialAustralisDebug'.
> 10:59.68 > com.android.build.api.transform.TransformException: com.android.ide.common.process.ProcessException: java.util.concurrent.ExecutionException: com.android.dex.DexIndexOverflowException: Cannot merge new index 70822 into a non-jumbo instruction!

Whatever caused this landed somewhere between https://hg.mozilla.org/mozilla-central/log?rev=6dfa56094f0c and https://hg.mozilla.org/mozilla-central/log?rev=e8da7192201e. The Exoplayer integration from bug 1341990 would be a prime suspect for taking our method count over the limit and indeed, if I hack my moz.configure to deactivate MOZ_ANDROID_HLS_SUPPORT, the build is working again.

Strangely enough, the Tier-2 Gradle builds on Treeherder are still working fine, though.

Comment 1

[Nick Alexander :nalexander [he/him]]

> Strangely enough, the Tier-2 Gradle builds on Treeherder are still working
> fine, though.

This can often be a local issue, caused by what I believe are long-existing bugs in the DEX merger in the Android-Gradle build plugin.  Try |./mach gradle clean app:...|, which often avoids the issue.

I don't know of any other workaround, although the issue may be fixed by newer versions of the various build toolchain components.  See Bug 1366644 for that; I'd love to have somebody _try_ and see what fails/gets better/needs to change in automation.

Comment 2

[Jan Henning [:JanH]]

Running ./mach gradle clean before ./mach build (just to clarify, this is with the --with-gradle build option) doesn't help and in fact it's still failing even on a clobber build.

Comment 3

[James Cheng[:JamesCheng]]

Hi Jan,
I don't really know the reason why the exception happened,
but I think it can be worked around by Bug 1341990 comment 29.

Comment 4

[Jan Henning [:JanH]]

Question:
Is anybody else seeing this on a clean build when using a mozconfig similar to mine, i.e. with "ac_add_options --with-gradle"? I.e. clobber and then just run ./mach build.

Comment 5

[Jan Henning [:JanH]]

So if I disable --with-gradle in my mozconfig to get ./mach build working again, the local build variant (e.g. ./mach gradle clean app:assembleLocalAustralisDebug) still works fine. It's only the OfficialAustralis builds that ./mach build uses when building --with-gradle that are apparently problematic when built locally with a plain mozconfig.

Comment 6

[Teng-pao Yu [:tyu]]

I thought I would not run into this since I didn't have the --with-gradle in the mozconifg. However I just run into this with my Gradle build (Android Studio run button, :app:assembleLocalOldAustralisDebug) after a resource change. Not sure if this is only triggered with a resource change since I did not run into this last week.

I'm not going to dig deep into this atm since a full mach build and mach package fixes the issue

Gradle log:

:app:compileLocalOldAustralisDebugJavaWithJavac
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

:app:compileLocalOldAustralisDebugSources
:app:transformClassesWithDexForLocalOldAustralisDebug
To run dex in process, the Gradle daemon needs a larger heap.
It currently has approximately 910 MB.
For faster builds, increase the maximum heap size for the Gradle daemon to more than 3072 MB.
To do this set org.gradle.jvmargs=-Xmx3072M in the project gradle.properties.
For more information see https://docs.gradle.org/current/userguide/build_environment.html

AGPBI: {"kind":"error","text":"Error converting bytecode to dex:\nCause: com.android.dex.DexIndexOverflowException: Cannot merge new index 71368 into a non-jumbo instruction!","sources":[{}],"original":"UNEXPECTED TOP-LEVEL EXCEPTION:\ncom.android.dex.DexIndexOverflowException: Cannot merge new index 71368 into a non-jumbo instruction!\n\tat com.android.dx.merge.InstructionTransformer.jumboCheck(InstructionTransformer.java:111)\n\tat com.android.dx.merge.InstructionTransformer.access$800(InstructionTransformer.java:26)\n\tat com.android.dx.merge.InstructionTransformer$StringVisitor.visit(InstructionTransformer.java:74)\n\tat com.android.dx.io.CodeReader.callVisit(CodeReader.java:114)\n\tat com.android.dx.io.CodeReader.visitAll(CodeReader.java:89)\n\tat com.android.dx.merge.InstructionTransformer.transform(InstructionTransformer.java:50)\n\tat com.android.dx.merge.DexMerger.transformCode(DexMerger.java:826)\n\tat com.android.dx.merge.DexMerger.transformMethods(DexMerger.java:800)\n\tat com.android.dx.merge.DexMerger.transformClassData(DexMerger.java:773)\n\tat com.android.dx.merge.DexMerger.transformClassDef(DexMerger.java:669)\n\tat com.android.dx.merge.DexMerger.mergeClassDefs(DexMerger.java:523)\n\tat com.android.dx.merge.DexMerger.mergeDexes(DexMerger.java:164)\n\tat com.android.dx.merge.DexMerger.merge(DexMerger.java:188)\n\tat com.android.dx.command.dexer.Main.mergeLibraryDexBuffers(Main.java:504)\n\tat com.android.dx.command.dexer.Main.runMonoDex(Main.java:334)\n\tat com.android.dx.command.dexer.Main.run(Main.java:277)\n\tat com.android.dx.command.dexer.Main.main(Main.java:245)\n\tat com.android.dx.command.Main.main(Main.java:106)\n","tool":"Dex"}

:app:transformClassesWithDexForLocalOldAustralisDebug FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:transformClassesWithDexForLocalOldAustralisDebug'.
> com.android.build.api.transform.TransformException: com.android.ide.common.process.ProcessException: java.util.concurrent.ExecutionException: com.android.ide.common.process.ProcessException: org.gradle.process.internal.ExecException: Process 'command '/Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java'' finished with non-zero exit value 2

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 49.833 secs

Comment 7

[Teng-pao Yu [:tyu]]

Hmmm sorry I was wrong in my last comment. mach build and mach package is not fixing the problem. In the meanwhile this reminds me I may not be running Gradle builds last week. I was running lint checks only with Gradle but maybe not Gradle builds. So my assumption regarding the resource change may be incorrect too. (In fact I tried to checkout to the current maven-central tip and still failed to build with Gradle)

Comment 8

[Teng-pao Yu [:tyu]]

Gradle fails with DexIndexOverflowException, consider enabling jumbo mode.

Comment 9

[Teng-pao Yu [:tyu]]

I've done a little bit investigation regarding our string table size in Bug 1370156. The conclusion is we're really having a (too) big string table and Bug 1341990 exploded it on our Gradle build. on pre-1341990 environment I've tested the apk size difference between jumbomode disabled and enabled. Enabling jumbomode gives a 0.02% increase[1] and I think it is acceptable especially given the context that the mach build table size is also near 60000 so we'll need this someday if not now.

What do you think Nick? Can I simply enable jumbomode to give this a quick fix? (I guess I won't land the code before merge day, though.)

I've also filed Bug 1370158 to track the table size difference between mach build and Gradle build, in hope that we can trim the Gradle build apk size just like mach build.

[1] 34963430 -> 34971233 tested on https://hg.mozilla.org/mozilla-central/rev/ef182af1ccd7, localOld-australis-debug

Comment 10

[Teng-pao Yu [:tyu]]

(In reply to Jan Henning [:JanH] from comment #0)
> Strangely enough, the Tier-2 Gradle builds on Treeherder are still working
> fine, though.

Per what I know, those Tier-2 tests does not require a Gradle build, they're more like static tests (which requires Gradle but not a build).

Comment 11

[Jan Henning [:JanH]]

(In reply to Teng-pao Yu [:tyu] from comment #10)
> (In reply to Jan Henning [:JanH] from comment #0)
> > Strangely enough, the Tier-2 Gradle builds on Treeherder are still working
> > fine, though.
> 
> Per what I know, those Tier-2 tests does not require a Gradle build, they're
> more like static tests (which requires Gradle but not a build).

I don't think so - looking at the output and logs (e.g. https://public-artifacts.taskcluster.net/NzS6VmOzQxS1TdIZ4TLVlQ/0/public/logs/live_backing.log) for a random Gradle build from Treeherder, it *is* a full build.

So something about our build environment or the mozconfigs (starting here https://dxr.mozilla.org/mozilla-central/rev/8a3aa1701537ea6b8334f432cd030d260d492fa3/mobile/android/config/mozconfigs/android-api-15-gradle/nightly and then looking further through the includes) in automation is causing some difference when compared to a local build.

Comment 12

[Teng-pao Yu [:tyu]]

(In reply to Jan Henning [:JanH] from comment #11)
> (In reply to Teng-pao Yu [:tyu] from comment #10)
> > (In reply to Jan Henning [:JanH] from comment #0)
> > > Strangely enough, the Tier-2 Gradle builds on Treeherder are still working
> > > fine, though.
> > 
> > Per what I know, those Tier-2 tests does not require a Gradle build, they're
> > more like static tests (which requires Gradle but not a build).
> 
> I don't think so - looking at the output and logs (e.g.
> https://public-artifacts.taskcluster.net/NzS6VmOzQxS1TdIZ4TLVlQ/0/public/
> logs/live_backing.log) for a random Gradle build from Treeherder, it *is* a
> full build.
> 
> So something about our build environment or the mozconfigs (starting here
> https://dxr.mozilla.org/mozilla-central/rev/
> 8a3aa1701537ea6b8334f432cd030d260d492fa3/mobile/android/config/mozconfigs/
> android-api-15-gradle/nightly and then looking further through the includes)
> in automation is causing some difference when compared to a local build.

Oops sorry, I thought by tier2 you're referring to android-test, android-lint ...etc for android-api-15. Now I see you mean tasks for build-android-api-15-gradle and you're definitely correct on this.

My guess is at build.gradle https://dxr.mozilla.org/mozilla-central/source/mobile/android/app/build.gradle we enable minifiy only on automations (Treeherder tests). With minify, the unused strings are eliminated so that's probably why its working on Treeherder.

I've tested this locally with

buildTypes {
    debug {
        minifyEnabled true
    }
}

and the table size shrinks to 35934 on tip.

I think this could be the reason but I'm not 100% sure minify is enabled on all automation tests. Nick can you confirm if I'm guessing it right?

Comment 13

[Teng-pao Yu [:tyu]]

Seems like the build-android-api-15-gradle is already utilizing jumbomode, see Bug 1370156#c10 for details.

Comment 14

[Nick Alexander :nalexander [he/him]]

(In reply to Teng-pao Yu [:tyu] from comment #12)
> (In reply to Jan Henning [:JanH] from comment #11)
> > (In reply to Teng-pao Yu [:tyu] from comment #10)
> > > (In reply to Jan Henning [:JanH] from comment #0)
> > > > Strangely enough, the Tier-2 Gradle builds on Treeherder are still working
> > > > fine, though.
> > > 
> > > Per what I know, those Tier-2 tests does not require a Gradle build, they're
> > > more like static tests (which requires Gradle but not a build).
> > 
> > I don't think so - looking at the output and logs (e.g.
> > https://public-artifacts.taskcluster.net/NzS6VmOzQxS1TdIZ4TLVlQ/0/public/
> > logs/live_backing.log) for a random Gradle build from Treeherder, it *is* a
> > full build.
> > 
> > So something about our build environment or the mozconfigs (starting here
> > https://dxr.mozilla.org/mozilla-central/rev/
> > 8a3aa1701537ea6b8334f432cd030d260d492fa3/mobile/android/config/mozconfigs/
> > android-api-15-gradle/nightly and then looking further through the includes)
> > in automation is causing some difference when compared to a local build.
> 
> Oops sorry, I thought by tier2 you're referring to android-test,
> android-lint ...etc for android-api-15. Now I see you mean tasks for
> build-android-api-15-gradle and you're definitely correct on this.

The tier 2 jobs are the android-{test,lint,findbugs,checkstyle} jobs.  They don't *really* require a build but they need enough build related stuff (like branding, strings.xml processing stuff) that they do an artifact build.  So they use a "frontend" (== artifact) mozconfig (see https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/builds/releng_sub_android_configs/64_lint.py#5) and then do some post-build Gradle (see https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/builds/releng_sub_android_configs/64_lint.py#11).

The android-api-15-gradle build is a full compile-environment build that uses Gradle for the Java bits (rather than mobile/android/base/Makefile.in Java invocations).

> My guess is at build.gradle
> https://dxr.mozilla.org/mozilla-central/source/mobile/android/app/build.
> gradle we enable minifiy only on automations (Treeherder tests). With
> minify, the unused strings are eliminated so that's probably why its working
> on Treeherder.

This is very delicate: see https://dxr.mozilla.org/mozilla-central/source/mobile/android/app/build.gradle#48-73.

Basically, all release builds are minified (everywhere), and even debug is minified in automation.  That is why all of the android-* jobs are succeeding: even the debug bits for, e.g., linting are getting minified even though they're just doing things based on source code, etc.

> I've tested this locally with
> 
> buildTypes {
>     debug {
>         minifyEnabled true
>     }
> }
> 
> and the table size shrinks to 35934 on tip.
> 
> I think this could be the reason but I'm not 100% sure minify is enabled on
> all automation tests. Nick can you confirm if I'm guessing it right?

See above.

I am surprised that jumboMode is enabled.  I will investigate locally, perhaps it is implied by `minifyEnabled true`?

Comment 15

[Teng-pao Yu [:tyu]]

Attachment: Bug 1367768 - Enable jumbo mode for Gradle debug builds;

Review commit: https://reviewboard.mozilla.org/r/147032/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/147032/

Comment 16

[Teng-pao Yu [:tyu]]

Enable jumbo mode for Gradle debug builds to deal with this problem on developers local devices.

Comment 17

[Max Liu [:maliu]]

Comment on attachment 8875617 [details]
Bug 1367768 - Enable jumbo mode for Gradle debug builds;

https://reviewboard.mozilla.org/r/147032/#review151258

LGTM

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/e77d611eee6e
Enable jumbo mode for Gradle debug builds; r=maliu

Comment 19

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/e77d611eee6e