Gradle build fails with DexIndexOverflowException, consider enabling jumbo mode.

RESOLVED FIXED in Firefox 55

Status

()

defect
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: JanH, Assigned: tyu)

Tracking

(Depends on 1 bug, Blocks 1 bug)

Trunk
Firefox 55
All
Android
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox55 fixed)

Details

Attachments

(2 attachments)

Reporter

Description

2 years ago
Posted file My mozconfig
Building from today's mozilla-central, my local build fails with
> 10:59.68 Execution failed for task ':app:transformClassesWithDexForOfficialAustralisDebug'.
> 10:59.68 > com.android.build.api.transform.TransformException: com.android.ide.common.process.ProcessException: java.util.concurrent.ExecutionException: com.android.dex.DexIndexOverflowException: Cannot merge new index 70822 into a non-jumbo instruction!

Whatever caused this landed somewhere between https://hg.mozilla.org/mozilla-central/log?rev=6dfa56094f0c and https://hg.mozilla.org/mozilla-central/log?rev=e8da7192201e. The Exoplayer integration from bug 1341990 would be a prime suspect for taking our method count over the limit and indeed, if I hack my moz.configure to deactivate MOZ_ANDROID_HLS_SUPPORT, the build is working again.

Strangely enough, the Tier-2 Gradle builds on Treeherder are still working fine, though.
> Strangely enough, the Tier-2 Gradle builds on Treeherder are still working
> fine, though.

This can often be a local issue, caused by what I believe are long-existing bugs in the DEX merger in the Android-Gradle build plugin.  Try |./mach gradle clean app:...|, which often avoids the issue.

I don't know of any other workaround, although the issue may be fixed by newer versions of the various build toolchain components.  See Bug 1366644 for that; I'd love to have somebody _try_ and see what fails/gets better/needs to change in automation.
Reporter

Comment 2

2 years ago
Running ./mach gradle clean before ./mach build (just to clarify, this is with the --with-gradle build option) doesn't help and in fact it's still failing even on a clobber build.
Hi Jan,
I don't really know the reason why the exception happened,
but I think it can be worked around by Bug 1341990 comment 29.
Reporter

Comment 4

2 years ago
Question:
Is anybody else seeing this on a clean build when using a mozconfig similar to mine, i.e. with "ac_add_options --with-gradle"? I.e. clobber and then just run ./mach build.
Reporter

Updated

2 years ago
Reporter

Comment 5

2 years ago
So if I disable --with-gradle in my mozconfig to get ./mach build working again, the local build variant (e.g. ./mach gradle clean app:assembleLocalAustralisDebug) still works fine. It's only the OfficialAustralis builds that ./mach build uses when building --with-gradle that are apparently problematic when built locally with a plain mozconfig.
Assignee

Comment 6

2 years ago
I thought I would not run into this since I didn't have the --with-gradle in the mozconifg. However I just run into this with my Gradle build (Android Studio run button, :app:assembleLocalOldAustralisDebug) after a resource change. Not sure if this is only triggered with a resource change since I did not run into this last week.

I'm not going to dig deep into this atm since a full mach build and mach package fixes the issue

Gradle log:

:app:compileLocalOldAustralisDebugJavaWithJavac
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

:app:compileLocalOldAustralisDebugSources
:app:transformClassesWithDexForLocalOldAustralisDebug
To run dex in process, the Gradle daemon needs a larger heap.
It currently has approximately 910 MB.
For faster builds, increase the maximum heap size for the Gradle daemon to more than 3072 MB.
To do this set org.gradle.jvmargs=-Xmx3072M in the project gradle.properties.
For more information see https://docs.gradle.org/current/userguide/build_environment.html

AGPBI: {"kind":"error","text":"Error converting bytecode to dex:\nCause: com.android.dex.DexIndexOverflowException: Cannot merge new index 71368 into a non-jumbo instruction!","sources":[{}],"original":"UNEXPECTED TOP-LEVEL EXCEPTION:\ncom.android.dex.DexIndexOverflowException: Cannot merge new index 71368 into a non-jumbo instruction!\n\tat com.android.dx.merge.InstructionTransformer.jumboCheck(InstructionTransformer.java:111)\n\tat com.android.dx.merge.InstructionTransformer.access$800(InstructionTransformer.java:26)\n\tat com.android.dx.merge.InstructionTransformer$StringVisitor.visit(InstructionTransformer.java:74)\n\tat com.android.dx.io.CodeReader.callVisit(CodeReader.java:114)\n\tat com.android.dx.io.CodeReader.visitAll(CodeReader.java:89)\n\tat com.android.dx.merge.InstructionTransformer.transform(InstructionTransformer.java:50)\n\tat com.android.dx.merge.DexMerger.transformCode(DexMerger.java:826)\n\tat com.android.dx.merge.DexMerger.transformMethods(DexMerger.java:800)\n\tat com.android.dx.merge.DexMerger.transformClassData(DexMerger.java:773)\n\tat com.android.dx.merge.DexMerger.transformClassDef(DexMerger.java:669)\n\tat com.android.dx.merge.DexMerger.mergeClassDefs(DexMerger.java:523)\n\tat com.android.dx.merge.DexMerger.mergeDexes(DexMerger.java:164)\n\tat com.android.dx.merge.DexMerger.merge(DexMerger.java:188)\n\tat com.android.dx.command.dexer.Main.mergeLibraryDexBuffers(Main.java:504)\n\tat com.android.dx.command.dexer.Main.runMonoDex(Main.java:334)\n\tat com.android.dx.command.dexer.Main.run(Main.java:277)\n\tat com.android.dx.command.dexer.Main.main(Main.java:245)\n\tat com.android.dx.command.Main.main(Main.java:106)\n","tool":"Dex"}

:app:transformClassesWithDexForLocalOldAustralisDebug FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:transformClassesWithDexForLocalOldAustralisDebug'.
> com.android.build.api.transform.TransformException: com.android.ide.common.process.ProcessException: java.util.concurrent.ExecutionException: com.android.ide.common.process.ProcessException: org.gradle.process.internal.ExecException: Process 'command '/Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java'' finished with non-zero exit value 2

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 49.833 secs
Assignee

Comment 7

2 years ago
Hmmm sorry I was wrong in my last comment. mach build and mach package is not fixing the problem. In the meanwhile this reminds me I may not be running Gradle builds last week. I was running lint checks only with Gradle but maybe not Gradle builds. So my assumption regarding the resource change may be incorrect too. (In fact I tried to checkout to the current maven-central tip and still failed to build with Gradle)
Assignee

Updated

2 years ago
Blocks: 1370156
Assignee

Comment 8

2 years ago
Gradle fails with DexIndexOverflowException, consider enabling jumbo mode.
Summary: Local build with gradle fails with DexIndexOverflowException → Gradle build fails with DexIndexOverflowException, consider enabling jumbo mode.
Assignee

Comment 9

2 years ago
I've done a little bit investigation regarding our string table size in Bug 1370156. The conclusion is we're really having a (too) big string table and Bug 1341990 exploded it on our Gradle build. on pre-1341990 environment I've tested the apk size difference between jumbomode disabled and enabled. Enabling jumbomode gives a 0.02% increase[1] and I think it is acceptable especially given the context that the mach build table size is also near 60000 so we'll need this someday if not now.

What do you think Nick? Can I simply enable jumbomode to give this a quick fix? (I guess I won't land the code before merge day, though.)

I've also filed Bug 1370158 to track the table size difference between mach build and Gradle build, in hope that we can trim the Gradle build apk size just like mach build.

[1] 34963430 -> 34971233 tested on https://hg.mozilla.org/mozilla-central/rev/ef182af1ccd7, localOld-australis-debug
Flags: needinfo?(nalexander)
Assignee

Comment 10

2 years ago
(In reply to Jan Henning [:JanH] from comment #0)
> Strangely enough, the Tier-2 Gradle builds on Treeherder are still working
> fine, though.

Per what I know, those Tier-2 tests does not require a Gradle build, they're more like static tests (which requires Gradle but not a build).
Reporter

Comment 11

2 years ago
(In reply to Teng-pao Yu [:tyu] from comment #10)
> (In reply to Jan Henning [:JanH] from comment #0)
> > Strangely enough, the Tier-2 Gradle builds on Treeherder are still working
> > fine, though.
> 
> Per what I know, those Tier-2 tests does not require a Gradle build, they're
> more like static tests (which requires Gradle but not a build).

I don't think so - looking at the output and logs (e.g. https://public-artifacts.taskcluster.net/NzS6VmOzQxS1TdIZ4TLVlQ/0/public/logs/live_backing.log) for a random Gradle build from Treeherder, it *is* a full build.

So something about our build environment or the mozconfigs (starting here https://dxr.mozilla.org/mozilla-central/rev/8a3aa1701537ea6b8334f432cd030d260d492fa3/mobile/android/config/mozconfigs/android-api-15-gradle/nightly and then looking further through the includes) in automation is causing some difference when compared to a local build.
No longer blocks: 1370156
Depends on: 1370156
Assignee

Comment 12

2 years ago
(In reply to Jan Henning [:JanH] from comment #11)
> (In reply to Teng-pao Yu [:tyu] from comment #10)
> > (In reply to Jan Henning [:JanH] from comment #0)
> > > Strangely enough, the Tier-2 Gradle builds on Treeherder are still working
> > > fine, though.
> > 
> > Per what I know, those Tier-2 tests does not require a Gradle build, they're
> > more like static tests (which requires Gradle but not a build).
> 
> I don't think so - looking at the output and logs (e.g.
> https://public-artifacts.taskcluster.net/NzS6VmOzQxS1TdIZ4TLVlQ/0/public/
> logs/live_backing.log) for a random Gradle build from Treeherder, it *is* a
> full build.
> 
> So something about our build environment or the mozconfigs (starting here
> https://dxr.mozilla.org/mozilla-central/rev/
> 8a3aa1701537ea6b8334f432cd030d260d492fa3/mobile/android/config/mozconfigs/
> android-api-15-gradle/nightly and then looking further through the includes)
> in automation is causing some difference when compared to a local build.

Oops sorry, I thought by tier2 you're referring to android-test, android-lint ...etc for android-api-15. Now I see you mean tasks for build-android-api-15-gradle and you're definitely correct on this.

My guess is at build.gradle https://dxr.mozilla.org/mozilla-central/source/mobile/android/app/build.gradle we enable minifiy only on automations (Treeherder tests). With minify, the unused strings are eliminated so that's probably why its working on Treeherder.

I've tested this locally with

buildTypes {
    debug {
        minifyEnabled true
    }
}

and the table size shrinks to 35934 on tip.

I think this could be the reason but I'm not 100% sure minify is enabled on all automation tests. Nick can you confirm if I'm guessing it right?
Reporter

Updated

2 years ago
Depends on: 1370158
Assignee

Comment 13

2 years ago
Seems like the build-android-api-15-gradle is already utilizing jumbomode, see Bug 1370156#c10 for details.
(In reply to Teng-pao Yu [:tyu] from comment #12)
> (In reply to Jan Henning [:JanH] from comment #11)
> > (In reply to Teng-pao Yu [:tyu] from comment #10)
> > > (In reply to Jan Henning [:JanH] from comment #0)
> > > > Strangely enough, the Tier-2 Gradle builds on Treeherder are still working
> > > > fine, though.
> > > 
> > > Per what I know, those Tier-2 tests does not require a Gradle build, they're
> > > more like static tests (which requires Gradle but not a build).
> > 
> > I don't think so - looking at the output and logs (e.g.
> > https://public-artifacts.taskcluster.net/NzS6VmOzQxS1TdIZ4TLVlQ/0/public/
> > logs/live_backing.log) for a random Gradle build from Treeherder, it *is* a
> > full build.
> > 
> > So something about our build environment or the mozconfigs (starting here
> > https://dxr.mozilla.org/mozilla-central/rev/
> > 8a3aa1701537ea6b8334f432cd030d260d492fa3/mobile/android/config/mozconfigs/
> > android-api-15-gradle/nightly and then looking further through the includes)
> > in automation is causing some difference when compared to a local build.
> 
> Oops sorry, I thought by tier2 you're referring to android-test,
> android-lint ...etc for android-api-15. Now I see you mean tasks for
> build-android-api-15-gradle and you're definitely correct on this.

The tier 2 jobs are the android-{test,lint,findbugs,checkstyle} jobs.  They don't *really* require a build but they need enough build related stuff (like branding, strings.xml processing stuff) that they do an artifact build.  So they use a "frontend" (== artifact) mozconfig (see https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/builds/releng_sub_android_configs/64_lint.py#5) and then do some post-build Gradle (see https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/configs/builds/releng_sub_android_configs/64_lint.py#11).

The android-api-15-gradle build is a full compile-environment build that uses Gradle for the Java bits (rather than mobile/android/base/Makefile.in Java invocations).

> My guess is at build.gradle
> https://dxr.mozilla.org/mozilla-central/source/mobile/android/app/build.
> gradle we enable minifiy only on automations (Treeherder tests). With
> minify, the unused strings are eliminated so that's probably why its working
> on Treeherder.

This is very delicate: see https://dxr.mozilla.org/mozilla-central/source/mobile/android/app/build.gradle#48-73.

Basically, all release builds are minified (everywhere), and even debug is minified in automation.  That is why all of the android-* jobs are succeeding: even the debug bits for, e.g., linting are getting minified even though they're just doing things based on source code, etc.

> I've tested this locally with
> 
> buildTypes {
>     debug {
>         minifyEnabled true
>     }
> }
> 
> and the table size shrinks to 35934 on tip.
> 
> I think this could be the reason but I'm not 100% sure minify is enabled on
> all automation tests. Nick can you confirm if I'm guessing it right?

See above.

I am surprised that jumboMode is enabled.  I will investigate locally, perhaps it is implied by `minifyEnabled true`?
Flags: needinfo?(nalexander)
Comment hidden (mozreview-request)
Assignee

Updated

2 years ago
Blocks: 1370156
No longer depends on: 1370156
Assignee

Updated

2 years ago
Assignee: nobody → osimpleo
Assignee

Comment 16

2 years ago
Enable jumbo mode for Gradle debug builds to deal with this problem on developers local devices.

Comment 17

2 years ago
mozreview-review
Comment on attachment 8875617 [details]
Bug 1367768 - Enable jumbo mode for Gradle debug builds;

https://reviewboard.mozilla.org/r/147032/#review151258

LGTM
Attachment #8875617 - Flags: review?(max) → review+
Assignee

Updated

2 years ago
Keywords: checkin-needed

Comment 18

2 years ago
Pushed by mozilla@noorenberghe.ca:
https://hg.mozilla.org/integration/mozilla-inbound/rev/e77d611eee6e
Enable jumbo mode for Gradle debug builds; r=maliu
Keywords: checkin-needed

Comment 19

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/e77d611eee6e
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 55
You need to log in before you can comment on or make changes to this bug.