Closed Bug 1304903 Opened 9 years ago Closed 9 years ago

[meta] Investigate method counts / reduction (> 65k methods trying to use support library 23.4)

Categories

(Firefox Build System :: Android Studio and Gradle Integration, defect, P1)

Product:
Firefox Build System
Component:
Android Studio and Gradle Integration
Type:
defect

Tracking

(firefox52 affected)

Status:
RESOLVED FIXED
Iteration:
1.5
Tracking Flags:
Tracking Status
firefox52 --- affected

People

(Reporter: ahunt, Assigned: ahunt)

References

Details

(Keywords: meta, Whiteboard: [MobileAS])

Trying to upgrade to support library 23.4 results in dexing errors as a result of there being too many methods. It looks like android.support.* gained around 2000 methods between 23.0.1 and 23.4.0, which just pushes us over the edge of 65k methods. This results in |mach build| failing. In a gradle build (where gradle library versions are updated after a full |mach build && mach package|), localOldDebug fails, but localOld can still build thanks to multidex. The general support library is split up in 24.X, which might allow us to reduce method counts, however it would be useful in general to reduce our method counts to give us more breathing room for future changes and/or libraries. The keepsafe dex-count plugin appears to be reporting erronous numbers: it claims 55/57k methods for local-debug (before/after switching to 23.4). Doing a manual dex count on the local-debug apk after switching to 23.4 shows a total of 65868 methods, the following tool gives me the same results when uploading app-local-debug.apk: http://inloop.github.io/apk-method-count/
Trying to run |mach build| after upgrading the library versions result in: 0:59.64 classes.dex 1:09.09 1:09.09 trouble writing output: Too many field references: 66810; max is 65536. 1:09.09 You may try using --multi-dex option. 1:09.09 References by package: 1:09.09 3 android.accounts 1:09.10 21 android.app 1:09.10 4 android.content 1:09.10 42 android.content.pm 1:09.10 6 android.content.res 1:09.10 43 android.graphics 1:09.10 8 android.hardware 1:09.10 6 android.media 1:09.10 3 android.media.audiofx 1:09.10 2 android.net 1:09.10 5 android.net.wifi 1:09.10 26 android.os 1:09.10 1 android.preference 1:09.10 1 android.print 1:09.10 5 android.provider 1:09.10 3861 android.support.customtabs 1:09.10 3854 android.support.design 1:09.10 43 android.support.design.internal 1:09.10 491 android.support.design.widget 1:09.10 3854 android.support.v4 1:09.10 17 android.support.v4.animation 1:09.10 383 android.support.v4.app 1:09.10 69 android.support.v4.content 1:09.10 1 android.support.v4.graphics 1:09.10 35 android.support.v4.graphics.drawable 1:09.10 3 android.support.v4.hardware.display 1:09.10 71 android.support.v4.media 1:09.10 46 android.support.v4.media.session 1:09.10 1 android.support.v4.net 1:09.10 10 android.support.v4.os 1:09.10 16 android.support.v4.text 1:09.10 57 android.support.v4.util 1:09.10 213 android.support.v4.view 1:09.10 16 android.support.v4.view.accessibility 1:09.11 5 android.support.v4.view.animation 1:09.11 324 android.support.v4.widget 1:09.11 394 android.support.v7.app 1:09.11 3854 android.support.v7.appcompat 1:09.11 3854 android.support.v7.cardview 1:09.11 48 android.support.v7.graphics 1:09.11 1 android.support.v7.graphics.drawable 1:09.11 188 android.support.v7.media 1:09.11 3854 android.support.v7.mediarouter 1:09.11 3854 android.support.v7.palette 1:09.11 3854 android.support.v7.recyclerview 1:09.11 1 android.support.v7.text 1:09.11 63 android.support.v7.view 1:09.11 148 android.support.v7.view.menu 1:09.11 1259 android.support.v7.widget 1:09.11 87 android.support.v7.widget.helper 1:09.11 1 android.support.v7.widget.util 1:09.11 4 android.telephony 1:09.11 9 android.text 1:09.11 12 android.util 1:09.11 26 android.view 1:09.11 1 android.view.accessibility 1:09.11 15 android.view.inputmethod 1:09.11 21 android.widget 1:09.11 15 ch.boye.httpclientandroidlib 1:09.11 5 ch.boye.httpclientandroidlib.androidextra 1:09.11 34 ch.boye.httpclientandroidlib.auth 1:09.11 31 ch.boye.httpclientandroidlib.client.config 1:09.11 1 ch.boye.httpclientandroidlib.client.entity 1:09.11 19 ch.boye.httpclientandroidlib.client.methods 1:09.11 9 ch.boye.httpclientandroidlib.client.protocol 1:09.11 25 ch.boye.httpclientandroidlib.client.utils 1:09.11 4 ch.boye.httpclientandroidlib.config 1:09.11 7 ch.boye.httpclientandroidlib.conn 1:09.12 4 ch.boye.httpclientandroidlib.conn.params 1:09.12 19 ch.boye.httpclientandroidlib.conn.routing 1:09.12 7 ch.boye.httpclientandroidlib.conn.scheme 1:09.12 11 ch.boye.httpclientandroidlib.conn.ssl 1:09.12 3 ch.boye.httpclientandroidlib.conn.util 1:09.12 7 ch.boye.httpclientandroidlib.cookie 1:09.12 14 ch.boye.httpclientandroidlib.entity 1:09.12 16 ch.boye.httpclientandroidlib.impl 1:09.12 77 ch.boye.httpclientandroidlib.impl.auth 1:09.12 72 ch.boye.httpclientandroidlib.impl.client 1:09.12 45 ch.boye.httpclientandroidlib.impl.conn 1:09.12 46 ch.boye.httpclientandroidlib.impl.conn.tsccm 1:09.12 38 ch.boye.httpclientandroidlib.impl.cookie 1:09.12 4 ch.boye.httpclientandroidlib.impl.entity 1:09.12 67 ch.boye.httpclientandroidlib.impl.io 1:09.12 53 ch.boye.httpclientandroidlib.message 1:09.12 1 ch.boye.httpclientandroidlib.params 1:09.12 8 ch.boye.httpclientandroidlib.pool 1:09.12 15 ch.boye.httpclientandroidlib.protocol 1:09.12 9 ch.boye.httpclientandroidlib.util 1:09.12 3854 com.google.android.gms 1:09.12 13 com.google.android.gms.ads.identifier 1:09.12 13 com.google.android.gms.auth.api.signin 1:09.12 4 com.google.android.gms.auth.api.signin.internal 1:09.12 3854 com.google.android.gms.base 1:09.12 3997 com.google.android.gms.cast 1:09.12 91 com.google.android.gms.cast.internal 1:09.12 36 com.google.android.gms.common 1:09.12 33 com.google.android.gms.common.api 1:09.12 145 com.google.android.gms.common.api.internal 1:09.12 5 com.google.android.gms.common.images 1:09.12 141 com.google.android.gms.common.internal 1:09.12 1 com.google.android.gms.common.internal.safeparcel 1:09.12 54 com.google.android.gms.common.stats 1:09.12 4 com.google.android.gms.dynamic 1:09.12 3871 com.google.android.gms.gcm 1:09.12 37 com.google.android.gms.iid 1:09.12 179 com.google.android.gms.internal 1:09.12 3870 com.google.android.gms.measurement 1:09.12 333 com.google.android.gms.measurement.internal 1:09.12 26 com.google.android.gms.signin.internal 1:09.12 18 com.googlecode.eyesfree.braille.selfbraille 1:09.12 40 com.jakewharton.disklrucache 1:09.12 4 com.keepsafe.switchboard 1:09.12 1 com.squareup.leakcanary 1:09.13 174 com.squareup.picasso 1:09.13 1 java.io 1:09.13 12 java.lang 1:09.13 7 java.lang.annotation 1:09.13 3 java.net 1:09.13 1 java.nio 1:09.13 1 java.nio.charset 1:09.13 7 java.util 1:09.13 6 java.util.concurrent 1:09.13 1 javax.microedition.khronos.egl 1:09.13 1 org.json 1:09.13 26 org.json.simple.parser 1:09.13 45 org.mozilla.apache.commons.codec.binary 1:09.13 5165 org.mozilla.gecko 1:09.13 40 org.mozilla.gecko.animation 1:09.13 6 org.mozilla.gecko.background.common 1:09.13 3 org.mozilla.gecko.background.common.log 1:09.13 9 org.mozilla.gecko.background.common.log.writers 1:09.13 2 org.mozilla.gecko.background.common.telemetry 1:09.13 4 org.mozilla.gecko.background.db 1:09.13 38 org.mozilla.gecko.background.fxa 1:09.13 21 org.mozilla.gecko.background.fxa.oauth 1:09.13 3 org.mozilla.gecko.background.fxa.profile 1:09.13 12 org.mozilla.gecko.background.preferences 1:09.13 7 org.mozilla.gecko.browserid 1:09.13 4 org.mozilla.gecko.cleanup 1:09.13 3 org.mozilla.gecko.customtabs 1:09.13 404 org.mozilla.gecko.db 1:09.13 14 org.mozilla.gecko.delegates 1:09.13 61 org.mozilla.gecko.distribution 1:09.13 3 org.mozilla.gecko.dlc 1:09.13 36 org.mozilla.gecko.dlc.catalog 1:09.13 4 org.mozilla.gecko.feeds 1:09.13 6 org.mozilla.gecko.feeds.action 1:09.13 11 org.mozilla.gecko.feeds.parser 1:09.13 7 org.mozilla.gecko.feeds.subscriptions 1:09.13 44 org.mozilla.gecko.firstrun 1:09.13 33 org.mozilla.gecko.fxa 1:09.13 62 org.mozilla.gecko.fxa.activities 1:09.13 46 org.mozilla.gecko.fxa.authenticator 1:09.13 59 org.mozilla.gecko.fxa.login 1:09.13 5 org.mozilla.gecko.fxa.receivers 1:09.13 51 org.mozilla.gecko.fxa.sync 1:09.13 4 org.mozilla.gecko.gcm 1:09.13 218 org.mozilla.gecko.gfx 1:09.13 5 org.mozilla.gecko.health 1:09.13 690 org.mozilla.gecko.home 1:09.13 20 org.mozilla.gecko.home.activitystream 1:09.13 32 org.mozilla.gecko.home.activitystream.topsites 1:09.13 40 org.mozilla.gecko.icons 1:09.13 28 org.mozilla.gecko.icons.decoders 1:09.13 3 org.mozilla.gecko.icons.loader 1:09.13 1 org.mozilla.gecko.icons.preparation 1:09.13 1 org.mozilla.gecko.icons.processing 1:09.13 10 org.mozilla.gecko.icons.storage 1:09.14 18 org.mozilla.gecko.javaaddons 1:09.14 18 org.mozilla.gecko.lwt 1:09.14 4 org.mozilla.gecko.mdns 1:09.14 72 org.mozilla.gecko.media 1:09.14 77 org.mozilla.gecko.menu 1:09.14 13 org.mozilla.gecko.mozglue 1:09.14 36 org.mozilla.gecko.notifications 1:09.14 10 org.mozilla.gecko.overlays.service 1:09.14 16 org.mozilla.gecko.overlays.service.sharemethods 1:09.14 49 org.mozilla.gecko.overlays.ui 1:09.14 11 org.mozilla.gecko.permissions 1:09.14 159 org.mozilla.gecko.preferences 1:09.14 29 org.mozilla.gecko.promotion 1:09.14 100 org.mozilla.gecko.prompts 1:09.14 40 org.mozilla.gecko.push 1:09.14 23 org.mozilla.gecko.push.autopush 1:09.14 11 org.mozilla.gecko.reader 1:09.14 42 org.mozilla.gecko.restrictions 1:09.14 18 org.mozilla.gecko.search 1:09.14 11 org.mozilla.gecko.sqlite 1:09.14 95 org.mozilla.gecko.sync 1:09.14 10 org.mozilla.gecko.sync.crypto 1:09.14 21 org.mozilla.gecko.sync.middleware 1:09.14 37 org.mozilla.gecko.sync.net 1:09.14 38 org.mozilla.gecko.sync.repositories 1:09.14 155 org.mozilla.gecko.sync.repositories.android 1:09.14 16 org.mozilla.gecko.sync.repositories.delegates 1:09.14 94 org.mozilla.gecko.sync.repositories.domain 1:09.14 47 org.mozilla.gecko.sync.repositories.uploaders 1:09.14 4 org.mozilla.gecko.sync.setup.activities 1:09.14 78 org.mozilla.gecko.sync.stage 1:09.14 45 org.mozilla.gecko.sync.synchronizer 1:09.14 39 org.mozilla.gecko.tabqueue 1:09.14 176 org.mozilla.gecko.tabs 1:09.14 21 org.mozilla.gecko.telemetry 1:09.14 4 org.mozilla.gecko.telemetry.measurements 1:09.14 5 org.mozilla.gecko.telemetry.pingbuilders 1:09.14 6 org.mozilla.gecko.telemetry.stores 1:09.14 16 org.mozilla.gecko.text 1:09.14 20 org.mozilla.gecko.tokenserver 1:09.14 272 org.mozilla.gecko.toolbar 1:09.14 5 org.mozilla.gecko.trackingprotection 1:09.14 33 org.mozilla.gecko.updater 1:09.14 84 org.mozilla.gecko.util 1:09.14 419 org.mozilla.gecko.widget 1:09.14 80 org.mozilla.gecko.widget.themed 1:09.14 11 org.mozilla.mozstumbler.service 1:09.14 1 org.mozilla.mozstumbler.service.mainthread 1:09.14 10 org.mozilla.mozstumbler.service.stumblerthread 1:09.14 5 org.mozilla.mozstumbler.service.stumblerthread.blocklist 1:09.14 34 org.mozilla.mozstumbler.service.stumblerthread.datahandling 1:09.14 32 org.mozilla.mozstumbler.service.stumblerthread.scanners 1:09.14 34 org.mozilla.mozstumbler.service.stumblerthread.scanners.cellscanner 1:09.14 13 org.mozilla.mozstumbler.service.uploadthread 1:09.15 22 org.mozilla.mozstumbler.service.utils 1:09.15 52 org.mozilla.search 1:09.15 31 org.mozilla.search.autocomplete 1:09.15 1 org.mozilla.search.ui 1:09.15 59 org.webrtc.videoengine 1:09.15 44 org.webrtc.voiceengine 1:09.24 make[5]: *** [classes.dex] Error 2 1:09.24 make[4]: *** [mobile/android/base/libs] Error 2 1:09.25 make[3]: *** [libs] Error 2 1:09.25 make[2]: *** [default] Error 2 1:09.25 make[1]: *** [realbuild] Error 2 1:09.25 make: *** [build] Error 2 1:09.26 0 compiler warnings present.
If I instead |mach build| && |mach package| with 23.0.1, and then change only the build.gradle versions to 23.4.0, I get: Dex: Error converting bytecode to dex: Cause: com.android.dex.DexIndexOverflowException: Cannot merge new index 67585 into a non-jumbo instruction! UNEXPECTED TOP-LEVEL EXCEPTION: com.android.dex.DexIndexOverflowException: Cannot merge new index 67585 into a non-jumbo instruction! at com.android.dx.merge.InstructionTransformer.jumboCheck(InstructionTransformer.java:111) at com.android.dx.merge.InstructionTransformer.access$800(InstructionTransformer.java:26) at com.android.dx.merge.InstructionTransformer$StringVisitor.visit(InstructionTransformer.java:74) at com.android.dx.io.CodeReader.callVisit(CodeReader.java:114) at com.android.dx.io.CodeReader.visitAll(CodeReader.java:89) at com.android.dx.merge.InstructionTransformer.transform(InstructionTransformer.java:50) at com.android.dx.merge.DexMerger.transformCode(DexMerger.java:826) at com.android.dx.merge.DexMerger.transformMethods(DexMerger.java:800) at com.android.dx.merge.DexMerger.transformClassData(DexMerger.java:773) at com.android.dx.merge.DexMerger.transformClassDef(DexMerger.java:669) at com.android.dx.merge.DexMerger.mergeClassDefs(DexMerger.java:523) at com.android.dx.merge.DexMerger.mergeDexes(DexMerger.java:164) at com.android.dx.merge.DexMerger.merge(DexMerger.java:188) at com.android.dx.command.dexer.Main.mergeLibraryDexBuffers(Main.java:504) at com.android.dx.command.dexer.Main.runMonoDex(Main.java:334) at com.android.dx.command.dexer.Main.run(Main.java:277) at com.android.dx.command.dexer.Main.main(Main.java:245) at com.android.dx.command.Main.main(Main.java:106) :app:transformClassesWithDexForLocalOldDebug FAILED FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':app:transformClassesWithDexForLocalOldDebug'. > com.android.build.api.transform.TransformException: com.android.ide.common.process.ProcessException: java.util.concurrent.ExecutionException: com.android.ide.common.process.ProcessException: org.gradle.process.internal.ExecException: Process 'command '/Library/Java/JavaVirtualMachines/jdk1.8.0_102.jdk/Contents/Home/bin/java'' finished with non-zero exit value 2 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. BUILD FAILED Total time: 36.248 secs
I can get the second (gradle-only) variant running if I replace leakcanary with the no-op version (leakcanary seems to bundle part of guava, resulting in a large method count). This makes me wonder if it's worth disabling leakcanary for localOld builds (no multidex available), however that wouldn't fix the pure |mach build| scenario, since (I think) leakcanary is only used in gradle builds. More promising might be getting rid of ch.boye.*, since that adds just over 5000 methods, see e.g. Bug 765064. That would give us a decent bit of breathing room for now.
I also need to look at the tools mentioned in Bug 1260827 to see how they differ in their counts.
See Also: → 1260827
In the mach build, a lot of the individual support libraries have suspiciously high (and similar counts): [Note: these counts are only for methods within a package, but not for subpackages] 1:43.56 3861 android.support.customtabs 1:43.56 3854 android.support.design 1:43.56 3854 android.support.v4 1:43.56 3854 android.support.v7.appcompat 1:43.56 3854 android.support.v7.cardview 1:43.56 3854 android.support.v7.mediarouter 1:43.56 3854 android.support.v7.palette 1:43.56 3854 android.support.v7.recyclerview The equivalents from the dexcount plugin are (first number: methods, second number: fields) [Note: these counts include counts from subpackage, hence support.v4 is larger here] 105 70 android.support.customtabs 1756 2224 android.support.design 9022 2818 android.support.v4 8 42 android.support.v7.cardview etc etc.
(In reply to Andrzej Hunt :ahunt from comment #0) > The keepsafe dex-count plugin appears to be reporting erronous numbers: it > claims 55/57k methods for local-debug (before/after switching to 23.4). > Doing a manual dex count on the local-debug apk after switching to 23.4 > shows a total of 65868 methods, the following tool gives me the same results > when uploading app-local-debug.apk: > > http://inloop.github.io/apk-method-count/ And I forgot to mention: the difference seems to be based on system/java libraries: these are ignored in the gradle dex-count plugin, but the apk-method-count page also counts all usages of system and or java built-in methods (and it seems like those contribute to the dex total method count).
(In reply to Andrzej Hunt :ahunt from comment #1) > 1:09.12 3997 com.google.android.gms.cast > 1:09.12 91 com.google.android.gms.cast.internal If we kill off chromecast (disabled since 36) and roku (likely at least somewhat broken as well + extremely low DAU) support, I think we can remove these as well. IF the count is correct, that's 4k methods.
(In reply to Andrzej Hunt :ahunt from comment #3) > More promising might be getting rid of ch.boye.*, since that adds just over > 5000 methods, see e.g. Bug 765064. That would give us a decent bit of > breathing room for now. I've noticed that our version of adjust also uses ch.boye.*, so if we do this we'd either need to update (adjust >= 4.1.2: https://github.com/adjust/android_sdk/issues/138 ), or modify our bundled version.
Chromecast has not been disabled it still ships to release users.
Assignee: nobody → ahunt
Priority: -- → P1
Whiteboard: [MobileAS]
For now it seems like it's enough to: - Reduce our R.java duplication mess, Bug 1306021 (almost ready for review) (it turns out our mach builds are field constrained, not method constrained - the mach builds run proguard even for debug builds) - Consider disabling leakcanary, and/or proguard libraries before dexing for localOldDebug gradle builds (this would mean that multidex localDebug builds have to be used for full functionality, i.e. only for Android > 5).
Depends on: 1306021
Status: NEW → ASSIGNED
Depends on: 1306104
Summary: Investigate method counts / reduction (> 65k methods trying to use support library 23.4) → [meta] Investigate method counts / reduction (> 65k methods trying to use support library 23.4)
I've managed to ameliorate the field issues in Bug 1306021, and we've disabbled leakcanary for non-multidex local gradle builds in Bug 1306104 which gives us enough space to upgrade the support libraries. I'm still looking into adding metrics in Bug 1260827. Longer term we will probably want to look into remove our ch.boye.* dependencies, and we might want to evaluate whether we want to keep casting support - but at least we've got a path forward for the short term.
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Iteration: --- → 1.5
Product: Firefox for Android → Firefox Build System
You need to log in before you can comment on or make changes to this bug.