https://treeherder.mozilla.org/logviewer.html#?job_id=1423441&repo=mozilla-beta#L1574 Android checkstyle tc(checkstyle) fails with 11:31:30 INFO - DEBUG: configure: error: Must specify --with-adjust-sdk-keyfile when MOZ_INSTALL_TRACKING is defined! 11:31:30 INFO - ERROR: old-configure failed 11:31:30 INFO - *** Fix above errors and then restart with\ 11:31:31 INFO - "/usr/bin/gmake -f client.mk build"
bug 1277553 was expected to fix this but if checkstyle is the only thing permafailing now, it seems it only fixed this for moz.build (and not gradle builds). To verify Adjust is set up correctly for our moz.build (i.e. release builds), I checked the Adjust dashboards and verified the install rate has not decreased within the past few days (assuming the build has been released & propagated to Google play). - Logs from the failure indicate: --enable-update-channel=beta The code to set this is : elif test "$MOZ_UPDATE_CHANNEL" = "beta" ; then ac_add_options --with-adjust-sdk-keyfile=/builds/adjust-sdk-beta.token If the path to the keyfile does not exist, I believe we get this error so my first guess would be the adjust sdk keyfile is missing on the builders for gradle builds. When we formerly did gradle builds, I assume we just silently failed because we didn't have the keyfile assertion. Unfortunately, I don't know the difference between the builders for gradle & moz.build. Nick, do you have any thoughts? : https://dxr.mozilla.org/mozilla-beta/rev/b4ead0ffd25ac75f0ef6124f34b74e86c1d573eb/mobile/android/config/mozconfigs/common#57
(In reply to Michael Comella (:mcomella) from comment #1) > bug 1277553 was expected to fix this but if checkstyle is the only thing > permafailing now, it seems it only fixed this for moz.build (and not gradle > builds). To verify Adjust is set up correctly for our moz.build (i.e. > release builds), I checked the Adjust dashboards and verified the install > rate has not decreased within the past few days (assuming the build has been > released & propagated to Google play). > - > > Logs from the failure indicate: --enable-update-channel=beta > > The code to set this is : > > elif test "$MOZ_UPDATE_CHANNEL" = "beta" ; then > ac_add_options --with-adjust-sdk-keyfile=/builds/adjust-sdk-beta.token > > If the path to the keyfile does not exist, I believe we get this error so my > first guess would be the adjust sdk keyfile is missing on the builders for > gradle builds. When we formerly did gradle builds, I assume we just silently > failed because we didn't have the keyfile assertion. > > Unfortunately, I don't know the difference between the builders for gradle & > moz.build. Nick, do you have any thoughts? > > : > https://dxr.mozilla.org/mozilla-beta/rev/ > b4ead0ffd25ac75f0ef6124f34b74e86c1d573eb/mobile/android/config/mozconfigs/ > common#57 I really don't understand how all these buildbot machines have all these different configurations. I think nthomas just arranged to get the same set of tokens onto the l10n repack machines; perhaps he can work the same magic here, so redirecting to him.
FWIW, this is a build/job done on Taskcluster, so it doesn't get any of the same mechanisms that happen in Buildbot builds. (Such as 'mock' and the api keys we install that way). The failing job from c#0 is https://tools.taskcluster.net/task-inspector/#ErBCG7LBTwOMNGl0hPD4QA/ And from that log: 11:31:27 INFO - Adding configure options from /home/worker/workspace/build/src/.mozconfig 11:31:27 INFO - --enable-crashreporter 11:31:27 INFO - --enable-release 11:31:27 INFO - --enable-js-shell 11:31:27 INFO - --enable-elf-hack 11:31:27 INFO - --enable-application=mobile/android 11:31:27 INFO - --with-android-sdk=/home/worker/workspace/build/src/android-sdk-linux 11:31:27 INFO - --with-android-version=9 11:31:27 INFO - --with-system-zlib 11:31:27 INFO - --enable-update-channel=beta 11:31:27 INFO - --enable-warnings-as-errors 11:31:27 INFO - --with-mozilla-api-keyfile=/builds/mozilla-fennec-geoloc-api.key 11:31:27 INFO - --with-adjust-sdk-keyfile=/builds/adjust-sdk-beta.token 11:31:27 INFO - --enable-stdcxx-compat 11:31:27 INFO - --with-android-cxx-stl=libc++ 11:31:27 INFO - --with-gradle=/home/worker/workspace/build/src/gradle/bin/gradle 11:31:27 INFO - --disable-compile-environment 11:31:27 INFO - --disable-tests 11:31:27 INFO - --enable-profiling 11:31:27 INFO - --with-android-min-sdk=15 11:31:27 INFO - --target=arm-linux-androideabi 11:31:27 INFO - --with-branding=mobile/android/branding/nightly 11:31:27 INFO - SOCORRO_SYMBOL_UPLOAD_TOKEN_FILE=/builds/crash-stats-api.token 11:31:27 INFO - GRADLE_MAVEN_REPOSITORY=file:///home/worker/workspace/build/src/jcentral 11:31:27 INFO - MOZILLA_OFFICIAL=1 11:31:27 INFO - MOZ_TELEMETRY_REPORTING=1 11:31:27 INFO - MOZ_PACKAGE_JSSHELL=1 11:31:27 INFO - PATH=/tools/buildbot/bin:/usr/local/bin:/bin:/usr/bin:/home/worker/workspace/build/src/java_home/bin:/home/worker/wo rkspace/build/src/java_home/bin:/home/worker/workspace/build/src/java_home/bin 11:31:27 INFO - NO_NDK=1 11:31:27 INFO - MOZ_ANDROID_GECKOLIBS_AAR=1 11:31:27 INFO - MOZ_SZIP_FLAGS=-D auto -f auto 11:31:27 INFO - MOZ_ADDON_SIGNING=1 11:31:27 INFO - MOZ_REQUIRE_SIGNING=0 11:31:27 INFO - ANDROID_NDK_VERSION=r10e 11:31:27 INFO - STRIP_FLAGS=--strip-debug 11:31:27 INFO - ANDROID_NDK_VERSION_32BIT=r8c 11:31:27 INFO - NO_CACHE=1 11:31:27 INFO - JS_BINARY=/home/worker/workspace/build/src/mobile/android/config/js_wrapper.sh 11:31:27 INFO - checking for a shell... /bin/sh
This can be set with stuff like: https://dxr.mozilla.org/mozilla-beta/source/testing/mozharness/configs/builds/releng_base_android_64_builds.py#44 (But admittedly I'm not an expert on how TC secrets work, so I suggest asking in #taskcluster)
And for the record, this is how its used: https://dxr.mozilla.org/mozilla-beta/source/testing/mozharness/mozharness/mozilla/secrets.py?q=get_secrets&redirect_type=single#31
6 automation job failures were associated with this bug in the last 7 days. Repository breakdown: * mozilla-beta: 6 Platform breakdown: * android-4-0-armv7-api15: 6 For more details, see: https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292580&startday=2016-08-01&endday=2016-08-07&tree=all
NI self to review this.
(In reply to Michael Comella (:mcomella) from comment #7) > NI self to review this. To be explicit, I don't primarily work on fennec anymore and it could be a few days before I get back to this. I wouldn't mind if someone else took this from me.
Looks like these jobs are still failing: https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&filter-searchStr=checkstyle&fromchange=144875f1f75ba3384346a24915114d2da6269d26 I'll try to get to this soon.
FYI, these jobs are hidden by bug 1299015. fwiw, this failure should only appear when: 1) Patches are uplifted onto beta 2) A full test is run on beta Since checkstyle is used to make the code more readable for developers, making the code easier to understand and preventing errors, failures in #2 are irrelevant. Addressing #1, since patches written on beta are low frequency, I feel preventing checkstyle errors on beta is low priority to fix.
Brian, do you know who might be able to explain how TC secret files work (as per comment 4)?
I'm happy to explain the secrets of secrets to the best of my abilities and then escalate to someone more knowledgeable when we hit the limits of my knowledge. ping me on irc when you've got time.
I talked to Jonas and Dustin. It sounds like the checkstyle task  does not contain the TC secrets scope  required to access the adjust key secrets . So we need to: * Add the scope to the task * Make sure mozharness downloads the appropriate secrets (which it may just do ) : https://hg.mozilla.org/releases/mozilla-beta/file/6106148d2caa25c93134c88d20f00c4a62e91715/testing/taskcluster/tasks/builds/android_checkstyle.yml#l21 : https://dxr.mozilla.org/mozilla-beta/source/taskcluster/ci/legacy/tasks/builds/firefox_docker_base.yml#8 : https://dxr.mozilla.org/mozilla-central/rev/f0e6cc6360213ba21fd98c887b55fce5c680df68/testing/mozharness/configs/builds/releng_base_linux_32_builds.py#55 : The task cluster task has the following attr: "MOZHARNESS_ACTIONS": "get-secrets build multi-l10n update"
Jonas is doubtful mozharness would just download the secrets: <&jonasfj> hehe, doubtful probably somewhere we have to add an entry to "secret_files", well, unless it uses the morharness file already configured 15:30 I would just expect morharness to complain if it failed to download a secret and couldn't. 15:30 hence, I'm thinking it didn't try
Also, the adjust secrets are declared level 2 (meaning they can't be accessed for try) so it's unlikely I'd be able to push try jobs to make sure this works! Jonas suggested I might be able to trigger it locally.
(In reply to Michael Comella (:mcomella) [not actively working on fennec: expect slow responses] from comment #15) > Also, the adjust secrets are declared level 2 (meaning they can't be > accessed for try) so it's unlikely I'd be able to push try jobs to make sure > this works! Spoke too soon: apparently I might be able to change the secret level to 1 and push to try anyway. In any case, TC provides a default key so configure shouldn't fail (...should we make it fail?) when we don't have permissions (e.g. on try).
We're no longer using legacy configs  so it seems it might just be as simple as adding `secrets: true` (like in )! : https://dxr.mozilla.org/mozilla-central/rev/f0e6cc6360213ba21fd98c887b55fce5c680df68/taskcluster/ci/build/android.yml#55 : https://dxr.mozilla.org/mozilla-central/rev/f0e6cc6360213ba21fd98c887b55fce5c680df68/taskcluster/ci/build/linux.yml#21
> In any case, TC provides a default key so configure shouldn't fail (...should we make it fail?) > when we don't have permissions (e.g. on try). NI self to answer (or file follow-up).
I don't have time to continue working on this but there should be enough info here for some to step in and continue pushing. This isn't high priority until we activate gradle for production builds – blocking bug 1254353. (In reply to Michael Comella (:mcomella) [not actively working on fennec: expect slow responses] from comment #18) > > In any case, TC provides a default key so configure shouldn't fail (...should we make it fail?) > > when we don't have permissions (e.g. on try). > > NI self to answer (or file follow-up). Filed bug 1306724 to follow up to make sure we fix this too.
Dupe of bug 1317359?