Beta Android checkstyle tc(checkstyle) builds fails due to configure error

NEW
Unassigned

Status

()

Firefox
Build Config
a year ago
17 days ago

People

(Reporter: Tomcat, Unassigned)

Tracking

(Blocks: 2 bugs, {intermittent-failure})

Firefox Tracking Flags

(Not tracked)

Details

(URL)

(Reporter)

Description

a year ago
https://treeherder.mozilla.org/logviewer.html#?job_id=1423441&repo=mozilla-beta#L1574

Android checkstyle tc(checkstyle) fails with 

 11:31:30     INFO -  DEBUG: configure: error: Must specify --with-adjust-sdk-keyfile when MOZ_INSTALL_TRACKING is defined!
 11:31:30     INFO -  ERROR: old-configure failed
 11:31:30     INFO -  *** Fix above errors and then restart with\
11:31:31 INFO - "/usr/bin/gmake -f client.mk build"
Flags: needinfo?(michael.l.comella)
See Also: → bug 1277553
bug 1277553 was expected to fix this but if checkstyle is the only thing permafailing now, it seems it only fixed this for moz.build (and not gradle builds). To verify Adjust is set up correctly for our moz.build (i.e. release builds), I checked the Adjust dashboards and verified the install rate has not decreased within the past few days (assuming the build has been released & propagated to Google play).
-

Logs from the failure indicate: --enable-update-channel=beta

The code to set this is [1]:

elif test "$MOZ_UPDATE_CHANNEL" = "beta" ; then
    ac_add_options --with-adjust-sdk-keyfile=/builds/adjust-sdk-beta.token

If the path to the keyfile does not exist, I believe we get this error so my first guess would be the adjust sdk keyfile is missing on the builders for gradle builds. When we formerly did gradle builds, I assume we just silently failed because we didn't have the keyfile assertion.

Unfortunately, I don't know the difference between the builders for gradle & moz.build. Nick, do you have any thoughts?

[1]: https://dxr.mozilla.org/mozilla-beta/rev/b4ead0ffd25ac75f0ef6124f34b74e86c1d573eb/mobile/android/config/mozconfigs/common#57
Flags: needinfo?(michael.l.comella) → needinfo?(nalexander)
(In reply to Michael Comella (:mcomella) from comment #1)
> bug 1277553 was expected to fix this but if checkstyle is the only thing
> permafailing now, it seems it only fixed this for moz.build (and not gradle
> builds). To verify Adjust is set up correctly for our moz.build (i.e.
> release builds), I checked the Adjust dashboards and verified the install
> rate has not decreased within the past few days (assuming the build has been
> released & propagated to Google play).
> -
> 
> Logs from the failure indicate: --enable-update-channel=beta
> 
> The code to set this is [1]:
> 
> elif test "$MOZ_UPDATE_CHANNEL" = "beta" ; then
>     ac_add_options --with-adjust-sdk-keyfile=/builds/adjust-sdk-beta.token
> 
> If the path to the keyfile does not exist, I believe we get this error so my
> first guess would be the adjust sdk keyfile is missing on the builders for
> gradle builds. When we formerly did gradle builds, I assume we just silently
> failed because we didn't have the keyfile assertion.
> 
> Unfortunately, I don't know the difference between the builders for gradle &
> moz.build. Nick, do you have any thoughts?
> 
> [1]:
> https://dxr.mozilla.org/mozilla-beta/rev/
> b4ead0ffd25ac75f0ef6124f34b74e86c1d573eb/mobile/android/config/mozconfigs/
> common#57

I really don't understand how all these buildbot machines have all these different configurations.  I think nthomas just arranged to get the same set of tokens onto the l10n repack machines; perhaps he can work the same magic here, so redirecting to him.
Flags: needinfo?(nalexander) → needinfo?(nthomas)
FWIW, this is a build/job done on Taskcluster, so it doesn't get any of the same mechanisms that happen in Buildbot builds. (Such as 'mock' and the api keys we install that way).

The failing job from c#0 is https://tools.taskcluster.net/task-inspector/#ErBCG7LBTwOMNGl0hPD4QA/

And from that log:

11:31:27     INFO -  Adding configure options from /home/worker/workspace/build/src/.mozconfig
11:31:27     INFO -    --enable-crashreporter
11:31:27     INFO -    --enable-release
11:31:27     INFO -    --enable-js-shell
11:31:27     INFO -    --enable-elf-hack
11:31:27     INFO -    --enable-application=mobile/android
11:31:27     INFO -    --with-android-sdk=/home/worker/workspace/build/src/android-sdk-linux
11:31:27     INFO -    --with-android-version=9
11:31:27     INFO -    --with-system-zlib
11:31:27     INFO -    --enable-update-channel=beta
11:31:27     INFO -    --enable-warnings-as-errors
11:31:27     INFO -    --with-mozilla-api-keyfile=/builds/mozilla-fennec-geoloc-api.key
11:31:27     INFO -    --with-adjust-sdk-keyfile=/builds/adjust-sdk-beta.token
11:31:27     INFO -    --enable-stdcxx-compat
11:31:27     INFO -    --with-android-cxx-stl=libc++
11:31:27     INFO -    --with-gradle=/home/worker/workspace/build/src/gradle/bin/gradle
11:31:27     INFO -    --disable-compile-environment
11:31:27     INFO -    --disable-tests
11:31:27     INFO -    --enable-profiling
11:31:27     INFO -    --with-android-min-sdk=15
11:31:27     INFO -    --target=arm-linux-androideabi
11:31:27     INFO -    --with-branding=mobile/android/branding/nightly
11:31:27     INFO -    SOCORRO_SYMBOL_UPLOAD_TOKEN_FILE=/builds/crash-stats-api.token
11:31:27     INFO -    GRADLE_MAVEN_REPOSITORY=file:///home/worker/workspace/build/src/jcentral
11:31:27     INFO -    MOZILLA_OFFICIAL=1
11:31:27     INFO -    MOZ_TELEMETRY_REPORTING=1
11:31:27     INFO -    MOZ_PACKAGE_JSSHELL=1
11:31:27     INFO -    PATH=/tools/buildbot/bin:/usr/local/bin:/bin:/usr/bin:/home/worker/workspace/build/src/java_home/bin:/home/worker/wo
rkspace/build/src/java_home/bin:/home/worker/workspace/build/src/java_home/bin
11:31:27     INFO -    NO_NDK=1
11:31:27     INFO -    MOZ_ANDROID_GECKOLIBS_AAR=1
11:31:27     INFO -    MOZ_SZIP_FLAGS=-D auto -f auto
11:31:27     INFO -    MOZ_ADDON_SIGNING=1
11:31:27     INFO -    MOZ_REQUIRE_SIGNING=0
11:31:27     INFO -    ANDROID_NDK_VERSION=r10e
11:31:27     INFO -    STRIP_FLAGS=--strip-debug
11:31:27     INFO -    ANDROID_NDK_VERSION_32BIT=r8c
11:31:27     INFO -    NO_CACHE=1
11:31:27     INFO -    JS_BINARY=/home/worker/workspace/build/src/mobile/android/config/js_wrapper.sh
11:31:27     INFO -  checking for a shell... /bin/sh
Flags: needinfo?(nthomas)
This can be set with stuff like:

https://dxr.mozilla.org/mozilla-beta/source/testing/mozharness/configs/builds/releng_base_android_64_builds.py#44

(But admittedly I'm not an expert on how TC secrets work, so I suggest asking in #taskcluster)
And for the record, this is how its used:

https://dxr.mozilla.org/mozilla-beta/source/testing/mozharness/mozharness/mozilla/secrets.py?q=get_secrets&redirect_type=single#31

Comment 6

a year ago
6 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-beta: 6

Platform breakdown:
* android-4-0-armv7-api15: 6

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1292580&startday=2016-08-01&endday=2016-08-07&tree=all
NI self to review this.
Flags: needinfo?(michael.l.comella)
(In reply to Michael Comella (:mcomella) from comment #7)
> NI self to review this.

To be explicit, I don't primarily work on fennec anymore and it could be a few days before I get back to this. I wouldn't mind if someone else took this from me.
Looks like these jobs are still failing:
  https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&filter-searchStr=checkstyle&fromchange=144875f1f75ba3384346a24915114d2da6269d26

I'll try to get to this soon.
Assignee: nobody → michael.l.comella
Flags: needinfo?(michael.l.comella)
FYI, these jobs are hidden by bug 1299015.

fwiw, this failure should only appear when:
 1) Patches are uplifted onto beta
 2) A full test is run on beta

Since checkstyle is used to make the code more readable for developers, making the code easier to understand and preventing errors, failures in #2 are irrelevant. Addressing #1, since patches written on beta are low frequency, I feel preventing checkstyle errors on beta is low priority to fix.
Depends on: 1299015
Brian, do you know who might be able to explain how TC secret files work (as per comment 4)?
Flags: needinfo?(bstack)
I'm happy to explain the secrets of secrets to the best of my abilities and then escalate to someone more knowledgeable when we hit the limits of my knowledge. ping me on irc when you've got time.
Flags: needinfo?(bstack)
I talked to Jonas and Dustin. It sounds like the checkstyle task [1] does not contain the TC secrets scope [2] required to access the adjust key secrets [3]. So we need to:
 * Add the scope to the task
 * Make sure mozharness downloads the appropriate secrets (which it may just do [4])

[1]: https://hg.mozilla.org/releases/mozilla-beta/file/6106148d2caa25c93134c88d20f00c4a62e91715/testing/taskcluster/tasks/builds/android_checkstyle.yml#l21
[2]: https://dxr.mozilla.org/mozilla-beta/source/taskcluster/ci/legacy/tasks/builds/firefox_docker_base.yml#8
[3]: https://dxr.mozilla.org/mozilla-central/rev/f0e6cc6360213ba21fd98c887b55fce5c680df68/testing/mozharness/configs/builds/releng_base_linux_32_builds.py#55

[4]: The task cluster task has the following attr:
  "MOZHARNESS_ACTIONS": "get-secrets build multi-l10n update"
Jonas is doubtful mozharness would just download the secrets:

<&jonasfj> hehe, doubtful probably somewhere we have to add an entry to "secret_files", well, unless it uses the morharness file already configured
15:30 I would just expect morharness to complain if it failed to download a secret and couldn't.
15:30 hence, I'm thinking it didn't try
Also, the adjust secrets are declared level 2 (meaning they can't be accessed for try) so it's unlikely I'd be able to push try jobs to make sure this works!

Jonas suggested I might be able to trigger it locally.
(In reply to Michael Comella (:mcomella) [not actively working on fennec: expect slow responses] from comment #15)
> Also, the adjust secrets are declared level 2 (meaning they can't be
> accessed for try) so it's unlikely I'd be able to push try jobs to make sure
> this works!

Spoke too soon: apparently I might be able to change the secret level to 1 and push to try anyway.

In any case, TC provides a default key so configure shouldn't fail (...should we make it fail?) when we don't have permissions (e.g. on try).
We're no longer using legacy configs [1] so it seems it might just be as simple as adding `secrets: true` (like in [2])!

[1]: https://dxr.mozilla.org/mozilla-central/rev/f0e6cc6360213ba21fd98c887b55fce5c680df68/taskcluster/ci/build/android.yml#55
[2]: https://dxr.mozilla.org/mozilla-central/rev/f0e6cc6360213ba21fd98c887b55fce5c680df68/taskcluster/ci/build/linux.yml#21
> In any case, TC provides a default key so configure shouldn't fail (...should we make it fail?)
> when we don't have permissions (e.g. on try).

NI self to answer (or file follow-up).
Flags: needinfo?(michael.l.comella)
Blocks: 1306724
See Also: → bug 1306724
I don't have time to continue working on this but there should be enough info here for some to step in and continue pushing. This isn't high priority until we activate gradle for production builds – blocking bug 1254353.

(In reply to Michael Comella (:mcomella) [not actively working on fennec: expect slow responses] from comment #18)
> > In any case, TC provides a default key so configure shouldn't fail (...should we make it fail?)
> > when we don't have permissions (e.g. on try).
> 
> NI self to answer (or file follow-up).

Filed bug 1306724 to follow up to make sure we fix this too.
Assignee: michael.l.comella → nobody
Blocks: 1254353
Flags: needinfo?(michael.l.comella)
See Also: bug 1306724
Dupe of bug 1317359?
No longer blocks: 1254353
Blocks: 1405378
You need to log in before you can comment on or make changes to this bug.