1277390 - Mochitest autophone permafail with double-digit number of env vars

Status: RESOLVED WORKSFORME

(Firefox for Android Graveyard :: Testing, defect)

Description

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

We pass values to APKs during testing through the starting Intent: each key is env# where # is an increasing number starting at 0.

In bug 1277214 comment 0, I experienced perma-failures because I added an env var that created env10 during mochitest autophone tests (autophone-mochitest-dom-browser-element & autophone-mochitest-toolkit-widgets). I assume this causes an issue because we match `env[0-9]` but not `env[0-9]+`. In bug 1277214, I opted for the quick fix (don't add a new env var) but we should fix the real issue here.

Unfortunately, it's unclear to me whether this might be a problem with the APK or the infrastructure. In all my searches, however, it seems we correctly handle the env vars in code.

Bob, do you have any ideas what might be going on here from an infrastructure perspective?

Comment 1

[Bob Clary [:bc] (inactive)]

I'm not sure what is going on. I'm running a series of tests on inbound and am not seeing the infrastructure failures there.

https://treeherder.allizom.org/#/jobs?repo=mozilla-inbound&exclusion_profile=false&filter-searchStr=autophone&filter-tier=1&filter-tier=2&filter-tier=3&fromchange=3d68250b133166b7d65dc99c963fac5fa0ef1439

In particular:

https://treeherder.allizom.org/#/jobs?repo=mozilla-inbound&revision=3d68250b133166b7d65dc99c963fac5fa0ef1439&exclusion_profile=false&filter-tier=1&filter-tier=2&filter-tier=3&filter-searchStr=autophone

https://autophone-dev.s3.amazonaws.com/pub/mobile/tinderbox-builds/mozilla-inbound-android-api-15-debug/1464873112/mochitest-dom-media-mochitests-dom-media-settings.ini-1-samsung-gs3-3-257543f6-616a-49a3-9ac7-fa3a004becdf-logcat.log

shows

06-02 06:53:00.987 D/GeckoLoader( 6167): Gecko environment env0: MOZ_CRASHREPORTER=1
06-02 06:53:00.987 D/GeckoLoader( 6167): env1: XPCOM_DEBUG_BREAK=stack
06-02 06:53:00.987 D/GeckoLoader( 6167): env2: R_LOG_VERBOSE=1
06-02 06:53:00.987 D/GeckoLoader( 6167): env3: MOZ_DISABLE_SWITCHBOARD=1
06-02 06:53:00.987 D/GeckoLoader( 6167): env4: MOZ_DISABLE_NONLOCAL_CONNECTIONS=1
06-02 06:53:00.987 D/GeckoLoader( 6167): env5: MOZ_LOG_FILE=/sdcard/tests/autophone/mozlog/moz.log
06-02 06:53:00.987 D/GeckoLoader( 6167): env6: DISABLE_UNSAFE_CPOW_WARNINGS=1
06-02 06:53:00.987 D/GeckoLoader( 6167): env7: R_LOG_DESTINATION=stderr
06-02 06:53:00.987 D/GeckoLoader( 6167): env8: MOZ_CRASHREPORTER_NO_REPORT=1
06-02 06:53:00.987 D/GeckoLoader( 6167): env9: MOZ_DISABLE_TELEMETRY=1
06-02 06:53:00.987 D/GeckoLoader( 6167): env10: NO_EM_RESTART=1
06-02 06:53:00.987 D/GeckoLoader( 6167): env11: MOZ_PROCESS_LOG=/tmp/tmpg35pT0pidlog
06-02 06:53:00.987 D/GeckoLoader( 6167): env12: R_LOG_LEVEL=6
06-02 06:53:00.987 D/GeckoLoader( 6167): env13: MOZ_HIDE_RESULTS_TABLE=1
06-02 06:53:00.987 D/GeckoLoader( 6167): env14: null

The existence of more than 10 env variables doesn't appear to be an issue for my gs3 running 4.0.4.

I'm still waiting on opt results but the oranges appear to be the normal failures we have been seeing.

Unfortunately your try run in bug 1277214 comment 14 had an unrelated build failure and didn't run.

Comment 2

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

It's possible my issue in bug 1277214 was that the env var I added (MOZ_DISABLE_TELEMETRY) clashed with other values, but dxr doesn't turn up anything:
  https://dxr.mozilla.org/mozilla-central/search?q=MOZ_DISABLE_TELEMETRY&=mozilla-central&redirect=true

(In reply to Bob Clary [:bc:] from comment #1)
> Unfortunately your try run in bug 1277214 comment 14 had an unrelated build
> failure and didn't run.

fwiw, bug 1277214 comment 9 is the passing run where I removed the extra env var whereas bug 1277214 comment 6 is an example failing run.

Comment 3

[Bob Clary [:bc] (inactive)]

Unfortunately Autophone isn't capturing the logcat properly in the event of this exception. Let me do a couple of runs today and see if I can fix that and get a handle on what is going on.

Comment 4

[Bob Clary [:bc] (inactive)]

:mcomella: bug 1277214 has landed and though the some of the Autophone unit tests are orange that is unrelated to the exceptions this bug was originally about. Is there any thing outstanding on this bug from your point of view? It appears fixed to me.

Comment 5

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

(In reply to Bob Clary [:bc:] from comment #4)
> :mcomella: bug 1277214 has landed and though the some of the Autophone unit
> tests are orange that is unrelated to the exceptions this bug was originally
> about. Is there any thing outstanding on this bug from your point of view?
> It appears fixed to me.

bug 1277214 didn't fix the issue I filed this bug for, which is that I was getting autophone permafail because (I inferred) there were 10+ env vars specified in build/mobile/remoteautomation.py. That being said, it's possible that this isn't actually why we were crashing and my fix for bug 1277214 was just luck. We could try:
 * Adding an env var directly after MOZ_IN_AUTOMATION (e.g. https://dxr.mozilla.org/mozilla-central/rev/1828937da9493b2cd54862b9c520b2ba5c7db92b/build/mobile/remoteautomation.py#94), which is where I added my new env var – perhaps it's an ordering issue
 * Adding an env var called MOZ_DISABLE_TELEMETRY, which might conflict with other env vars

However, if you can't reproduce (e.g. comment 1), it sounds like WORKSFORME (unless it breaks again).

Comment 6

[Bob Clary [:bc] (inactive)]

I added an additional env var and ran the unit tests locally and didn't run into exception problems. Not sure how informative that is since we already had more than 10. Lets WFM and move on.