1118751 - Autophone - PROFILE-ERROR | No crash directory (/data/local/tmp/profile/minidumps) found on remote device for Android 2.3

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Profile Handling, defect)

Description

[Bob Clary [:bc] (inactive)]

Nexus S Android 2.3 devices fail with:

PROFILE-ERROR | ... | No crash directory (/data/local/tmp/profile/minidumps) found on remote device

https://treeherder.allizom.org/#/jobs?repo=mozilla-central&revision=206205dd8bd1&filter-searchStr=autophone

This is due to the new crash processing code in bug 1080783. It may be a race condition or something else.

Comment 1

[Bob Clary [:bc] (inactive)]

snorp, do you think this is a fennec issue or that autophone isn't giving fennec enough time to create the profile properly?

Current TH link: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=6e7b69e6d3eb&filter-searchStr=autophone&exclusion_state=all

Comment 2

[James Willcox (:snorp) ([e-mail]) (he/him)]

Bob it looks like it's running the webapp startup test for everything on 2.3? I see the "WEBAPP STARTUP COMPLETED" message and then it gets killed (via autophone I assume) shortly after.

Comment 3

[Bob Clary [:bc] (inactive)]

snorp, it happens for all three tests on the nexus s. The smoketest and webappstartup kill fennec but the s1s2 shuts down cleanly via the quitter extension. The issue is the failure to create the minidumps directory in the profile which the normal unittests which run in treeherder consider an error. It doesn't happen on all 2.3 devices.

My local nexus-one doesn't have the same problem: https://treeherder.allizom.org/#/jobs?repo=mozilla-central&revision=af476081c7fc&filter-searchStr=autophone

Comment 4

[Bob Clary [:bc] (inactive)]

snorp, can you provide a link to the logcat for smoketest or s1s2test that contains the "WEBAPP STARTUP COMPLETED" message.

Comment 5

[James Willcox (:snorp) ([e-mail]) (he/him)]

I could've sworn it was in this one[0] yesterday, but I don't see it now. That log looks truncated, though.

https://autophone.s3.amazonaws.com/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-api-9/1422368306/autophone-smoketest-1-nexus-s-2-logcat.log

Comment 6

[Bob Clary [:bc] (inactive)]

Attachment: pr 18

This error only serves to make the nexus s tests orange. Let's disable it. In the future if nexus s is no longer supported we can add it back.

Comment 7

[Bob Clary [:bc] (inactive)]

After some discussion in the pr, mcote, gbrown and I decided this is a real issue. I'll try to add more information to this bug so we can try to get it fixed.

Comment 8

[Bob Clary [:bc] (inactive)]

We see this on the Nexus S devices running Android 2.3. I do not see this on my local Nexus One 2.3 (CyanogenMod) devices.

https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=cb5cb9f6943b&exclusion_profile=false&filter-searchStr=autophone

Marking security sensitive due to the tombstone contains uninitialized data in d4. I only see the 3feccccccccccccd on nightly and aurora branches and not beta or release. I presume we don't mark uninitialized data there. Please remove s-s if that is not significant.

https://autophone.s3.amazonaws.com/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android-api-9/1428943897/autophone-s1s2-1-nexus-s-3-tombstone_00.1.txt

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'google/sojua/crespo:2.3.4/GTJ61/129998:user/release-keys'
pid: 10051, tid: 10061  >>> org.mozilla.fennec <<<
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 00000000
 r0 8045a0d0  r1 80421345  r2 00000202  r3 00000000
 r4 8045a0d0  r5 00000000  r6 40008834  r7 000000ad
 r8 00001000  r9 00000000  10 8045a0d0  fp 0000000c
 ip 80459db4  sp 44fff6e0  lr afd1cad5  pc 80421362  cpsr 60000030
 d0  0000000000000000  d1  0000000800000308
 d2  0000005c00000018  d3  003daea800000040
 d4  3feccccccccccccd  d5  4186800000000000
 d6  4038000000000000  d7  0000000000000000
 d8  0000000000000000  d9  0000000000000000
 d10 0000000000000000  d11 0000000000000000
 d12 0000000000000000  d13 0000000000000000
 d14 0000000000000000  d15 0000000000000000
 d16 0000000000000000  d17 3fe0000000000000
 d18 4000000000000000  d19 bf9861b842ee3fc9
 d20 3f11565ab512bf53  d21 bebbbd282e3b04a8
 d22 3ff0000000000000  d23 3fef3ffffffffffe
 d24 3e66376972bea4d0  d25 3fc39a09d078c69f
 d26 3feff86800000000  d27 bec4bea35ece3901
 d28 c000048ed9d151f2  d29 bf6239ac63a487b9
 d30 bc06439c9a98bc08  d31 be125224f7200000
 scr 60000013

Comment 9

[Bob Clary [:bc] (inactive)]

fyi, my local nexus ones do show the same crash via tombstone but they don't show the missing minidumps directory error.

https://autophone-dev.s3.amazonaws.com/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android-api-9/1429024157/autophone-s1s2-1-nexus-one-2-tombstone_00.1.txt

Comment 10

[James Willcox (:snorp) ([e-mail]) (he/him)]

I think this may be the same as bug 1152308

Comment 11

[Andrew McCreight [:mccr8]]

I'm going to mark this sec-moderate, because it sounds like this is some kind of shutdown weirdness which would be hard to exploit.

Comment 12

[Bob Clary [:bc] (inactive)]

Attachment: TypeError: this._currentEnvironment.profile is undefined

bug 1152308 fixed the crash for Android 2.3. The tombstones are no longer being created but the minidumps directory is still not being created. The only thing I can see in logcat is the "TypeError: this._currentEnvironment.profile is undefined".

See https://autophone.s3.amazonaws.com/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android-api-9/1429500692/autophone-s1s2-1-nexus-s-2-logcat.log

snorp: Any thoughts?

Comment 13

[James Willcox (:snorp) ([e-mail]) (he/him)]

That telemetry error should be harmless. I see it here too (bug 1154902). What's the issue with the minidump directory? I don't think it will exist unless there is a crash, right?

Comment 14

[Bob Clary [:bc] (inactive)]

Actually, it is normally created when the profile is created. The regular in-tree automation checks for its existence and flags it as an error. I modelled the autophone crash handling on that. See https://dxr.mozilla.org/mozilla-central/source/build/mobile/remoteautomation.py#196

Comment 15

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to Bob Clary [:bc:] from comment #14)
> Actually, it is normally created when the profile is created. The regular
> in-tree automation checks for its existence and flags it as an error. I
> modelled the autophone crash handling on that. See
> https://dxr.mozilla.org/mozilla-central/source/build/mobile/remoteautomation.
> py#196

Do you know for sure that the directory doesn't exist? How are you checking?

Comment 16

[Bob Clary [:bc] (inactive)]

The remote_dump_dir is defined in
https://github.com/mozilla/autophone/blob/master/autophonecrash.py#L55
as the profile directory/minidumps

The existence of the directory is checked in:
https://github.com/mozilla/autophone/blob/master/autophonecrash.py#L284

There may be an issue with permissions. I'll work up a patch to specify root on the calls which access anything in the profile.

Comment 17

[Bob Clary [:bc] (inactive)]

Attachment: pr 25

For cases where I used root=True in functions where I accessed the profile, etc I elevated the root=True to the function argument list to centralize the specification but not change the behavior. This is preparation for work to remove the rooted requirement.

The relevant changes for this bug are in autophonecrash.py in get_crashes where I add a root requirement when checking if minidumps exists, and add a chmod for the minidumps directory before trying to pull it.

We will have to deploy this to see if it actually eliminates the missing minidumps error.

Comment 18

[Bob Clary [:bc] (inactive)]

https://github.com/mozilla/autophone/commit/bd0ccdbe1d7422a23ccfec8c75d143e0203c2e89