875640 - Fix and re-enable crashtest for bug 855796 on Android

Status: VERIFIED FIXED

(Core :: WebRTC: Signaling, defect)

Description

[Phil Ringnalda (:philor)]

If you look at the bulk of bug 689856 between the time bug 855796 landed and now, you'll see that it's crashtest runs, silently stopping during or right after 855796.html. If we'd gotten around to filing on the things like

https://tbpl.mozilla.org/php/getParsedLog.php?id=23346764&tree=Mozilla-Inbound

REFTEST TEST-START | http://10.250.49.157:30053/tests/dom/media/tests/crashtests/855796.html | 564 / 2452 (23%)

INFO | automation.py | Application ran for: 0:04:09.417291
INFO | zombiecheck | Reading PID log: /tmp/tmpMBaTbMpidlog
/data/anr/traces.txt not found
PROCESS-CRASH | java-exception | java.lang.RuntimeException: Unable to start service com.mozilla.watcher.WatcherService@442ce890 with null: java.lang.NullPointerExceptionat android.app.ActivityThread.handleServiceArgs(ActivityThread.java:3063)

you'd see that they were too. Be nice to have it back running on Android, but right now it's just too frequently and too violently failing, taking down the whole run, to tolerate.

Comment 1

[Phil Ringnalda (:philor)]

skip-if(Android) in https://hg.mozilla.org/integration/mozilla-inbound/rev/09c1c5ae1c1c

Comment 2

[Jason Smith [:jsmith]]

gcp - any ideas?

Comment 3

[Gian-Carlo Pascutto [:gcp]]

Looks like an OOM. No errors from the low-level Android WebRTC stack, just this, which suggests the C++ core code is eating all RAM:

05-23 21:52:50.556 I/ActivityManager( 1017): Process android.process.acore (pid 1266) has died.
05-23 21:52:50.976 I/ActivityManager( 1017): Process com.android.settings (pid 1226) has died.
05-23 21:52:50.986 I/ActivityManager( 1017): Process com.svox.pico (pid 1570) has died.
05-23 21:52:50.986 I/ActivityManager( 1017): Process com.android.launcher (pid 1219) has died.
05-23 21:52:50.986 I/ActivityManager( 1017): Process com.android.providers.calendar (pid 1393) has died.
05-23 21:52:50.986 I/ActivityManager( 1017): Process com.android.email (pid 1348) has died.
05-23 21:52:50.986 I/ActivityManager( 1017): Process com.android.music (pid 1420) has died.
05-23 21:52:50.986 I/ActivityManager( 1017): Process android.process.media (pid 1339) has died.
05-23 21:52:50.986 I/ActivityManager( 1017): Process com.android.quicksearchbox (pid 1431) has died.
05-23 21:52:50.986 I/ActivityManager( 1017): Process com.mozilla.watcher (pid 1411) has died.
05-23 21:52:51.066 I/WindowManager( 1017): WIN DEATH: Window{4443cf50 com.android.launcher/com.android.launcher2.Launcher paused=false}
05-23 21:52:51.086 W/ActivityManager( 1017): Scheduling restart of crashed service com.mozilla.watcher/.WatcherService in 5000ms
05-23 21:52:51.266 I/WindowManager( 1017): WIN DEATH: Window{446208b0 com.mozilla.SUTAgentAndroid/com.mozilla.SUTAgentAndroid.SUTAgentAndroid paused=false}
05-23 21:52:51.266 I/WindowManager( 1017): WIN DEATH: Window{445e7390 SurfaceView paused=false}
05-23 21:52:51.276 I/ActivityManager( 1017): Process com.android.deskclock (pid 1325) has died.
05-23 21:52:51.276 I/ActivityManager( 1017): Process com.android.defcontainer (pid 1559) has died.
05-23 21:52:51.276 I/ActivityManager( 1017): Process org.mozilla.f3nn3c.UpdateService (pid 1770) has died.
05-23 21:52:51.276 I/ActivityManager( 1017): Process com.android.mms (pid 1362) has died.
05-23 21:52:51.276 I/WindowManager( 1017): WIN DEATH: Window{445e57f8 org.mozilla.fennec/org.mozilla.fennec.App paused=false}
05-23 21:52:51.276 I/ActivityManager( 1017): Process com.android.bluetooth (pid 1384) has died.
05-23 21:52:51.276 I/ActivityManager( 1017): Process com.cooliris.media (pid 1440) has died.
05-23 21:52:51.276 I/ActivityManager( 1017): Low Memory: No more background processes.
05-23 21:52:51.276 I/ActivityManager( 1017): Process com.mozilla.SUTAgentAndroid (pid 1497) has died.
05-23 21:52:51.516 W/ActivityManager( 1017): Scheduling restart of crashed service com.mozilla.SUTAgentAndroid/.service.ASMozStub in 14805ms
05-23 21:52:51.516 I/ActivityManager( 1017): Low Memory: No more background processes.

Comment 4

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/09c1c5ae1c1c

Comment 5

[Jason Smith [:jsmith]]

Randell - Any thoughts what might be failing here in the C++ area of the stack?

Comment 6

[Jason Smith [:jsmith]]

Not a blocker for pref on.

Comment 7

[Jason Smith [:jsmith]]

Talking with Randell - someone needs to debug this to find out why this is happening.

Ethan - Since you wrote the original patch on bug 855796, could you look into this to find out why we are OOMing when we run the crashtest from bug 855796?

Comment 8

[Ethan Hugg [:ehugg]]

I don't see anything specific to this test, but perhaps it's related to bug 879743, the fake=true memory leak.  I'll see what I can find there.

Comment 9

[Jason Smith [:jsmith]]

Moving to non-blocking until we get confirmation this could happen during production use with real users hitting this.

Comment 10

[Ethan Hugg [:ehugg]]

I think this would be fixed by the patch to bug 879743 as evidenced by this Try run:

https://tbpl.mozilla.org/?tree=Try&rev=5e0de529c370

Comment 11

[Ethan Hugg [:ehugg]]

Attachment: re-enable crashtest for Android

Comment 12

[Ethan Hugg [:ehugg]]

Bug 879743 is now on M-C.  Running a try re-enabling this:

https://tbpl.mozilla.org/?tree=Try&rev=6b037bc397d0

Comment 13

[Ethan Hugg [:ehugg]]

Comment on attachment 765669 [details] [diff] [review]
re-enable crashtest for Android


The Try in comment 12 came up green with 3 retries.  We should turn this one back on.

Comment 14

[Phil Ringnalda (:philor)]

Comment on attachment 765669 [details] [diff] [review]
re-enable crashtest for Android

\o/

Comment 15

[Ethan Hugg [:ehugg]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/17c4c99f1ac2

Comment 16

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/17c4c99f1ac2

Comment 17

[Jason Smith [:jsmith]]

Verified via landing with crashtest enabled and running cleanly.