1509670 - Android 7.0 x86 test tasks sometimes retry because "no space left on device" found in logcat

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Testing, defect, P1)

Description

[Geoff Brown [:gbrown]]

https://taskcluster-artifacts.net/cg-Jt_1lRlmA6VM5DqlO7Q/0/public/logs/live_backing.log

[task 2018-11-24T00:13:35.701Z] 00:13:35     INFO -  11-24 00:10:34.310 W/BatteryStats( 1323): Error writing battery statistics
[task 2018-11-24T00:13:35.701Z] 00:13:35 CRITICAL -  11-24 00:10:34.310 W/BatteryStats( 1323): java.io.FileNotFoundException: /data/system/batterystats.bin.tmp (No space left on device)
[task 2018-11-24T00:13:35.701Z] 00:13:35     INFO -  11-24 00:10:34.310 W/BatteryStats( 1323): 	at java.io.FileOutputStream.open(Native Method)
[task 2018-11-24T00:13:35.701Z] 00:13:35     INFO -  11-24 00:10:34.310 W/BatteryStats( 1323): 	at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
[task 2018-11-24T00:13:35.701Z] 00:13:35     INFO -  11-24 00:10:34.310 W/BatteryStats( 1323): 	at java.io.FileOutputStream.<init>(FileOutputStream.java:169)
[task 2018-11-24T00:13:35.701Z] 00:13:35     INFO -  11-24 00:10:34.310 W/BatteryStats( 1323): 	at com.android.internal.os.BatteryStatsImpl.commitPendingDataToDisk(BatteryStatsImpl.java:9899)
[task 2018-11-24T00:13:35.701Z] 00:13:35     INFO -  11-24 00:10:34.310 W/BatteryStats( 1323): 	at com.android.internal.os.BatteryStatsImpl$5.run(BatteryStatsImpl.java:9880)
[task 2018-11-24T00:13:35.702Z] 00:13:35     INFO -  11-24 00:10:34.310 W/BatteryStats( 1323): 	at android.os.Handler.handleCallback(Handler.java:751)
[task 2018-11-24T00:13:35.702Z] 00:13:35     INFO -  11-24 00:10:34.310 W/BatteryStats( 1323): 	at android.os.Handler.dispatchMessage(Handler.java:95)
[task 2018-11-24T00:13:35.702Z] 00:13:35     INFO -  11-24 00:10:34.310 W/BatteryStats( 1323): 	at android.os.Looper.loop(Looper.java:154)
[task 2018-11-24T00:13:35.702Z] 00:13:35     INFO -  11-24 00:10:34.310 W/BatteryStats( 1323): 	at android.os.HandlerThread.run(HandlerThread.java:61)

[task 2018-11-24T00:13:37.610Z] 00:13:37 CRITICAL - # TBPL RETRY #

Lots of possible solutions: Disable BatteryStats? Filter BatteryStats from logcat? Stop dumping logcat to main log? Remove "No space left on device" from retry regex?

Comment 1

[Geoff Brown [:gbrown]]

Attachment: filter out more logcat messages

Desktop and android-em tests will automatically retry if certain text strings are found in the test log. One of those strings is "No space left on device"; I'm not sure of the history, but it seems like a reasonable response: If we get a bad instance without any storage available, throw it away in hopes that we get a better instance next time (and possible a reboot or re-assignment will resolve the issue for the bad instance).

At issue here is that, for android-em tests, part of the logcat is often included in the test log - at least for mochitests and reftests. If the logcat contains "No space left on device" -- intermittently reported by BatteryStats on the Android 7.0 emulator -- that will trigger a retry unnecessarily.

My solution here is to filter out BatteryStats messages from logcat, and while I'm here, also remove some other message types commonly seen on Android 7.0 but generally not relevant to mozilla tests.

Would you be interested in the BatteryStats messages for power testing on android-hw?


https://treeherder.mozilla.org/#/jobs?repo=try&revision=0e3324f5394f35710489a5e7048372b94390c737

Comment 2

[Bob Clary [:bc] (inactive)]

Comment on attachment 9027973 [details] [diff] [review]
filter out more logcat messages

Review of attachment 9027973 [details] [diff] [review]:
-----------------------------------------------------------------

We might but I haven't had a use case in mind. Let's go ahead and revisit if there is a need in the future.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/d87b95a8c72d
Avoid some android task retries by limiting logcat messages; r=bc

Comment 4

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/d87b95a8c72d