1359288 - Every Android jsreftest run, green or not, includes a "REFTEST ERROR | Force stop because we keep running into test "None""

Status: RESOLVED FIXED

(Testing :: Reftest, enhancement)

Description

[Phil Ringnalda (:philor)]

Okay, strictly speaking I don't know about "every" since I only checked a random selection of the 40 opt and 100 debug chunks, but every green run I checked looked like https://treeherder.mozilla.org/logviewer.html#?job_id=93767161&repo=mozilla-inbound - particularly bad in the way I first found out about it, with orange runs which had an actual test failure which failed to produce any sane parsable output, so that the "REFTEST ERROR | Force stop because we keep running into test "None"" was the only thing that Treeherder thought had gone wrong.

Comment 1

[Bob Clary [:bc] (inactive)]

I've seen it as well.

Comment 2

[Geoff Brown [:gbrown]]

:shinglyu - Can you have a look at this?

Comment 3

[Shing Lyu [:shinglyu]]

Sorry for the confusion. In bug 1344991 I made reftest able to recover from a crash, but to prevent the test from re-running a crashed test forever, I added the fuse mechanism that produces this error. The root cause is that I only enable the recovery feature on desktop, but used a stub for Android.

The best solution is to add the crash recovery feature to android as well, but I can't estimate how long it will take. If this is very urgent I can suppress the error on Android platform.

Comment 4

[Bob Clary [:bc] (inactive)]

I would say that if the error can be suppressed without causing negative effects for android then you should do it immediately. If not, then the original patch should be backed out until this is fixed.

Comment 5

[Shing Lyu [:shinglyu]]

Attachment: Bug 1359288 - Enable Android reftest to resume from crashes.

Review commit: https://reviewboard.mozilla.org/r/137760/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/137760/

Comment 6

[Shing Lyu [:shinglyu]]

Bob,

The fix was simpler then I expected. So I not pushed a patch that will fix this permanently. 

Dbaron,

Would you mind reviewing this for me? This was a followup from the previous resume-from-crash reftest feature. But I didn't do the Android part because I'm not familiar with the code for Android back then. Now I fixed it and it will help reduce the noise for Android try server log. Thanks!

Comment 7

[Shing Lyu [:shinglyu]]

Here is a try run without the errors: https://treeherder.mozilla.org/#/jobs?repo=try&revision=78a285dadd1def3b8f01426d76bda90c0ede96fe

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8866153 [details]
Bug 1359288 - Enable Android reftest to resume from crashes.

https://reviewboard.mozilla.org/r/137760/#review140988

thanks, I agree this looks simple

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/66513d27bc96
Enable Android reftest to resume from crashes. r=jmaher

Comment 10

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Backed out for throwing exceptions during Android robocop tests:

https://hg.mozilla.org/integration/autoland/rev/cede71157daf70061897b0d5d8ef723f48bf7620

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=66513d27bc96ec905604b45056ba0a40b622e4c2&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=success&filter-searchStr=robocop
Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=98631969&repo=autoland

[task 2017-05-12T05:46:17.425939Z] 05:46:17     INFO -  SUITE-START | Running 14 tests
[task 2017-05-12T05:50:37.236250Z] 05:50:37     INFO -  INFO | automation.py | Application pid: 0
[task 2017-05-12T05:50:37.573622Z] 05:50:37     INFO -  SimpleTest START
[task 2017-05-12T05:50:37.574364Z] 05:50:37     INFO -  TEST-START | testBrowserDiscovery
[task 2017-05-12T05:50:37.574453Z] 05:50:37     INFO -  GECKO | EventExpecter: no longer listening for Gecko:Ready
[task 2017-05-12T05:50:37.589927Z] 05:50:37     INFO -  GECKO | EventExpecter: no longer listening for Robocop:Java
[task 2017-05-12T05:50:37.590006Z] 05:50:37     INFO -  TEST-OK | testBrowserDiscovery | took 74596ms
[task 2017-05-12T05:50:37.590734Z] 05:50:37     INFO -  TEST-START | Shutdown
[task 2017-05-12T05:50:37.591117Z] 05:50:37     INFO -  Passed: 201
[task 2017-05-12T05:50:37.591809Z] 05:50:37     INFO -  Failed: 0
[task 2017-05-12T05:50:37.591865Z] 05:50:37     INFO -  Todo: 0
[task 2017-05-12T05:50:37.592047Z] 05:50:37     INFO -  SimpleTest FINISHED
[task 2017-05-12T05:50:59.054270Z] 05:50:59     INFO -  INFO | automation.py | Application ran for: 0:04:40.645413
[task 2017-05-12T05:50:59.054526Z] 05:50:59     INFO -  INFO | zombiecheck | Reading PID log: /tmp/tmpCCSpIFpidlog
[task 2017-05-12T05:50:59.643946Z] 05:50:59     INFO -  /data/tombstones does not exist; tombstone check skipped
[task 2017-05-12T05:51:00.355487Z] 05:51:00     INFO -  0 ERROR Automation Error: Exception caught while running tests
[task 2017-05-12T05:51:00.355781Z] 05:51:00     INFO -  Traceback (most recent call last):
[task 2017-05-12T05:51:00.356080Z] 05:51:00     INFO -    File "/home/worker/workspace/build/tests/mochitest/runrobocop.py", line 465, in runSingleTest
[task 2017-05-12T05:51:00.356729Z] 05:51:00     INFO -      self.log.debug("runApp completes with status %d" % result)
[task 2017-05-12T05:51:00.356835Z] 05:51:00     INFO -  TypeError: not all arguments converted during string formatting
[task 2017-05-12T05:54:48.370052Z] 05:54:48     INFO -  INFO | automation.py | Application pid: 0

Comment 11

[Shing Lyu [:shinglyu]]

Comment on attachment 8866153 [details]
Bug 1359288 - Enable Android reftest to resume from crashes.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/137760/diff/1-2/

Comment 12

[Shing Lyu [:shinglyu]]

Comment on attachment 8866153 [details]
Bug 1359288 - Enable Android reftest to resume from crashes.

I fixed the reason for the autophone failures. Autophone mochitest and robocop also inherits the runApp() API, so I change them to fit the reftest runApp() API, but ignoring the second argument to keep the behavior the same.

Try result:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=ef4a93d3fccd0a93a9addd8f8047788f1b86522a

There are some Mdm1 fails but they also appear in other m-c pushes, so looks like intermittents.

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8866153 [details]
Bug 1359288 - Enable Android reftest to resume from crashes.

https://reviewboard.mozilla.org/r/137760/#review142982

thanks for the update!

Comment 14

[Shing Lyu [:shinglyu]]

Hi Bob, 

We have some gl* errors here, are they known issues for Autophone test? I didn't see them being run very often. Do you think this patch is OK to land? Is there any other android tests that might be affected?

https://treeherder.mozilla.org/#/jobs?repo=try&revision=71324e70156f6ba9cf6b8b98740953c215b7b7a9

Comment 15

[Bob Clary [:bc] (inactive)]

The gl conf tests are mochitests and shouldn't be affected by reftest only changes. The failures for the gl tests are already filed under Bug 1359078. The Cdm{1..4}, Rov and Rwv tests are reftest based and are green on your try.

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3241038a0214
Enable Android reftest to resume from crashes. r=jmaher

Comment 17

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/3241038a0214