1261270 - Various Android robocop tests are going to permafail when Gecko 48 merges to Aurora

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Testing, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

[Tracking Requested - why for this release]: Android robocop permafails when Gecko 48 merges to Aurora.

Geoff, any idea what might be going on here?

https://treeherder.mozilla.org/logviewer.html#?job_id=18725473&repo=try
https://treeherder.mozilla.org/logviewer.html#?job_id=18725437&repo=try

Comment 1

[Geoff Brown [:gbrown]]

Sorry, not really. 

These tests were failing for a long time; I recently enabled them when I noticed they were passing - bug 1144874 - but I have no idea what changed, or why they fail on Aurora.

Comment 2

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Could APZ be held behind a nightly flag?

Comment 3

[Geoff Brown [:gbrown]]

(In reply to Michael Comella (:mcomella) from comment #2)
> Could APZ be held behind a nightly flag?

Good idea! http://hg.mozilla.org/mozilla-central/annotate/c40c0b2f3b4c/mobile/android/confvars.sh#l100

Comment 4

[Geoff Brown [:gbrown]]

I verified that testAxisLocking and testPanCorrectness fail on 48 if APZ is disabled.

What needs to happen here? Can those tests be disabled on Aurora after the merge, or do the tests need to be disabled on trunk until APZ rides the trains?

Comment 5

[Ryan VanderMeulen [:RyanVM]]

I'll double-check, but I think mozinfo actually has nightly_build in it, so we can skip it that way. Though a fails-if = !nightly_build is probably better so we'll remember to re-enable the tests when APZ is ready to ride the trains.

Comment 6

[Ryan VanderMeulen [:RyanVM]]

Turns out mozinfo has a release_build option, but not nightly_build. Seems like we could easily add it, though. Which I think is a reasonable option as long as we use it with a fails-if condition so the annotation doesn't stay there in perpetuity.

Comment 7

[Ryan VanderMeulen [:RyanVM]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=821003b63cc0

Comment 8

[Ryan VanderMeulen [:RyanVM]]

Excitingly, while fail-if is perfectly valid syntax for regular mochitests, it apparently isn't for robocop. My patch with |fail-if = !nightly_build| doesn't work, but skip-if does.

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Going off https://hg.mozilla.org/mozilla-central/rev/102371151dc6, I guess that's expected though. *sigh*

Comment 10

[Ryan VanderMeulen [:RyanVM]]

Attachment: skip the 3 failing tests on !nightly_build

Looks like this is the best we can do here. I've confirmed that this makes robocop green on Try simulations.

Comment 11

[Geoff Brown [:gbrown]]

Comment on attachment 8738358 [details] [diff] [review]
skip the 3 failing tests on !nightly_build

Review of attachment 8738358 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks! I'll look into the robocop fail-if situation; for now, this is the best we can do.

Comment 12

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/86c187556f7a

Comment 13

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/86c187556f7a

Comment 14

[Ryan VanderMeulen [:RyanVM]]

Looks like bug 1265418 removed the nightly ifdefs for APZ.

Comment 15

[Geoff Brown [:gbrown]]

Yes. I think we can back out the test skipping.

Comment 16

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/2a9e61c94ffd

Comment 17

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/2a9e61c94ffd