834382 - Disable failing B2G emulator crashtests

Status: RESOLVED FIXED

(Testing :: Reftest, defect)

Description

[Andrew Halberstadt [:ahal]]

Attachment: Patch 1.0 - disable crashtests

We need to disable failing crashtests on various branches before enabling them on emulator arm builds there.

Comment 1

[Andrew Halberstadt [:ahal]]

If we want to run crashtests on the b2g18 branch we'll need to land this there first. I'll let release drivers decide whether we want to enable more tests there or not.

Comment 2

[Jonathan Griffin (:jgriffin)]

Comment on attachment 705997 [details] [diff] [review]
Patch 1.0 - disable crashtests

rubber stamp, applied!

Comment 3

[Andrew Halberstadt [:ahal]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/382dae2711c9

Comment 4

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/382dae2711c9

Comment 5

[Andrew Halberstadt [:ahal]]

Dietrich, I'm a little confused how to proceed. This is a test-only change, but we were asked to refrain from checking them in by our partners as we may break their automation.

Does this mean that I now need to get tef+, leo+ or shira+? Or can I go ahead and land without them. FWIW this patch shouldn't affect anything our partners are doing as it's just some test disabling.

Comment 6

[Andrew Halberstadt [:ahal]]

I found https://etherpad.mozilla.org/B2G-Clarifications, looks like I can go ahead and land this on the b2g18 branch. I'm probably not going to worry about v1.0.0 and v1.0.1 unless people ask for it.

Comment 7

[Andrew Halberstadt [:ahal]]

https://hg.mozilla.org/releases/mozilla-b2g18/rev/54ae714b53d8

Comment 8

[Andrew Halberstadt [:ahal]]

With the retrigger there were a bunch of additional failure on the b2g18 branch that weren't present on m-c. They are hidden on b2g18 for now but we need to disable some more tests.

Comment 9

[Andrew Halberstadt [:ahal]]

Attachment: Patch 2.0 - Disable b2g18 specific failures

This patch disables the failures that show up on tbpl. Because each chunk is timing out, I wouldn't be surprised if we need some follow-up patches to disable further failures.

Comment 10

[Andrew Halberstadt [:ahal]]

https://hg.mozilla.org/releases/mozilla-b2g18/rev/b8d06a2cdca0

I'll wait to see if this greens up crashtests or another patch is needed before closing.

Comment 11

[Andrew Halberstadt [:ahal]]

Still a bunch of load failures (and even a hard timeout with no failures). I'll try running this locally to see if I can get it green there before pushing to b2g18 again.

Comment 12

[Andrew Halberstadt [:ahal]]

Attachment: Patch 3.0 - disable all xul crashtests

For some reason remote xul isn't working on b2g18, even though this line should be fixing it: http://mxr.mozilla.org/mozilla-b2g18/source/layout/tools/reftest/b2g_start_script.js#32

Since afaik we don't care about xul on b2g, I'd rather just disable those tests than fix the problem. This patch greens up crashtests for me locally.

Comment 13

[Jonathan Griffin (:jgriffin)]

Comment on attachment 716138 [details] [diff] [review]
Patch 3.0 - disable all xul crashtests

Review of attachment 716138 [details] [diff] [review]:
-----------------------------------------------------------------

Seems like a good idea.

Comment 14

[Andrew Halberstadt [:ahal]]

https://hg.mozilla.org/releases/mozilla-b2g18/rev/5d43dc542abb

Comment 15

[Andrew Halberstadt [:ahal]]

Looking good!