1362424 - linux64-qr debug R-e10s(R8,Ru8) permafail in editor/reftests/672709-ref.html, editor/reftests/338427-1.html

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, defect, P3)

Description

[Kartikaya Gupta (email:[e-mail])]

These tests started permafailing with a recent merge from m-c to graphics. The failure doesn't happen on m-c itself, so it's some interaction between stuff on m-c and stuff on graphics. I'm gonna try bisecting to see where it's coming from.

Comment 1

[Kartikaya Gupta (email:[e-mail])]

For reference, the m-c range is [1] and the graphics range is [2].

[1] https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=82c2d17e74ef&tochange=33b92d9c4056
[2] https://hg.mozilla.org/projects/graphics/pushloghtml?fromchange=29f8f98fb7ba&tochange=cbd6439cd06f (easier to see if you collapse the m-c merges)

Comment 2

[Kartikaya Gupta (email:[e-mail])]

First bisection is at [1]. This narrows the m-c range to [2].

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=1225d0f4829aeac05082272e674cee908a8b8544
[2] https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=fbad88a61127&tochange=33b92d9c4056

Comment 3

[Kartikaya Gupta (email:[e-mail])]

First bisection of gfx is at [1]. This narrows the gfx range to [2].

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=bfee25505b20fe2ebf2b0f54e6a8430f3ab2d85d
[2] https://hg.mozilla.org/projects/graphics/pushloghtml?fromchange=29f8f98fb7ba&tochange=ae9f92d42542

Comment 4

[Kartikaya Gupta (email:[e-mail])]

m-c bisection at [1] narrows range to [2].
gfx bisection at [3] narrows range to [4].

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=7b5118a1916ba0a1676a49c08c8d54a57b517e34
[2] https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=fbad88a61127&tochange=4a6a71f4aa22
[3] https://treeherder.mozilla.org/#/jobs?repo=try&revision=4323736f2d3dc4cc0566e70b816aa4cdf733cd6b
[4] https://hg.mozilla.org/projects/graphics/pushloghtml?fromchange=29f8f98fb7ba&tochange=8f1d4b0a03e4

Comment 5

[Kartikaya Gupta (email:[e-mail])]

gfx bisection at [1] narrows range to [2].
m-c bisection at [3] narrows range to [4].
gfx bisection at [5] further narrows range to [6].

So on the graphics side this seems to be caused by bug 1360001.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=7e25590521ae965fa397f47ab317cad588255da6
[2] https://hg.mozilla.org/projects/graphics/pushloghtml?fromchange=eb77f99d677f&tochange=8f1d4b0a03e4
[3] https://treeherder.mozilla.org/#/jobs?repo=try&revision=4c6d799b7b9c93da823640619332191db3d0f279
[4] https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=fbad88a61127&tochange=d19472763d1d
[5] https://treeherder.mozilla.org/#/jobs?repo=try&revision=aa8b63554181356ac13d9816d65d8b283c6c767d
[6] https://hg.mozilla.org/projects/graphics/pushloghtml?fromchange=9bfbae06d8e2&tochange=8f1d4b0a03e4

Comment 6

[Jeff Muizelaar [:jrmuizel]]

It's hard to see how that could be possible. Bug 1360001 should have any noticeable behaviour impact.

Comment 7

[Kartikaya Gupta (email:[e-mail])]

Yeah I'm not sure. I'm noticing this failure on m-c now as well, even though the graphics branch hasn't been merged to m-c. So that seems to indicate that it's just an intermittent failure that started on m-c but happens more frequently when combined with gfx. I'll finish bisecting the m-c regression.

bisection at [1] narrows range to [2].

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=ad95ad2b02dc2c8c7304e20eb0ebc1193fb828a9
[2] https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=78b648c9c7d1&tochange=9c7f749e9015

Comment 8

[Intermittent Failures Robot]

12 failures in 770 pushes (0.016 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* graphics: 12

Platform breakdown:
* linux64-qr: 12

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1362424&startday=2017-05-01&endday=2017-05-07&tree=all

Comment 9

[Kartikaya Gupta (email:[e-mail])]

More bisections [1][3] narrow range [2][4] 

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=2d28d1b278dada3e0bc3e30d2de59d45a2f7eb5f
[2] https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=78b648c9c7d1&tochange=2877aa1e02c8
[3] https://treeherder.mozilla.org/#/jobs?repo=try&revision=8a1022a40eeedc625cfe15069d698d1246fb7508
[4] https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=78b648c9c7d1&tochange=328fbb1a7bb0

Comment 10

[Kartikaya Gupta (email:[e-mail])]

And final bisection at [1] narrows range to [2], which pins it on bug 1359952.

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=b1c30d08d41c068189bea770b4acad7fa46ea8a7
[2] https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=78b648c9c7d1&tochange=c06620d52d5f

I'm doing some retriggers on the linux32-e10s-Ru8 job on autoland around where bug 1359952 landed to see if I can confirm this regression range. Stay tuned:
https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&tochange=2877aa1e02c8&fromchange=cba7bc0b2e18a24ea40719bc1bcf2b26701de6ee&filter-searchStr=linux32%20debug%20reftest%20no-accel

Comment 11

[Kartikaya Gupta (email:[e-mail])]

Shoot. Looks like the bisection produced an incorrect result, it's happening on autoland even before bug 1359952 landed. Not sure where to go from here :(

FWIW bug 1361820 is being used for the intermittent failures on m-c.

Comment 12

[Kartikaya Gupta (email:[e-mail])]

I took a closer look at the failures. It's almost certainly being caused by bug 1354641. The spellchecking code is taking too long, causing the reftest to time out. And then it leaves WR in a bad state causing the next test (338427-1.html) to also fail. I'm going to skip the bad test to get our R8 job green again, and then try to get the QR reftests running on autoland and inbound as well to catch these regressions sooner rather than wasting time bisecting afterwards.

Try push with the test skipped: https://treeherder.mozilla.org/#/jobs?repo=try&revision=0e62aeb1c24450f7b8dd3e5303f7e506bed8cbeb

Comment 13

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1362424 - Skip a spellchecking reftest that is now permafailing on QuantumRender builds.

This test is failing intermittently even on non-QuantumRender builds, and bug
1361820 is tracking that issue. Once that is fixed we should be able to re-enable
this test.

Review commit: https://reviewboard.mozilla.org/r/137254/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/137254/

Comment 14

[Jeff Muizelaar [:jrmuizel]]

Comment on attachment 8865641 [details]
Bug 1362424 - Skip a spellchecking reftest that is now permafailing on QuantumRender builds.

https://reviewboard.mozilla.org/r/137254/#review140306

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e426d513c19c
Skip a spellchecking reftest that is now permafailing on QuantumRender builds. r=jrmuizel

Comment 16

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/e426d513c19c

Comment 17

[(no longer active)]

This can be re-enabled now, FWIW.  Bug 1354641 was backed out.

Comment 18

[Kartikaya Gupta (email:[e-mail])]

Looks like it was accidentally re-enabled already in https://hg.mozilla.org/mozilla-central/rev/d5b141b9583e#l3.126 by an unrelated stylo change. So nothing to do here.

Comment 19

[Intermittent Failures Robot]

5 failures in 879 pushes (0.006 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* graphics: 5

Platform breakdown:
* linux64-qr: 5

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1362424&startday=2017-05-08&endday=2017-05-14&tree=all