1607620 - Permafailing REFTEST TEST-UNEXPECTED-FAIL | layout/reftests/mathml/mfenced-12.html == layout/reftests/mathml/mfenced-12-ref.html | image comparison, max difference: 255, number of differing pixels: 164

Status: RESOLVED FIXED

(Core :: MathML, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=283915091&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Niig-TILRReigpF9TvZIwQ/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Niig-TILRReigpF9TvZIwQ/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

---

[task 2020-01-07T23:26:18.183Z] 23:26:18 INFO - REFTEST TEST-START | file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/mathml/mfenced-12.html == file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/mathml/mfenced-12-ref.html
[task 2020-01-07T23:26:18.184Z] 23:26:18 INFO - REFTEST INFO | RESTORE PREFERENCE pref(mathml.mfenced_element.disabled,true)
[task 2020-01-07T23:26:18.185Z] 23:26:18 INFO - REFTEST INFO | SET PREFERENCE pref(mathml.mfenced_element.disabled,false)
[task 2020-01-07T23:26:18.186Z] 23:26:18 INFO - REFTEST TEST-LOAD | file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/mathml/mfenced-12.html | 51 / 384 (13%)
[task 2020-01-07T23:26:18.527Z] 23:26:18 INFO - REFTEST INFO | RESTORE PREFERENCE pref(mathml.mfenced_element.disabled,true)
[task 2020-01-07T23:26:18.528Z] 23:26:18 INFO - REFTEST INFO | SET PREFERENCE pref(mathml.mfenced_element.disabled,false)
[task 2020-01-07T23:26:18.529Z] 23:26:18 INFO - REFTEST TEST-LOAD | file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/mathml/mfenced-12-ref.html | 51 / 384 (13%)
[task 2020-01-07T23:26:18.926Z] 23:26:18 INFO - REFTEST TEST-UNEXPECTED-PASS | file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/mathml/mfenced-12.html == file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/mathml/mfenced-12-ref.html | image comparison, max difference: 0, number of differing pixels: 0

Comment 1

[Intermittent Failures Robot]

22 failures in 795 pushes (0.028 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 22

Platform breakdown:

• linux1804-64-asan-qr: 22

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607620&startday=2020-01-07&endday=2020-01-07&tree=all

Comment 2

[Cristina Coroiu [:ccoroiu]]

Retriggers:
https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=linux1804-64-asan-qr%2Copt%2Creftests%2Ctest-linux1804-64-asan-qr%2Fopt-reftest-e10s-2%2Cr%28r2%29&fromchange=9f0cfd2aaa21590530f222b71ce648a6bf80707e&selectedJob=283977012

Comment 3

[Ting-Yu Lin [:TYLin] (PDT, UTC-7)]

mathml/mfenced-12.html is already unexpected pass on the push prior to mine, but the test was in R5 https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=39c4e3bab99a7dc933cf97593157204b4a280acd&selectedJob=283957486

Edwin, I see you have a patch in bug 1599654 that can resolve this. Perhaps you know what's going on here?

Comment 4

[Cristina Coroiu [:ccoroiu]]

Thank you for looking.
Verified again and started to fail from here: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=linux1804-64-asan-qr%2Copt%2Creftests%2Ctest-linux1804-64-asan-qr%2Fopt-reftest-e10s-2%2Cr%28r2%29&tochange=39c4e3bab99a7dc933cf97593157204b4a280acd&fromchange=ae6bb3a1066959a8c43d003a3caab0af769455bf&selectedJob=283901080

Comment 5

[(Away)]

The needinfo should go to egao

Comment 6

[Edwin Takahashi (:egao | infrequent contributor)]

Acknowledging ni.

This is expected outcome from the migration from ubuntu1604 to ubuntu1804.

TYLin, I have a planned migration as you noted - I do plan on landing this today, which would resolve this unexpected pass.

Comment 7

[Edwin Takahashi (:egao | infrequent contributor)]

The patch to migrate reftests to run on ubuntu1804, and to update the expected outcome of the mfenced-12.html test result has landed on autoland:
https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=9b598ca58e8d10c73c72956ce2e1b5a1bc7945b0

Comment 8

[Razvan Maries]

Tier 1 perma fails: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=Linux%2Cx64%2Cshippable%2Copt%2CReftests%2Ctest-linux64-shippable%2Fopt-reftest-no-accel-e10s-1%2CR%28Ru1%29&tochange=c2b3e2651a6a36b5e9cf324836a6d066fc5b5426&fromchange=32803329a54890816b12073799cb1c1c78ff9a2f

Comment 9

[Edwin Takahashi (:egao | infrequent contributor)]

My patch will address the issues with the normal reftest. I'm trying to green up the reftest-no-accel variant at the moment.

Comment 10

[Edwin Takahashi (:egao | infrequent contributor)]

The issue is not because of new failure, rather it has to do with my migration work to run these tests under ubuntu1804 where this test does not fail. Reftest manifests don't allow enough flexibility to mark individual platforms sadly.

Comment 11

[Edwin Takahashi (:egao | infrequent contributor)]

I'm simply going to disable this platform linux1804-64-asan-qr for the time being as it was not meant to be enabled at the moment.

We will not lose coverage since the linux64-asan-qr variant still runs and it is identical except for that it uses the legacy ubuntu1604 image.

Comment 12

[Andreea Pavel [:apavel]]

Hi Edwin, are you referring to the patch in bug 1428705 ? if so, that reached central and not this is failing as

REFTEST TEST-UNEXPECTED-FAIL | file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/mathml/mfenced-12.html == file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/mathml/mfenced-12-ref.html | image comparison, max difference: 255, number of differing pixels: 164

Can you please take a look?

Treeherder link: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception&fromchange=c0a6eb95b65cce34497c3f6aca502934ed0191cc&selectedJob=284139166

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=284139166&repo=mozilla-central&lineNumber=10006

Comment 13

[Intermittent Failures Robot]

49 failures in 897 pushes (0.055 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 39
• mozilla-central: 10

Platform breakdown:

• linux1804-64-asan-qr: 26
• linux64-shippable: 7
• linux64-asan-qr: 4
• linux64: 10
• linux64-ccov: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607620&startday=2020-01-08&endday=2020-01-08&tree=all

Comment 15

[Edwin Takahashi (:egao | infrequent contributor)]

(In reply to Andreea Pavel [:apavel] from comment #12)

Hi Edwin, are you referring to the patch in bug 1428705 ? if so, that reached central and not this is failing as

REFTEST TEST-UNEXPECTED-FAIL | file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/mathml/mfenced-12.html == file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/mathml/mfenced-12-ref.html | image comparison, max difference: 255, number of differing pixels: 164

Can you please take a look?

Treeherder link: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception&fromchange=c0a6eb95b65cce34497c3f6aca502934ed0191cc&selectedJob=284139166

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=284139166&repo=mozilla-central&lineNumber=10006

:apavel - that is not the patch I was referring to. This is the patch that should silence failures on test-linux1804-asan-qr and test-linux1804-ccov: https://phabricator.services.mozilla.com/D59214

Comment 16

[Edwin Takahashi (:egao | infrequent contributor)]

Looks like that patch landed and the failures have been silenced. I will close this bug for the time being.

Comment 17

[Narcis Beleuzu [:NarcisB]]

:egao , It`s perma failing on "Linux x64 debug"
https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception%2Crunnable&searchStr=linux%2Cx64%2Cdebug%2Creftests%2Ctest-linux64%2Fdebug-reftest-no-accel-e10s-2%2Cr%28ru2%29&tochange=ae04cf04ee369fb46df16726562ca056abc06abc&fromchange=4347a8aad65b6859d0e2dabd98930793750e68e0

Comment 18

[Edwin Takahashi (:egao | infrequent contributor)]

It should be closed because this patch is making its way to mozilla-central, which will resolve the problem: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedJob=284133674&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunning%2Cpending%2Crunnable&revision=53b1fcb5d9d57dbd4fd395829a232c885cd42580

Comment 19

[Edwin Takahashi (:egao | infrequent contributor)]

I noticed that this is the bug that reports unexpected failure, I think I mixed it up with a very similar bug 1607916 - my mistake.

I'll wait for the patch to land successfully on mozilla-central first.

Comment 20

[Intermittent Failures Robot]

107 failures in 819 pushes (0.131 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 64
• mozilla-central: 38
• try: 2
• mozilla-beta: 3

Platform breakdown:

• linux64: 64
• linux64-asan-qr: 15
• linux64-shippable: 17
• linux64-ccov: 8
• linux1804-64-asan-qr: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607620&startday=2020-01-09&endday=2020-01-09&tree=all

Comment 21

[Edwin Takahashi (:egao | infrequent contributor)]

This push should have resolved the failure of this test on linux platforms.

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&group_state=expanded&revision=558a2ae20a30f33cc1b042dab27ef04eccb4ea58&searchStr=reftest-

Comment 22

[Bogdan Tara[:bogdan_tara | bogdant]]

Seems to have stopped. Last failure was ~22 hours ago.

Comment 23

[Edwin Takahashi (:egao | infrequent contributor)]

Yes, the patch to disable the test made its way to mozilla-central. Now I need to work carefully to re-enable the test is appropriate.

Comment 24

[Edwin Takahashi (:egao | infrequent contributor)]

Closing as the work to re-enable this test is being handled elsewhere.

Comment 25

[Intermittent Failures Robot]

179 failures in 4472 pushes (0.04 failures/push) were associated with this bug in the last 7 days.

This is the #3 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 125
• mozilla-central: 48
• try: 3
• mozilla-beta: 3

Platform breakdown:

• linux1804-64-asan-qr: 51
• linux64-asan-qr: 19
• linux64: 75
• linux64-shippable: 24
• linux64-ccov: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607620&startday=2020-01-06&endday=2020-01-12&tree=all

Comment 26

[Intermittent Failures Robot]

1 failures in 3773 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform breakdown:

• linux64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607620&startday=2020-01-13&endday=2020-01-19&tree=all