Closed Bug 1499459 Opened 6 years ago Closed 2 years ago

Intermittent reftest failures from a downloadable font failing to be used (on Windows 7)

Categories

(Core :: Graphics: Text, defect, P3)

Product:
Core
Component:
Graphics: Text
Platform:
x86
Windows 7
Type:
defect

Tracking

()

Status:
RESOLVED WORKSFORME
Tracking Flags:
Tracking Status
firefox64 --- affected

People

(Reporter: dholbert, Assigned: dholbert)

References

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell unknown])

Attachments

(1 file)

Bug 1499459: Remove skip-if-win7 reftest annotation for an issue that seems to have gone away. r?#layout-reviewers
48 bytes, text/x-phabricator-request
Details | Review 
We've gotten (at least) 3 bugs filed in 3 days, where the reftest snapshot shows that a downloadable font failed to render and we used the default font instead.

(In bug 1498954 and bug 1499268, the font is supposed to be used in testcase + reference case, and it only failed to load in the testcase. In bug 1499370, it's a "!=" comparison where only the testcase is supposed to use the downloadable font, but it fails & uses the default font instead.)

So far, this seems to be Win7 specific.

Let's track these bugs collectively by duping them to this bug.  jfkthame, do you know what might be going on here, and if anything might've changed recently to introduce this?  (I'm guessing there might've been a change around font loading, or [less likely] maybe something around the reftest harness's reftest-snaphot-timing.)
Flags: needinfo?(jfkthame)
Priority: -- → P3
Offhand, I'm not aware of recent font-loading changes that might have affected this.

My guess (FWIW) is that this is similar to bug 1392106; the Windows graphics/font subsystem is being pushed a little too hard, and font activation occasionally fails. I'm pretty sure we have some existing intermittents on file that look like this, though it's certainly possible that something has changed recently such that it's become more frequent.

(Aha -- see bug 1430572. Although that was resolved as "fixed", I'm not sure it ever went away completely.)
Flags: needinfo?(jfkthame)
Depends on: 1430572
Summary: Intermittent reftest failures from a downloadable font failing to download (on Windows 7) → Intermittent reftest failures from a downloadable font failing to be used (on Windows 7)
There are 40 failures in the last 7 days.
They occur on windows7-32 and the affected build types are opt, debug, pgo.
Recent failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=209032687&repo=autoland&lineNumber=25783
lsalzman: Hi, Can you please have a look at this? Thanks!
Flags: needinfo?(lsalzman)
Looks like a different manifestation of bug 1392106. See comment 4
Flags: needinfo?(lsalzman)
OS: Unspecified → Windows 7
Hardware: Unspecified → x86
There are  73 total failures in the last 7 days,.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=212783203&repo=autoland&lineNumber=24295
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]
Whiteboard: [stockwell disable-recommended] → [stockwell needswork]
There are 122 failures in the last 7 days. Recent failures log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=215521195&repo=mozilla-inbound&lineNumber=33784
Whiteboard: [stockwell disable-recommended] → [stockwell needswork]
There are 109 total failures in the last 7 days all on windows7-32 debug,pgo and opt. 

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=215853511&repo=autoland&lineNumber=88151

00:01:32     INFO - REFTEST TEST-START | file:///Z:/task_1544138443/build/tests/reftest/tests/layout/reftests/text/justification-cjk-extension.html == file:///Z:/task_1544138443/build/tests/reftest/tests/layout/reftests/text/justification-cjk-extension-ref.html
00:01:32     INFO - REFTEST TEST-LOAD | file:///Z:/task_1544138443/build/tests/reftest/tests/layout/reftests/text/justification-cjk-extension.html | 11 / 284 (3%)
00:01:32     INFO - ++DOMWINDOW == 37 (0AC31400) [pid = 4724] [serial = 37] [outer = 01386080]
00:01:33     INFO - REFTEST TEST-LOAD | file:///Z:/task_1544138443/build/tests/reftest/tests/layout/reftests/text/justification-cjk-extension-ref.html | 11 / 284 (3%)
00:01:33     INFO - ++DOMWINDOW == 38 (0AC31C00) [pid = 4724] [serial = 38] [outer = 01386080]
00:01:33     INFO - --DOMWINDOW == 37 (090AA800) [pid = 4724] [serial = 5] [outer = 00000000] [url = about:blank]
00:01:33     INFO - --DOMWINDOW == 36 (0136A400) [pid = 4724] [serial = 2] [outer = 00000000] [url = about:blank]
00:01:33     INFO - REFTEST TEST-UNEXPECTED-FAIL | file:///Z:/task_1544138443/build/tests/reftest/tests/layout/reftests/text/justification-cjk-extension.html == file:///Z:/task_1544138443/build/tests/reftest/tests/layout/reftests/text/justification-cjk-extension-ref.html | image comparison, max difference: 255, number of differing pixels: 820


Joel, is there someone that can take a look at this?
Flags: needinfo?(jmaher)
I spent some time looking into this and found that there isn't much we can do.  Looking at a big picture, I don't see any unique regressions being found by windows7/opt reftests which account for maybe 1/3 of the regressions in this bug and a larger majority of the regreessions in bug 1392106.

With that said, I think reconsidering why we run reftests on windows 7 should be on the table.  I know that reftests are mostly duplicated in web-platform-tests and because of that we can in a couple months turn off a large portion of windows 7 reftests and let them run on web-platform-tests only.

Unfortunately I don't know a lot we can do right now.
Flags: needinfo?(jmaher)
Probably not the right forum for this, but I don't think I agree that reftests are mostly duplicated by web-platform-tests...
Thanks for clarifying that :dbaron, I had heard data from the web-platform-tests team that it was, possibly it is a subset that is duplicated.
That's correct: some reftests are duplicated in WPT, but it's far from all of them.
In the last 7 days there have been 118 occurrences on Andoid, Linux and Windows, build types debug, opt and pgo.
Blocks: 1520029

In the last 7 days, there are 306 failures.
They occur on mostly on windows7-32, all build types and windows7-32-nightly opt.
Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=224903149&repo=autoland&lineNumber=38579

This bug failed 358 times in the last 7 days. Occurs on windows7-32 on all build types.

Recent log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=226749112&repo=mozilla-inbound&lineNumber=30309

Whiteboard: [stockwell disable-recommended]

This bug failed 218 times in the last 7 days. Occurs on windows7-32 on all build types.

Recent log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=229365372&repo=autoland&lineNumber=15326

This bug failed 186 times in the last 7 days. Occurs on windows7-32 on all build types.

Recent log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=231122612&repo=mozilla-inbound&lineNumber=47398

Whiteboard: [stockwell disable-recommended] → [stockwell needswork]
Whiteboard: [stockwell disable-recommended] → [stockwell needswork]

This failure has spiked again at 114 failures in the last 7 days.

recent log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=235983932&repo=mozilla-inbound&lineNumber=11781

Whiteboard: [stockwell disable-recommended] → [stockwell needswork]

There are 58 failures in the last 7 days.

In the last 7 days there have been 81 occurrences.

Whiteboard: [stockwell disable-recommended] → [stockwell needswork]

Over the last 7 days this bug has 111 failures.

:lsalzman can you please take a look at this? Thanks!

Flags: needinfo?(lsalzman)

See comment 46.

Flags: needinfo?(lsalzman)
Whiteboard: [stockwell disable-recommended] → [stockwell needswork]
Whiteboard: [stockwell disable-recommended] → [stockwell needswork]

In the last 7 days this has 120 failures

autoland: 63
mozilla-central: 19
try: 9
mozilla-inbound: 19
mozilla-beta: 10

Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]
Whiteboard: [stockwell disable-recommended]
Whiteboard: [stockwell disable-recommended]

There are 109 failures in the past week happening on windows 7 platform.

Recent log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=245758426&repo=autoland&lineNumber=19133

Whiteboard: [stockwell disable-recommended]
Whiteboard: [stockwell disable-recommended]
Whiteboard: [stockwell disable-recommended] → [ Comment 46]
Whiteboard: [ Comment 46][stockwell disable-recommended] → [ Comment 46]
Whiteboard: [ Comment 46][stockwell disable-recommended] → [ Comment 46]
Whiteboard: [ Comment 46][stockwell disable-recommended] → [ Comment 46]

This bug failed 111 times in the last 7 days. Occurs on windows7-32-shippable, windows7-32-devedition, windows7-32, windows7-32-nightly on opt and debug build types.

Log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=247451618&repo=autoland&lineNumber=24298

Whiteboard: [ Comment 46][stockwell disable-recommended] → [ Comment 46]
Whiteboard: [ Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]

There are 57 failures associated to this bug in the last 7 days. These are occurring mainly on windows7-32, windows7-32-shippable.

Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]

There are 34 failures associated to this bug in the last 7 days. These are occurring mainly on windows7-32, windows7-32-shippable.

Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Blocks: 1502782

There are 50 failures associated to this bug in the last 7 days. These are occurring on windows7-32 debug builds and windows7-32-devedition opt builds.

Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]

There are 71 failures associated to this bug in the last 7 days. These are occurring on windows 7 all builds.

Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]

Note that in most cases this has been a downloadable font (per the bug title). However, I think the just-duped bug 1578960 is an instance of the same underlying problem (or a similar one) where there's simply font-family:monospace -- i.e. a custom font-family, but not a downloadable font.

Similarly, https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=264376653&repo=try&lineNumber=15171 had a failure in the reftest
https://searchfox.org/mozilla-central/rev/e5327b05c822cdac24e233afa37d72c0552dbbaf/layout/reftests/bugs/23604-2.html
where there's no downloadable font, but still an unexpected/inconsistent choice of font.

The failure screenshots simply show a different choice of font for the third (italicized) line (whose font is merely determined by font-family:sans-serif along with the italicization from <em>.)

Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]

There are 128 total failures in the last 7 days, majority on windows7-32 debug and opt and windows7-32-mingwclang and windows7-32-shippable opt.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=270805755&repo=autoland&lineNumber=33987
[task 2019-10-11T00:21:07.516Z] 00:21:07 INFO - REFTEST TEST-START | file:///Z:/task_1570749582/build/tests/reftest/tests/layout/reftests/css-placeholder/textarea/placeholdershown.html == file:///Z:/task_1570749582/build/tests/reftest/tests/layout/reftests/css-placeholder/textarea/placeholdershown-ref.html
[task 2019-10-11T00:21:07.516Z] 00:21:07 INFO - REFTEST TEST-LOAD | file:///Z:/task_1570749582/build/tests/reftest/tests/layout/reftests/css-placeholder/textarea/placeholdershown.html | 9 / 13 (69%)
[task 2019-10-11T00:21:07.644Z] 00:21:07 INFO - REFTEST TEST-LOAD | file:///Z:/task_1570749582/build/tests/reftest/tests/layout/reftests/css-placeholder/textarea/placeholdershown-ref.html | 9 / 13 (69%)
[task 2019-10-11T00:21:07.803Z] 00:21:07 INFO - REFTEST TEST-UNEXPECTED-FAIL | file:///Z:/task_1570749582/build/tests/reftest/tests/layout/reftests/css-placeholder/textarea/placeholdershown.html == file:///Z:/task_1570749582/build/tests/reftest/tests/layout/reftests/css-placeholder/textarea/placeholdershown-ref.html | image comparison, max difference: 199, number of differing pixels: 146

Analyzer: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/duTGXMvdRpinAuXz_3vokQ/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]

In the last 7 days there have been 140 occurrences on Windows 7 32 build types: debug and opt.

Recent failure: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=272015747&repo=autoland&lineNumber=41919
https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/ZdT4gqSATOWWRLfkF881aw/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

[task 2019-10-19T06:43:41.794Z] 06:43:41 INFO - REFTEST TEST-START | file:///Z:/task_1571465366/build/tests/reftest/tests/layout/reftests/font-face/variation-format-hint-1a.html == file:///Z:/task_1571465366/build/tests/reftest/tests/layout/reftests/font-face/variation-format-hint-1A-ref.html
[task 2019-10-19T06:43:41.794Z] 06:43:41 INFO - REFTEST TEST-LOAD | file:///Z:/task_1571465366/build/tests/reftest/tests/layout/reftests/font-face/variation-format-hint-1a.html | 134 / 142 (94%)
[task 2019-10-19T06:43:41.795Z] 06:43:41 INFO - ++DOMWINDOW == 28 (1666F800) [pid = 5612] [serial = 371] [outer = 0156FBD0]
[task 2019-10-19T06:43:41.841Z] 06:43:41 INFO - REFTEST TEST-LOAD | file:///Z:/task_1571465366/build/tests/reftest/tests/layout/reftests/font-face/variation-format-hint-1A-ref.html | 134 / 142 (94%)
[task 2019-10-19T06:43:41.848Z] 06:43:41 INFO - ++DOMWINDOW == 29 (16C37C00) [pid = 5612] [serial = 372] [outer = 0156FBD0]
[task 2019-10-19T06:43:41.991Z] 06:43:41 INFO - REFTEST TEST-UNEXPECTED-FAIL | file:///Z:/task_1571465366/build/tests/reftest/tests/layout/reftests/font-face/variation-format-hint-1a.html == file:///Z:/task_1571465366/build/tests/reftest/tests/layout/reftests/font-face/variation-format-hint-1A-ref.html | image comparison, max difference: 255, number of differing pixels: 2509

Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]

In the last 7 days there have been 147 occurrences on Windows 7 32 build types debug and opt.

Recent failure: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=274066798&repo=autoland&lineNumber=41894

[task 2019-11-01T11:51:36.333Z] 11:51:36 INFO - REFTEST TEST-START | file:///Z:/task_1572607285/build/tests/reftest/tests/layout/reftests/text/justification-cjk-extension.html == file:///Z:/task_1572607285/build/tests/reftest/tests/layout/reftests/text/justification-cjk-extension-ref.html
[task 2019-11-01T11:51:36.333Z] 11:51:36 INFO - REFTEST TEST-LOAD | file:///Z:/task_1572607285/build/tests/reftest/tests/layout/reftests/text/justification-cjk-extension.html | 11 / 288 (3%)
[task 2019-11-01T11:51:36.340Z] 11:51:36 INFO - ++DOMWINDOW == 26 (0B421000) [pid = 4744] [serial = 37] [outer = 00B6FCE0]
[task 2019-11-01T11:51:36.814Z] 11:51:36 INFO - REFTEST TEST-LOAD | file:///Z:/task_1572607285/build/tests/reftest/tests/layout/reftests/text/justification-cjk-extension-ref.html | 11 / 288 (3%)
[task 2019-11-01T11:51:36.821Z] 11:51:36 INFO - ++DOMWINDOW == 27 (08895C00) [pid = 4744] [serial = 38] [outer = 00B6FCE0]
[task 2019-11-01T11:51:36.978Z] 11:51:36 INFO - REFTEST TEST-UNEXPECTED-FAIL | file:///Z:/task_1572607285/build/tests/reftest/tests/layout/reftests/text/justification-cjk-extension.html == file:///Z:/task_1572607285/build/tests/reftest/tests/layout/reftests/text/justification-cjk-extension-ref.html | image comparison, max difference: 255, number of differing pixels: 814
[task 2019-11-01T11:51:36.979Z] 11:51:36 INFO - REFTEST IMAGE 1 (TEST): data:image/png;base64,iVBORw0KGgoAAAANSUhE...
[task 2019-11-01T11:51:36.979Z] 11:51:36 INFO - REFTEST IMAGE 2 (REFERENCE): data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAyAAAAPoC...
[task 2019-11-01T11:51:36.979Z] 11:51:36 INFO - REFTEST INFO | Saved log: START file:///Z:/task_1572607285/build/tests/reftest/tests/layout/reftests/text/justification-cjk-extension.html
[task 2019-11-01T11:51:36.980Z] 11:51:36 INFO - REFTEST INFO | Saved log: [CONTENT] OnDocumentLoad triggering AfterOnLoadScripts
[task 2019-11-01T11:51:36.981Z] 11:51:36 INFO - REFTEST INFO | Saved log: Initializing canvas snapshot
[task 2019-11-01T11:51:36.981Z] 11:51:36 INFO - REFTEST INFO | Saved log: DoDrawWindow 0,0,800,1000
[task 2019-11-01T11:51:36.982Z] 11:51:36 INFO - REFTEST INFO | Saved log: [CONTENT] RecordResult fired
[task 2019-11-01T11:51:36.982Z] 11:51:36 INFO - REFTEST INFO | Saved log: RecordResult fired
[task 2019-11-01T11:51:36.982Z] 11:51:36 INFO - REFTEST INFO | Saved log: START file:///Z:/task_1572607285/build/tests/reftest/tests/layout/reftests/text/justification-cjk-extension-ref.html
[task 2019-11-01T11:51:36.982Z] 11:51:36 INFO - REFTEST INFO | Saved log: [CONTENT] OnDocumentLoad triggering AfterOnLoadScripts
[task 2019-11-01T11:51:36.982Z] 11:51:36 INFO - REFTEST INFO | Saved log: Initializing canvas snapshot
[task 2019-11-01T11:51:36.982Z] 11:51:36 INFO - REFTEST INFO | Saved log: DoDrawWindow 0,0,800,1000
[task 2019-11-01T11:51:36.983Z] 11:51:36 INFO - REFTEST INFO | Saved log: [CONTENT] RecordResult fired
[task 2019-11-01T11:51:36.983Z] 11:51:36 INFO - REFTEST INFO | Saved log: RecordResult fired
[task 2019-11-01T11:51:36.983Z] 11:51:36 INFO - REFTEST TEST-END | file:///Z:/task_1572607285/build/tests/reftest/tests/layout/reftests/text/justification-cjk-extension.html == file:///Z:/task_1572607285/build/tests/reftest/tests/layout/reftests/text/justification-cjk-extension-ref.html

Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]
Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 46]

There are 308 total failures in the last 7 days on win7-32 (please disregard the android classifications since this is a win specific bug)

Joel is there something we can do here? (please see comment as a reminder of your previous investigations)

Thank you.

Flags: needinfo?(jmaher)

Thanks for checking back on this bug :apavel. I am not aware of any magic tricks. What is interesting is we don't see this on WPT, so there must be a key in how the reftest harness runs vs the WPT harness.

In the last year the windows 7 share has declined to <30% of our market share, still significant to keep test coverage.

I wonder if the subset of tests duplicated between reftest and WPT have been resolved? David, do you know if that has been resolved?

Flags: needinfo?(jmaher) → needinfo?(dburns)

They havent be resolved. We are waiting for https://bugzilla.mozilla.org/show_bug.cgi?id=1595564 before we can see about removing things.

Flags: needinfo?(dburns)

bug 1518097, bug 1573075, and bug 1595265 (which are duplicates of this bug) show that we do see this on WPT.

Whiteboard: [Comment 46][stockwell disable-recommended] → [Comment 376]
Whiteboard: [Comment 376][stockwell disable-recommended] → [Comment 376]
Whiteboard: [Comment 376][stockwell disable-recommended] → [comment 377]

(In reply to David Burns :automatedtester from comment #376)

They havent be resolved. We are waiting for https://bugzilla.mozilla.org/show_bug.cgi?id=1595564 before we can see about removing things.

Hi David, the bug you mentioned is now fixed. Are there any updates here or something we can help with?

Flags: needinfo?(dburns)
Whiteboard: [comment 377][stockwell disable-recommended] → [comment 377][stockwell needswork:owner]

(In reply to Andreea Pavel [:apavel] from comment #385)

(In reply to David Burns :automatedtester from comment #376)

They havent be resolved. We are waiting for https://bugzilla.mozilla.org/show_bug.cgi?id=1595564 before we can see about removing things.

Hi David, the bug you mentioned is not fixed. Are there any updates here or something we can help with?

The patch that landed was about giving us an opportunity to start looking at reftest duplication not for fixing this issue. Sorry you got confused.

Flags: needinfo?(dburns)

Thank you for clarifying that.

There seems to have been a spike here starting with the 29th of November: https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-11-25&endday=2019-12-02&tree=trunk&bug=1499459

Joel do you know who I can ni here?

Flags: needinfo?(jmaher)

this increases when we run more tests, typically on weekends when the volume is low and we run all the tests more frequently vs the limited subset of tests (via SETA). November 29th was part of the long US holiday weekend. If you look at a 30 day window, you can see the spikes on the weekends.

Flags: needinfo?(jmaher)

Oh that makes sense. Thank you.

Whiteboard: [comment 377][stockwell disable-recommended] → [comment 377][stockwell needswork:owner]
Whiteboard: [comment 377][stockwell disable-recommended] → [comment 377]

There are 84 total failures in the last 7 days on windows7-32 opt and debug and windows7-32-mingwclang debug.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=280193574&repo=autoland&lineNumber=54727

[task 2019-12-08T23:28:31.985Z] 23:28:31 INFO - REFTEST TEST-START | http://localhost:49434/1575846202545/153/font-face/reflow-sanity-1.html == http://localhost:49434/1575846202545/153/font-face/reflow-sanity-1-ref.html
[task 2019-12-08T23:28:31.985Z] 23:28:31 INFO - REFTEST TEST-LOAD | http://localhost:53277/1575847682372/5/font-face/reflow-sanity-1.html | 114 / 142 (80%)
[task 2019-12-08T23:28:31.986Z] 23:28:31 INFO - [Child 392, Main Thread] WARNING: 'NS_FAILED(rv)', file z:/build/build/src/widget/windows/nsLookAndFeel.cpp, line 853
[task 2019-12-08T23:28:31.993Z] 23:28:31 INFO - [Child 392, Main Thread] WARNING: NS_ENSURE_TRUE(mPresShell) failed: file z:/build/build/src/layout/base/nsPresContext.cpp, line 847
[task 2019-12-08T23:28:32.028Z] 23:28:32 INFO - [Child 392, Main Thread] WARNING: 'NS_FAILED(GetAccentColor(unused))', file z:/build/build/src/widget/windows/nsLookAndFeel.cpp, line 481
[task 2019-12-08T23:28:32.075Z] 23:28:32 INFO - ### XPCOM_MEM_BLOAT_LOG defined -- logging bloat/leaks to c:\users\task_1575845216\appdata\local\temp\tmpy1zjvw.mozrunner\runreftest_leaks_tab_pid3964.log
[task 2019-12-08T23:28:32.095Z] 23:28:32 INFO - [Child 392, Main Thread] WARNING: '!aWin', file z:/build/build/src/dom/base/Document.cpp, line 1378
[task 2019-12-08T23:28:32.095Z] 23:28:32 INFO - [Child 392, Main Thread] WARNING: '!aWin', file z:/build/build/src/dom/base/Document.cpp, line 1378
[task 2019-12-08T23:28:32.110Z] 23:28:32 INFO - [Child 3964, Main Thread] WARNING: No CID found when attempting to map contract ID: file z:/build/build/src/xpcom/components/nsComponentManager.cpp, line 727
[task 2019-12-08T23:28:32.160Z] 23:28:32 INFO - [Parent 1512, Main Thread] WARNING: we only accept nsIURI interface type, patch welcome: file z:/build/build/src/dom/ipc/PropertyBagUtils.cpp, line 112
[task 2019-12-08T23:28:32.296Z] 23:28:32 INFO - nsStringStats
[task 2019-12-08T23:28:32.297Z] 23:28:32 INFO - => mAllocCount: 10504
[task 2019-12-08T23:28:32.298Z] 23:28:32 INFO - => mReallocCount: 0
[task 2019-12-08T23:28:32.298Z] 23:28:32 INFO - => mFreeCount: 10504
[task 2019-12-08T23:28:32.298Z] 23:28:32 INFO - => mShareCount: 12678
[task 2019-12-08T23:28:32.298Z] 23:28:32 INFO - => mAdoptCount: 630
[task 2019-12-08T23:28:32.299Z] 23:28:32 INFO - => mAdoptFreeCount: 690
[task 2019-12-08T23:28:32.299Z] 23:28:32 INFO - => Process ID: 1872, Thread ID: 672
[task 2019-12-08T23:28:32.414Z] 23:28:32 INFO - [Parent 1512, Main Thread] WARNING: IPC message discarded: actor cannot send: file z:/build/build/src/ipc/glue/ProtocolUtils.cpp, line 481
[task 2019-12-08T23:28:32.414Z] 23:28:32 INFO - [Parent 1512, Main Thread] WARNING: Error sending reply: file z:/build/build/src/obj-firefox/ipc/ipdl/PContentParent.cpp, line 11132
[task 2019-12-08T23:28:32.415Z] 23:28:32 INFO - [Parent 1512, Main Thread] WARNING: IPC message discarded: actor cannot send: file z:/build/build/src/ipc/glue/ProtocolUtils.cpp, line 481
[task 2019-12-08T23:28:32.415Z] 23:28:32 INFO - [Parent 1512, Main Thread] WARNING: Error sending reply: file z:/build/build/src/obj-firefox/ipc/ipdl/PContentParent.cpp, line 11132
[task 2019-12-08T23:28:33.869Z] 23:28:33 INFO - REFTEST TEST-LOAD | http://localhost:53277/1575847682372/5/font-face/reflow-sanity-1-ref.html | 114 / 142 (80%)
[task 2019-12-08T23:28:33.884Z] 23:28:33 INFO - [Parent 1512, Main Thread] WARNING: we only accept nsIURI interface type, patch welcome: file z:/build/build/src/dom/ipc/PropertyBagUtils.cpp, line 112
[task 2019-12-08T23:28:34.540Z] 23:28:34 INFO - REFTEST TEST-UNEXPECTED-FAIL | http://localhost:49434/1575846202545/153/font-face/reflow-sanity-1.html == http://localhost:49434/1575846202545/153/font-face/reflow-sanity-1-ref.html | image comparison, max difference: 255, number of differing pixels: 12893

Whiteboard: [comment 377][stockwell disable-recommended] → [comment 377]
Whiteboard: [comment 377][stockwell disable-recommended] → [comment 377]
Whiteboard: [comment 377][stockwell disable-recommended] → [comment 377]

In the last 7 days there were 74 failures on this bug. These happend on windows7-32, windows7-32-shippable opt and debug build types.

Recent failure log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=312391947&repo=autoland&lineNumber=27930

Lee, could you assign someone to take a look?

Flags: needinfo?(lsalzman)
Whiteboard: [comment 377][stockwell needswork:owner] → [stockwell needswork:owner]

(In reply to Atila Butkovits from comment #440)

In the last 7 days there were 74 failures on this bug. These happend on windows7-32, windows7-32-shippable opt and debug build types.

Recent failure log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=312391947&repo=autoland&lineNumber=27930

Lee, could you assign someone to take a look?

Nothing has changed with regards to the explanation or status of the bug as noted in previous status updates through this bug thread. Progress is still stalled.

Flags: needinfo?(lsalzman)

what if we ran these tests on windows10x64 in wow64 to give a 32 bit environment. This way we would continue to ensure 32 bit coverage, but possibly have more support for fixing these issues.

There are 24 total failures in the last 7 days on windows7-32, windows7-32-shippable and windows7-32-devedition

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=313751324&repo=mozilla-central&lineNumber=16025

[task 2020-08-22T22:48:48.889Z] 22:48:48 INFO - REFTEST TEST-START | layout/reftests/text-decoration/skip-ink-multiline-position.html == layout/reftests/text-decoration/skip-ink-multiline-position-ref.html
[task 2020-08-22T22:48:48.889Z] 22:48:48 INFO - REFTEST TEST-LOAD | http://localhost:51657/1598136510785/1/text-decoration/skip-ink-multiline-position.html | 116 / 122 (95%)
[task 2020-08-22T22:48:48.896Z] 22:48:48 INFO - [Child 1220, Main Thread] WARNING: NS_ENSURE_TRUE(browserChrome) failed: file /builds/worker/checkouts/gecko/docshell/base/nsDocShell.cpp, line 11609
[task 2020-08-22T22:48:48.986Z] 22:48:48 INFO - [Child 1220, Main Thread] WARNING: NS_ENSURE_TRUE(info) failed: file /builds/worker/checkouts/gecko/extensions/permissions/PermissionDelegateHandler.cpp, line 348
[task 2020-08-22T22:48:49.201Z] 22:48:49 INFO - [Child 3800, Main Thread] WARNING: Extra shutdown CC: 'i < NORMAL_SHUTDOWN_COLLECTIONS', file /builds/worker/checkouts/gecko/xpcom/base/nsCycleCollector.cpp, line 3359
[task 2020-08-22T22:48:49.224Z] 22:48:49 INFO - [Child 3800, Main Thread] WARNING: NS_ENSURE_TRUE(Preferences::InitStaticMembers()) failed: file /builds/worker/checkouts/gecko/modules/libpref/Preferences.cpp, line 4349
[task 2020-08-22T22:48:50.006Z] 22:48:50 INFO - REFTEST TEST-LOAD | http://localhost:51657/1598136510785/1/text-decoration/skip-ink-multiline-position-ref.html | 116 / 122 (95%)
[task 2020-08-22T22:48:50.221Z] 22:48:50 INFO - REFTEST TEST-UNEXPECTED-FAIL | layout/reftests/text-decoration/skip-ink-multiline-position.html == layout/reftests/text-decoration/skip-ink-multiline-position-ref.html | image comparison, max difference: 255, number of differing pixels: 752

Whiteboard: [stockwell disable-recommended]

There are 28 total failures in the last 7 days on windows7 opt and debug

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=318927362&repo=autoland&lineNumber=39257

[task 2020-10-17T17:02:04.544Z] 17:02:04 INFO - REFTEST TEST-START | layout/reftests/text-decoration/skip-ink-multiline-position.html == layout/reftests/text-decoration/skip-ink-multiline-position-ref.html
[task 2020-10-17T17:02:04.544Z] 17:02:04 INFO - REFTEST TEST-LOAD | http://localhost:55953/1602954114163/1/text-decoration/skip-ink-multiline-position.html | 116 / 122 (95%)
[task 2020-10-17T17:02:04.813Z] 17:02:04 INFO - REFTEST TEST-LOAD | http://localhost:55953/1602954114163/1/text-decoration/skip-ink-multiline-position-ref.html | 116 / 122 (95%)
[task 2020-10-17T17:02:04.995Z] 17:02:04 INFO - REFTEST TEST-UNEXPECTED-FAIL | layout/reftests/text-decoration/skip-ink-multiline-position.html == layout/reftests/text-decoration/skip-ink-multiline-position-ref.html | image comparison, max difference: 255, number of differing pixels: 752

In the last 7 days there were 23 failures. Happens on windows7-32-shippable opt and windows7-32 debug and opt.

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=320344580&repo=mozilla-central&lineNumber=26477

In the last 7 days there have been 44 occurrences on windows7-32 debug and opt.

Recent failure: https://treeherder.mozilla.org/logviewer?job_id=321848432&repo=autoland&lineNumber=28559

There have been 41 total failures in the last 7 days:

Affected platforms:

  • windows7-32 opt & debug
  • windows7-32-shippable opt

In the last 7 days there have been 42 occurrences, all on windows7-32 debug and opt.

Recent failure: https://treeherder.mozilla.org/logviewer?job_id=325060764&repo=mozilla-central&lineNumber=12842

Whiteboard: [stockwell disable-recommended]

Spike here is under layout/reftests/text-decoration/skip-ink-multiline-position.html

Emilio can it be related to any of you pushes ? Spike starts January 29th/30th: https://treeherder.mozilla.org/intermittent-failures/bugdetails?startday=2021-01-25&endday=2021-02-01&tree=trunk&bug=1499459

Flags: needinfo?(emilio)

I doubt so... If it's win7-only it's fairly unlikely. It might be because some of the tests we've added have moved the test around so that it triggers this more easily. Jonathan do you happen to have ideas about the failure above?

Flags: needinfo?(emilio) → needinfo?(jfkthame)

I don't have any specific ideas here. As far as I know, this seems to happen intermittently on win7 when we put the graphics system under too much strain. Anything we do that alters page-load timing or test chunking may shift things around such that failures become more (or less) frequent on particular tests.

If we're seeing a persistently high failure rate on this test, I'd be fine with just skipping it on win7. I don't think there's much we can do to resolve the issue for one specific test like this; it's a systemic problem in the win7 graphics system, and this test just happens to be a favorite victim at the moment.

(Though there's a nonzero risk that if we skip the currently-failing test, we'll just see the same type of failure pop up on the next testcase that tries to load a font...)

Flags: needinfo?(jfkthame)
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

There are 111 total failures in the last 7 days on windows7-32 opt and debug and windows7-32-shippable opt

Recent failure log:

[task 2021-02-09T10:02:11.825Z] 10:02:11 INFO - REFTEST TEST-START | layout/reftests/text-decoration/1415214.html != layout/reftests/text-decoration/1415214-notref.html
[task 2021-02-09T10:02:11.825Z] 10:02:11 INFO - REFTEST TEST-LOAD | file:///Z:/task_1612863044/build/tests/reftest/tests/layout/reftests/text-decoration/1415214.html | 114 / 121 (94%)
[task 2021-02-09T10:02:11.862Z] 10:02:11 INFO - REFTEST TEST-LOAD | file:///Z:/task_1612863044/build/tests/reftest/tests/layout/reftests/text-decoration/1415214-notref.html | 114 / 121 (94%)
[task 2021-02-09T10:02:11.891Z] 10:02:11 INFO - REFTEST TEST-PASS | layout/reftests/text-decoration/1415214.html != layout/reftests/text-decoration/1415214-notref.html | image comparison, max difference: 255, number of differing pixels: 47
[task 2021-02-09T10:02:11.891Z] 10:02:11 INFO - REFTEST TEST-END | layout/reftests/text-decoration/1415214.html != layout/reftests/text-decoration/1415214-notref.html
[task 2021-02-09T10:02:11.899Z] 10:02:11 INFO - REFTEST TEST-START | layout/reftests/text-decoration/skip-ink-multiline-position.html == layout/reftests/text-decoration/skip-ink-multiline-position-ref.html
[task 2021-02-09T10:02:11.899Z] 10:02:11 INFO - REFTEST INFO | Pref layout.css.text-underline-offset.enabled not found, will be added
[task 2021-02-09T10:02:11.899Z] 10:02:11 INFO - REFTEST INFO | SET PREFERENCE pref(layout.css.text-underline-offset.enabled,true)
[task 2021-02-09T10:02:11.899Z] 10:02:11 INFO - REFTEST TEST-LOAD | http://localhost:52367/1612864920689/1/text-decoration/skip-ink-multiline-position.html | 115 / 121 (95%)
[task 2021-02-09T10:02:12.161Z] 10:02:12 INFO - REFTEST INFO | RESTORE PREFERENCE pref(layout.css.text-underline-offset.enabled, <no value set>) (clearing user pref)
[task 2021-02-09T10:02:12.162Z] 10:02:12 INFO - REFTEST TEST-LOAD | http://localhost:52367/1612864920689/1/text-decoration/skip-ink-multiline-position-ref.html | 115 / 121 (95%)
[task 2021-02-09T10:02:12.348Z] 10:02:12 INFO - REFTEST TEST-UNEXPECTED-FAIL | layout/reftests/text-decoration/skip-ink-multiline-position.html == layout/reftests/text-decoration/skip-ink-multiline-position-ref.html | image comparison, max difference: 255, number of differing pixels: 752

Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]
Whiteboard: [stockwell disable-recommended]
Whiteboard: [stockwell disable-recommended]
Whiteboard: [stockwell disable-recommended]
Whiteboard: [stockwell disable-recommended]
Whiteboard: [stockwell disable-recommended]
Whiteboard: [stockwell disable-recommended]
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]
Whiteboard: [stockwell disable-recommended]
Whiteboard: [stockwell disable-recommended]
Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]
Whiteboard: [stockwell disable-recommended]
Whiteboard: [stockwell disable-recommended] → [stockwell needswork]

Like bug 1392106 (which may've had a related underlying cause), this issue seems to have gone away about a year ago or so (yay!)

It looks like we only have one test-failure/skip annotation that's associated with this bug number, added in bug 1705826 comment 10. So we can remove that annotation now, and aside from that I think we can consider this WORKSFORME.

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → WORKSFORME

This annotation was added in bug 1705826, for what was at-the-time a known and
very common category of intermittent test-failures on Win7 (with downloadable
fonts mysteriously failing to render, in pretty much any test that uses
a downloadable font).

These intermittent failures seem to have stopped (possibly due to some
architectural change like WebRender), so we no longer need this annotation.

Assignee: nobody → dholbert
Pushed by dholbert@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ab22dfaba16e
Remove skip-if-win7 reftest annotation for an issue that seems to have gone away. r=emilio
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: