1499124 - Perma [tier 2 ccov] http://localhost:49890/1539262358382/160/font-face/font-display-1.html == http://localhost:49890/1539262358382/160/font-face/font-display-1-ref.html | image comparison, max difference: 255, number of differing pixels: 5633

Status: RESOLVED FIXED

(Core :: Layout: Text and Fonts, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: archaeopteryx [at] coole-files.de

https://treeherder.mozilla.org/logviewer.html#?job_id=204773109&repo=mozilla-central

https://queue.taskcluster.net/v1/task/BF22KFnVT1iQzM5Alpzogg/runs/0/artifacts/public/logs/live_backing.log

https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/BF22KFnVT1iQzM5Alpzogg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

Comment 1

[Intermittent Failures Robot]

1 failures in 2759 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* windows10-64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1499124&startday=2018-10-15&endday=2018-10-21&tree=all

Comment 2

[Intermittent Failures Robot]

3 failures in 2584 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 3

Platform breakdown:
* windows10-64-ccov: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1499124&startday=2018-10-22&endday=2018-10-28&tree=all

Comment 3

[Intermittent Failures Robot]

1 failures in 2978 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* windows10-64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1499124&startday=2018-10-29&endday=2018-11-04&tree=all

Comment 4

[Intermittent Failures Robot]

1 failures in 2917 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 1

Platform breakdown:
* windows10-64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1499124&startday=2018-11-12&endday=2018-11-18&tree=all

Comment 5

[Intermittent Failures Robot]

5 failures in 2808 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 5

Platform breakdown:
* windows10-64-ccov: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1499124&startday=2018-11-26&endday=2018-12-02&tree=all

Comment 6

[Intermittent Failures Robot]

14 failures in 1862 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 14

Platform breakdown:
* windows10-64-ccov: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1499124&startday=2018-12-03&endday=2018-12-09&tree=all

Comment 7

[Intermittent Failures Robot]

25 failures in 2485 pushes (0.01 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-central: 25

Platform breakdown:
* windows10-64-ccov: 25

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1499124&startday=2018-12-10&endday=2018-12-16&tree=all

Comment 8

[Intermittent Failures Robot]

38 failures in 2648 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

This is the #49 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-central: 38

Platform breakdown:
* windows10-64-ccov: 38

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1499124&startday=2018-12-17&endday=2018-12-23&tree=all

Comment 9

[Intermittent Failures Robot]

21 failures in 1008 pushes (0.021 failures/push) were associated with this bug in the last 7 days.

This is the #35 most frequent failure this week.

Repository breakdown:
* mozilla-inbound: 1
* mozilla-central: 20

Platform breakdown:
* linux64: 1
* windows10-64-ccov: 18
* android-em-4-3-armv7-api16: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1499124&startday=2018-12-24&endday=2018-12-30&tree=all

Comment 10

[Cristina Coroiu [:ccoroiu]]

This started to perma fail with this merge: 
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=windows%2C10%2Cx64%2Cccov%2Cdebug%2Creftests%2Cwith%2Ce10s%2Ctest-windows10-64-ccov%2Fdebug-reftest-e10s-1%2Cr-e10s%28r1%29&tochange=435cf6bbfa9ea152e8bf79799e327d12edd947db&fromchange=a791cbab57a3908cf66afbfddf9bbc563353e700&selectedJob=219077381

:emilio, could you please take a look at this? could be related to https://bugzilla.mozilla.org/show_bug.cgi?id=1422235? 
It is failing on your push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&resultStatus=testfailed%2Cbusted%2Cexception%2Crunning%2Cpending%2Crunnable&searchStr=ccov&revision=9495b83aad6421af9b2f00cb248257d845344e04&selectedJob=220195542

Thank you

Comment 11

[Emilio Cobos Álvarez [:emilio]]

It's not (directly, at least) related to that bug. That bug introduced new reftests which means that these reftests may run at a different time, which could, perhaps trickle this in a different way.

The letter that shows up incorrectly is the one using `font-display: fallback`. This could be either a bug in our font tests, or the test having faulty assumptions about the font loading fast enough.

IIRC font-display: fallback does not unconditionally wait for the font load, and ccov builds are slow, so the test may be racy in a sense, where the font may be slow enough to come from disk that we end up canceling the load and displaying the fallback. Jonathan, does that sound right?

Comment 12

[Jonathan Kew [:jfkthame]]

That seems possible, though I'm slightly surprised it would go so abruptly from being a fairly rare intermittent to totally perma-failing. Still, I don't see any reason why that bug should have affected it. It certainly makes sense that the test could be unreliable, particularly on slow builds/environments.

Comment 13

[Intermittent Failures Robot]

33 failures in 1668 pushes (0.02 failures/push) were associated with this bug in the last 7 days.

This is the #23 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-central: 33

Platform breakdown:
* windows10-64-ccov: 33

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1499124&startday=2018-12-31&endday=2019-01-06&tree=all

Comment 14

[Cristina Coroiu [:ccoroiu]]

I did a backfill and before that push, the jobs are green: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&resultStatus=success%2Cpending%2Crunning%2Ctestfailed%2Cbusted%2Cexception&classifiedState=unclassified&searchStr=windows%2C10%2Cx64%2Cccov%2Cdebug%2Creftests%2Cwith%2Ce10s%2Ctest-windows10-64-ccov%2Fdebug-reftest-e10s-1%2Cr-e10s%28r1%29&tochange=9495b83aad6421af9b2f00cb248257d845344e04&fromchange=49233ec55bff867eaa73b1e5c5eb6d2e639da1ad&selectedJob=220195542

All the jobs are failing on mozilla-central ccov:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2019-01-01&endday=2019-01-08&tree=trunk&bug=1499124

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=windows%2C10%2Cx64%2Cccov%2Cdebug%2Creftests%2Cwith%2Ce10s%2Ctest-windows10-64-ccov%2Fdebug-reftest-e10s-1%2Cr-e10s%28r1%29&tochange=aac6849c6ff0eb4981b569988a094f04e42f14dd&fromchange=e54046e15abef84a2cd6ec0fc4825a6cf15878ac&selectedJob=220457694

Comment 15

[Intermittent Failures Robot]

40 failures in 2627 pushes (0.015 failures/push) were associated with this bug in the last 7 days.

This is the #26 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 1
• mozilla-central: 39

Platform breakdown:

• android-em-4-3-armv7-api16: 1
• windows10-64-ccov: 39

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1499124&startday=2019-01-07&endday=2019-01-13&tree=all

Comment 16

[Dorel Luca [:dluca]]

This bug failed 40 times in the last 7 days, Occurs only on windows10-64-ccov on debug.

Recent log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=222100500&repo=mozilla-central&lineNumber=33927

INFO - REFTEST TEST-START | http://localhost:50385/1547598152908/160/font-face/font-display-1.html == http://localhost:50385/1547598152908/160/font-face/font-display-1-ref.html
01:12:40 INFO - REFTEST TEST-LOAD | http://localhost:54641/1547601044286/13/font-face/font-display-1.html | 122 / 142 (85%)
01:12:40 INFO - ++DOMWINDOW == 12 (000001230E1E6000) [pid = 3052] [serial = 345] [outer = 000001230DCB0800]
01:12:44 INFO - --DOMWINDOW == 11 (000001230E0BD800) [pid = 3052] [serial = 337] [outer = 0000000000000000] [url = http://localhost:54641/1547601044275/10/font-face/reflow-sanity-1-ref.html]
01:12:44 INFO - --DOMWINDOW == 10 (000001230E0B9000) [pid = 3052] [serial = 336] [outer = 0000000000000000] [url = http://localhost:54641/1547601044275/10/font-face/reflow-sanity-delay-1c.html]
01:12:44 INFO - --DOMWINDOW == 9 (000001230E0C6800) [pid = 3052] [serial = 338] [outer = 0000000000000000] [url = data:text/html;charset=UTF-8,%3C%21%2D%2DCLEAR%2D%2D%3E]
01:12:44 INFO - --DOMWINDOW == 8 (000001230E065400) [pid = 3052] [serial = 335] [outer = 0000000000000000] [url = data:text/html;charset=UTF-8,%3C%21%2D%2DCLEAR%2D%2D%3E]
01:12:45 INFO - REFTEST TEST-LOAD | http://localhost:54641/1547601044286/13/font-face/font-display-1-ref.html | 122 / 142 (85%)
01:12:46 INFO - ++DOMWINDOW == 9 (000001230E0BAC00) [pid = 3052] [serial = 346] [outer = 000001230DCB0800]
01:12:47 INFO - REFTEST TEST-UNEXPECTED-FAIL | http://localhost:50385/1547598152908/160/font-face/font-display-1.html == http://localhost:50385/1547598152908/160/font-face/font-display-1-ref.html | image comparison, max difference: 255, number of differing pixels: 5633
01:12:47 INFO - REFTEST IMAGE 1 (TEST): data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAyAAAAPoCAYAAAAmy5qxAAAd3klEQVR4nO3dMYicZ3rA8e22UHGoUHUc4tg6qIg7Vb7OjUHVVVsFQVS4uEAaB6LqAmqukrFBkOYKFUqhIgGfQBA4iAs5BwbBuVHh5ArjQpWL7d4U8SU3nl3tt9LM/5tv5veDpzzdSPOA3//wzTtHAwAAIHI09wsAAAAOhwABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAgQAAMgIEAAAICNAAACAjAABAAAyAmTF9+O7b74Z3yx2vhvfz/1PyAzsLQCwHAJkxdNxenQ0jhY7p+Pp3P+EzMDeAgDLIUBWOMixRPYWAFgOAbLCQY4lsrcAwHIIkBUOciyRvQUAlkOArHCQY4nsLQCwHAJkhYMcS2RvAYDlECArHORYInsLACyHAFnhIMcS2VsAYDkEyAoHOZbI3gIAyyFAVjjIsUT2FgBYDgGywkGOJbK3AMByCJAVDnIskb0FAJZDgKxwkGOJ7C0AsBwCZIWDHEtkbwGA5RAgKxzkWCJ7CwAshwBZ4SDHEtlbAGA5BMgKBzmWyN4CAMshQFY4yLFE9hYAWA4BssJBjiWytwDAcggQAAAgI0AAAICMAAEAADICBAAAyAgQAAAgI0AAAICMAAEAADICBAAAyAgQAAAgs/wAef3V+PzJk/HEbHg+H1+9nvvN3WP21t4CwIFafoC8fDBuHR2NI7PhuTUevJz7zd1j9tbeAsCBEiDGQW4O9tbeAsCBEiDGQW4O9tbeAsCBEiDGQW4O9tbeAsCBEiDGQW4O9tbeAsCBEiDGQW4O9tbeAsCBEiDGQW4O9tbeAsCBEiDGQW4O9tbeAsCBEiDGQW4O9tbeAsCBEiDGQW4O9tbeAsCBEiDGQW4O9tbeAsCBEiDGQW4O9tbeAsCBEiDGQW4O9tbeAsCBEiDGQW4O9tbeAsCBEiDGQW4O9tbeAsCBEiDGQW4O9tbeAsCBEiDGQW4O9tbeAsCBEiDGQW4O9tbeAsCBWn6AfP3p+ODmzXHTbHg+GJ9+Pfebu8fs7ZbG3gLArlt+gAAAAIshQAAAgIwAAQAAMgIEAADICBAAACAjQAAAgIwAAQAAMgIEAADICBAAACAzT4C8eDTu3r1rDmgevZhl0zbL3h7c7MXeAsCOmSdAnp6Oo6Mjc0Bz+nSWTdsse3twsxd7CwA7RoAYB7mp7O3BzV7sLQDsGAFiHOSmsrcHN3uxtwCwYwSIcZCbyt4e3OzF3gLAjhEgxkFuKnt7cLMXewsAO0aAGAe5qeztwc1e7C0A7BgBYhzkprK3Bzd7sbcAsGMEiHGQm8reHtzsxd4CwI4RIMZBbip7e3CzF3sLADtGgBgHuans7cHNXuwtAOwYAWIc5Kaytwc3e7G3ALBjBIhxkJvK3h7c7MXeAsCOESDGQW4qe3twsxd7CwA7RoAYB7mp7O3BzV7sLQDsGAFiHOSmsrcHN3uxtwCwYwSIcZCbyt4e3OzF3gLAjhEgxkFuKnt7cLMXewsAO0aAGAe5qeztwc1e7C0A7Jh5AuTFo3H37l1zQPPoxSybtln29uBmL/YWAHbMPAECAAAcJAECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARICu+H9998834ZrHz3fh+7n9CAAB4AwGy4uk4PToaR4ud0/F07n9CAAB4AwGyQoAAAMA2CZAVAoS5nI1n966Po6Pjcefx67lfDExkbwG4OgGyQoAwk29/Oz48/uF9vP1wvJr79cAU9haAtyBAVggQ5vHl/ZO/eB9vjI+en839kuBS9haAtyFAVggQZnD2bNy7vvpeHt95PDzQwk6ztwC8JQGyQoDQe/34zjheey9vj4eeZ2GH2VsA3pYAWSFAqL0aD2+f/36e3P9y7hcHF7C3ALw9AbJCgBD78v44uej9vH5vPPNIPbvI3gLwDgTICgFC6c9XmF70frralF1kbwF4NwJkhQAh9PrxuHN8yXvqalN2jb0F4B0JkBUChM7qFaYXzcnwSD27xN4C8K4EyAoBQuScK0wvmuv3ng2P1LMT7C0AGyBAVggQGudfYXrBHN8ZHqlnF9hbADZBgKwQIBQuvsL0orntxxWYnb0FYDMEyAoBQuBNV5heNCf3h0fqmZW9BWBDBMgKAcK2nXOF6bVr49ql7+31cc+PKzAbewvA5giQFQKELTvnCtPbD38/6dGW4zuPh0fqmYW9BWCDBMgKAcJ2vXxwa/U9++FXo6d9uff28Eg9c7C3AGySAIHKOVeYnvz5xxImXm964scVqNlbADZMgEBk/dPi1U+GJ/3A2w+fPEPF3gKwaQIEEutXmK49G//qs/H+8SUHuaPjccePK5CxtwBsngCBwtoVpjfGR89//JHwOTcNnTe3Hw6P1JOwtwBsgQCBrTvngHbRYWzSby2cDI/Us332FoDtECCwbWtXmL7pcZRpvzZ9/d6z4ZF6tsreArAlAgS27NXD21f6Qu6kq02P7wyP1LNN9haAbREgsE1vusL0Cv+b8+a2H1dgW+wtAFskQGCL1j4VnvgJ8KSrTU/uD4/Usw32FoBtWn6AvP5qfP7kyXhiNjyfj688KvGO1p+Ln/wM/KuH4/alX+q9Pu75cQU2zt4CsF3LD5CXD8atS/+DZ64+t8aDl3O/uQu3djPQVW4Bej0e3zm+9H1a+00GeFf2FoAtEyBGgGzFFa4wvehPeHZvXL/0fVr9VWp4N/YWgO0TIEaAbMPaFabXx+nTq37m++W4f3L5e3Xpl4NhKnsLQECAGAGyBWtXmL7lF29fffb+5VebXnI9KkxlbwEoCBAjQDbt7Pn46Mbqv+dbXz066WrTN/1AHExkbwGICBAjQDbsba8wvcikq02v+Jw+/Ji9BaAiQIwA2ah3uML0wj9yytWmV7mpCH7M3gLQESBGgGzS2hWmm7jtZ9rVpu98YORw2VsAQgLECJCNWb/CdFO/dzDpatN3fGSGQ2VvAWgJECNANuWcK0w394vP0642fesvDXO47C0AMQFiBMiGbOoK08l//nmz4f9P9p+9BaAmQIwA2YSzL8bHK5/0Ho/3P9vwp7prn1SfN5v89Jq9Z28BmIEAMQJkA9auMN3Kj6ytP6t/7rjalInsLQBzECBGgLyz9StM5x1XmzKFvQVgHgLEXDACZLK1K0znH1ebcil7C8BMBIi5YATINBMfL6nH1aa8kb0FYD4CxFwwAmSSc64w/fCfnownT7Y5/zDev/RLva425Q3sLQAzEiBGgLyDbV9her6Jn1672pQL2FsA5rT8APn60/HBzZvjptnwfDA+/XruN3fXrf/IWvbp7aTn911tynnsLQDzWn6AwEzWrzA9HU+z59en3WB0fOfx8Eg9f8neAjA3AQJvZf0gdRLfIbp2kDx3bg+P1PP/7C0A8xMg8DbWHiWZ4cB09mzcu375p8n1AZMdZm8B2AECBK5s/cu0cz0y8uX9k8u/1LuVX7dmeewtALtBgMBVnXOF6Wxfmn31cNy+9HGW43HHjytgbwHYEQIErmieK0wv8no8vnN8+afJtx8Oj9QfNnsLwK4QIHAlM15heoGzZ/fG9Us/TT4ZHqk/ZPYWgN0hQOAKXj89XT00Hd8Z8z8lsn64PG+u33s2PFJ/mOwtALtEgMBk61eY7srhaNKXenfi0EnP3gKwW+YJkBePxt27d80BzaMXs2zaRp09/2jcWDkc/Wx8/MXcr+oHLx+MW5c+zuJq00NkbwHYNfMEyNPTyz/1Mns1p09n2bQNejU+e/9HX5r92cdjV85xY7wcD25NeC98mnxg7C0Au0eAmGSWHiDnfWH2+G/+be6XteL5R9cnvRcnH3+xE4/fsH32FoBdJEBMMosOkLMvxsfnfFn2vd98PfcrW/H6tx9Oez+O3xu/eTn3q2Xr7C0AO0qAmGSWGyBn46tf3zr37/T+Z/8194tbcfYvv5z8fhy/92B85ePkPWZvAdhdAsQks9QA+fbp6Y++wPsXh6EPfzu+nfsF/p+z8ezetEdZ/jw3Tp/u0Otnk+wtALtMgJhklhcgZ+O///3j8d7xm/5ex+O9X//n+H7ulzrG+Pb3fz/pNqEfv/6f/vKfxx934S/AhthbAHafADHJLCVAvv/uD+P5J/84Pvzrn4zjiX+3az//xfi7T/51/Mcf/zRel4+HnL0ef/rD8/HJ3/7VuPYu78+1n49f/N0n4/kfvtuJQylXZ28BWBIBYpLZ+QD53a/GjWsb+vteuzFufvDp2M5Xfb8en35wc/z0J8ebea1rczx+8tOb41e/28qLZ9Psrb0FWCABYpLZ+QDZ9E7eejC2c2HPxN9N2Pf3i/9lb+0twAIJEJOMgwEAAGMIECNAAAAICRAjQAAAyAgQI0AAAMgIECNAAADICBAjQAAAyAgQI0AAAMgIECNAAADICBAjQAAAyMwTIC8ejbt375oDmkcvZtk0AAB2zDwBAgAAHCQBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZP4HkF/5l9A5SsUAAAAASUVORK5CYII=
01:12:47 INFO - REFTEST IMAGE 2 (REFERENCE): data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAyAAAAPoCAYAAAAmy5qxAAAeS0lEQVR4nO3dMchd53nA8W/7Bg1Bg6cQRNBcNNSbJmfzYtCUSVMRVIOHBLq4UE0paMkkY4OgSwYN6qChBUcgKATqQW7AIIgXDW4zGA+aPGh7u9itr+/99B3Zn/7nnHt+P3jGKFe6D+T9597z3pMBAAAQOZn7BQAAANshQAAAgIwAAQAAMgIEAADICBAAACAjQAAAgIwAAQAAMgIEAADICBAAACAjQAAAgIwAAQAAMgIEAADICBAAACAjQAAAgIwAAQAAMgIEAADICBAAACAjQAAAgIwAAQAAMgIEAADICBAAACAjQAAAgIwAAQAAMgIEAADICBAAACAjQAAAgIwAAQAAMgIEAADICBAAACAjQAAAgIwAAQAAMgIEAADICBAAACAjQAAAgIwAAQAAMgIEAADICBAAACAjQAAAgIwAAQAAMgIEAADICBAAACAjQAAAgIwAAQAAMgIEAADICBAAACAjQAAAgIwA2fHN+PrLL8eXq52vxzdz/xMyA3sLAKyHANnxaNw8ORknq52b49Hc/4TMwN4CAOshQHY4yLFG9hYAWA8BssNBjjWytwDAegiQHQ5yrJG9BQDWQ4DscJBjjewtALAeAmSHgxxrZG8BgPUQIDsc5FgjewsArIcA2eEgxxrZWwBgPQTIDgc51sjeAgDrIUB2OMixRvYWAFgPAbLDQY41srcAwHoIkB0OcqyRvQUA1kOA7HCQY43sLQCwHgJkh4Mca2RvAYD1ECA7HORYI3sLAKyHANnhIMca2VsAYD0EyA4HOdbI3gIA6yFAdjjIsUb2FgBYDwGyw0GONbK3AMB6CBAAACAjQAAAgIwAAQAAMgIEAADICBAAACAjQAAAgIwAAQAAMgIEAADICBAAACCz/gB58fn45OHD8dBc8HwyPn8x95t7xOytvQWAjVp/gDy7O66dnIwTc8Fzbdx9Nvebe8Tsrb0FgI0SIMZBbg721t4CwEYJEOMgNwd7a28BYKMEiHGQm4O9tbcAsFECxDjIzcHe2lsA2CgBYhzk5mBv7S0AbJQAMQ5yc7C39hYANkqAGAe5OdhbewsAGyVAjIPcHOytvQWAjRIgxkFuDvbW3gLARgkQ4yA3B3trbwFgowSIcZCbg721twCwUQLEOMjNwd7aWwDYKAFiHOTmYG/tLQBslAAxDnJzsLf2FgA2SoAYB7k52Ft7CwAbJUCMg9wc7K29BYCNEiDGQW4O9tbeAsBGrT9AvvhovHvlyrhiLnjeHR99Mfebe8Ts7RsaewsAS7f+AAEAAFZDgAAAABkBAgAAZAQIAACQESAAAEBGgAAAABkBAgAAZAQIAACQESAAAEBmngB5en/cunXLbGjuP51l0y6Wvd3cHMXeAsDCzBMgj26Ok5MTs6G5+WiWTbtY9nZzcxR7CwALI0CMg9xU9nZzcxR7CwALI0CMg9xU9nZzcxR7CwALI0CMg9xU9nZzcxR7CwALI0CMg9xU9nZzcxR7CwALI0CMg9xU9nZzcxR7CwALI0CMg9xU9nZzcxR7CwALI0CMg9xU9nZzcxR7CwALI0CMg9xU9nZzcxR7CwALI0CMg9xU9nZzcxR7CwALI0CMg9xU9nZzcxR7CwALI0CMg9xU9nZzcxR7CwALI0CMg9xU9nZzcxR7CwALI0CMg9xU9nZzcxR7CwALI0CMg9xU9nZzcxR7CwALI0CMg9xU9nZzcxR7CwALI0CMg9xU9nZzcxR7CwALI0CMg9xU9nZzcxR7CwALM0+APL0/bt26ZTY095/OsmkXy95ubo5ibwFgYeYJEAAAYJMECAAAkBEgAABARoAAAAAZAQIAAGQECAAAkBEgAABARoAAAAAZAQIAAGQECAAAkBEgAABARoAAAAAZAQIAAGQECAAAkBEgAABARoAAAAAZAQIAAGQECAAAkBEgAABARoAAAAAZAQIAAGQECAAAkBEgAABARoAAAAAZAQIAAGQECAAAkBEgAABARoAAAAAZAQIAAGQECAAAkBEgAABARoAAAAAZAQIAAGQECAAAkBEgAABARoAAAAAZAQIAAGQECAAAkBEgAABARoAAAAAZAQIAAGQECAAAkBEgAABARoAAAAAZAQIAAGQECAAAkBEgAABARoAAAAAZAQIAAGQECAAAkBEgAABARoDs+GZ8/eWX48vVztfjm7n/CZmBvQUA1kOA7Hg0bp6cjJPVzs3xaO5/QmZgbwGA9RAgOxzkWCN7CwCshwDZ4SDHGtlb5vJyPL59eZycnI4bD17M/WIAWAkBssNBjjWyt8zkqz+M906/fR+v3xvP5349AKyCANnhIMca2Vvm8dmdq997H98a7z95OfdLAmAFBMgOBznWyN4yg5ePx+3Lu+/l6Y0HwxexADiPANnhIMca2Vt6Lx7cGKd77+X1cc/3sAA4hwDZ4SDHGtlbas/HveuH38+rdz6b+8UBsHACZIeDHGtkb4l9dmdcPev9vHx7PPYoCACvIEB2OMixRvaW0ndX7571frqSF4BXEyA7HORYI3tL6MWDceP0nPfUlbwAvIIA2eEgxxrZWzq7V++eNVeHR0EAOIsA2eEgxxrZWyIHrt49ay7ffjw8CgLAIQJkh4Mca2RvaRy+eveMOb0xPAoCwCECZIeDHGtkbymcffXuWXPdj4IAcIAA2eEgxxrZWwKvunr3rLl6Z3gUBIAfEiA7HORYI3vLm3bg6t1Ll8alc9/by+O2HwUB4AcEyA4HOdbI3vKGHbh69/q9P036StbpjQfDoyAAfJ8A2eEgxxrZW96sZ3ev7b5n3/7a+bSH0q8Pj4IA8H0CBICzHbh69+p3P/Ix8Vreq34UBIDvESAAnGn/U47dTzQm/TDht5+YAMAYAgSAM+1fvbv3TMfzj8c7p+d9CnI6bvhREAC+JUAAOGzv6t23xvtPfvhRxoEbsg7N9XvDoyAAjCFAADjoQFicFRGTfiPk6vAoCABjCBAADtm7evdVX6Oa9ivpl28/Hh4FAUCAALDn+b3rr/Ug+aQreU9vDI+CACBAANj1qqt3X+M/c2iu+1EQgM0TIADs2Ps0Y+InF5Ou5L16Z3gUBGDb1h8gLz4fnzx8OB6aC55Pxue+KvHm2Ft7u1j7z3NMfnbj+b1x/dyH0S+P234UBGDT1h8gz+6Oa+f+D555/bk27j6b+809YvbW3i7V3o1Wr3N71Yvx4Mbpue/T3m+JALApAsQ4yM3B3trbRXqNq3fP+hMe3x6Xz32fdn9NHYBtESDGQW4O9tbeLtHe1buXx81Hr/tZxWfjztXz36tzH2oH4GgJEOMgNwd7a28XaO/q3R/5wPjzj985/0rec671BeB4CRDjIDcHe2tvl+blk/H+W7v/nj/6ytxJV/K+6ocNAThmAsQ4yM3B3trbhfmxV++eZdKVvK/5fAkAx0GAGAe5Odhbe7soP+Hq3TP/yClX8r7ODVsAHAsBYhzk5mBv7e2S7F29exG3VE27kvcnhw4AqyNAjIPcHOytvV2M/at3L+p3OiZdyfsTv+oFwPoIEOMgNwd7a2+X4sDVuxf3S+XTruT90Q+7A7BKAsQ4yM3B3trbhbioq3cn//mH5oL/OwFYNgFiHOTmYG/t7RK8/HR8sPMJxel45+ML/jRi7xOWQ3ORn7oAsHQCxDjIzcHe2tsF2Lt69438OOD+MyYHx5W8AJshQIyD3Bzsrb2d3f7Vu/OOK3kBtkKAGAe5Odhbezu3vat35x9X8gJsgwAxDnJzsLf2dlYTvxZVjyt5ATZBgBgHuTnYW3s7pwNX7773zw/Hw4dvcv5xvHPuw+iu5AXYAgFiHOTmYG/t7Yze9NW7h0381MWVvABHb/0B8sVH490rV8YVc8Hz7vjoi7nf3CNmb9/Q2Nvz7f84YPapw6TnTlzJC3Ds1h8gAEy2f/XuzfEoe+5i2s1bpzceDI+CABwvAQKwGfsBcDW++3YvgA7O9eFREIDjJUAAtmLvK1AzHPRfPh63L5//KUgdRgB0BAjAJuw/BD7XV50+u3P1/IfR38ivsgOwBAIEYAsOXL0728Pez++N6+d+Det03PCjIABHSYAAbMA8V++e5cV4cOP0/E9Brt8bHgUBOD4CBODozXj17hlePr49Lp/7KcjV4VEQgOMjQACO3ItHN3cP+6c3xvzfbtqPokNz+fbj4VEQgOMiQACO2v7Vu0s51E96GH0RsQTARZonQJ7eH7du3TIbmvtPZ9m0i2VvNzfHsLcvn7w/3to51P9ifPDp3K/qW8/ujmvnfg3LlbwAx2aeAHl08/z/18sc1dx8NMumXSx7u7lZ/94+Hx+/84OHvX/xwVhKf4zxbNy9NuG98CkIwFERICaZ9R/khr3d4Kx9bw896H36d/8+98va8eT9y5Pei6sffLqIr40B8NMJEJPM2g9yYwx7u8FZ9d6+/HR8cOAh77d//8Xcr2zHiz+8N+39OH17/P7Z3K8WgIsgQEwyqz7Ifcfebm7Wu7cvx+e/u3bw7/TOx/8994vb8fJffz35/Th9++743McgAKsnQEwy6z3IfY+93dysdW+/enTzBw+e//+cvveH8dXcL/D/vByPb0/7CtZ389bNRwt6/QD8GALEJLPWg9wOe7u5Wd/evhz/8x8fjLdPX/X3Oh1v/+6/xjdzv9Qxxld/+odJt2D98PX//Nf/Mv6yhL8AAD+KADHJrO8gd4C93dysZW+/+frP48mH/zTe+9ufjdOJf7dLv/zV+O2H/zb+8y9/HS/KrzW9fDH++ucn48O//5tx6ae8P5d+OX712w/Hkz9/vYiYAmA6AWKSWctB7pXs7eZm8Xv7x9+Mty5d0N/30lvjyrsfjTfziPoX46N3r4yf/+z0Yl7r3pyOn/38yvjNH9/IiwfgggkQk8ziD3JT2NvNzeL39qJ38trd8WYumpr4ex/H/n4BMMYQICaaozgY2NvNzVHsLQAsjAAxDnJT2dvNzVHsLQAsjAAxDnJT2dvNzVHsLQAsjAAxDnJT2dvNzVHsLQAsjAAxDnJT2dvNzVHsLQAsjAAxDnJT2dvNzVHsLQAsjAAxDnJT2dvNzVHsLQAsjAAxDnJT2dvNzVHsLQAsjAAxDnJT2dvNzVHsLQAszDwB8vT+uHXrltnQ3H86y6ZdLHu7uTmKvQWAhZknQAAAgE0SIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQEaAAAAAGQECAABkBAgAAJARIAAAQOZ/ATPjePtYqqi7AAAAAElFTkSuQmCC
01:12:47 INFO - REFTEST INFO | Saved log: START http://localhost:54641/1547601044286/13/font-face/font-display-1.html
01:12:47 INFO - REFTEST INFO | Saved log: [CONTENT] OnDocumentLoad triggering AfterOnLoadScripts
01:12:47 INFO - REFTEST INFO | Saved log: Initializing canvas snapshot
01:12:47 INFO - REFTEST INFO | Saved log: DoDrawWindow 0,0,800,1000
01:12:47 INFO - REFTEST INFO | Saved log: [CONTENT] RecordResult fired
01:12:47 INFO - REFTEST INFO | Saved log: RecordResult fired
01:12:47 INFO - REFTEST INFO | Saved log: START http://localhost:54641/1547601044286/13/font-face/font-display-1-ref.html
01:12:47 INFO - REFTEST INFO | Saved log: [CONTENT] OnDocumentLoad triggering AfterOnLoadScripts
01:12:47 INFO - REFTEST INFO | Saved log: Initializing canvas snapshot
01:12:47 INFO - REFTEST INFO | Saved log: DoDrawWindow 0,0,800,1000
01:12:47 INFO - REFTEST INFO | Saved log: [CONTENT] RecordResult fired
01:12:47 INFO - REFTEST INFO | Saved log: RecordResult fired
01:12:47 INFO - REFTEST TEST-END | http://localhost:50385/1547598152908/160/font-face/font-display-1.html == http://localhost:50385/1547598152908/160/font-face/font-display-1-ref.html

jfkthame: Can you please take a look at this bug?

Comment 17

[Jonathan Kew [:jfkthame]]

Occurs only on windows10-64-ccov on debug

The test depends on timing (necessarily, because the font-display descriptor is all about timing and fallbacks), and ccov-debug builds are slow. Maybe we can try increasing the gfx.downloadable_fonts.fallback_delay{_short} settings when running the test (and adjusting the delay in the test) so that it's more tolerant of running really slowly. I'll push a try job and see what happens.

Comment 18

[Jonathan Kew [:jfkthame]]

Attachment: Increase font-loading timeouts for the font-display-1 reftest to allow for slow debug builds

This appears to let the font-display test run successfully even on ccov-debug builds, according to try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=9c68f4b393ec436ea7e166efe40b4192bbd6552b&searchStr=Windows%2C10%2Cx64%2CCCov%2Cdebug&selectedJob=222193690. As a bonus, it looks like it'll let us remove the existing skip/random annotations and resolve bug 1238222 at the same time.

Comment 19

[Emilio Cobos Álvarez [:emilio]]

Comment on attachment 9036994 [details] [diff] [review]
Increase font-loading timeouts for the font-display-1 reftest to allow for slow debug builds

Review of attachment 9036994 [details] [diff] [review]:
-----------------------------------------------------------------

Looks reasonable, r=me, thanks for looking into it!

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/6f2303192174
Increase font-loading timeouts for the font-display-1 reftest to allow for slow debug builds. r=emilio

Comment 21

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/6f2303192174

Comment 23

[Intermittent Failures Robot]

16 failures in 2378 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 16

Platform breakdown:

• windows10-64-ccov: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1499124&startday=2019-01-14&endday=2019-01-20&tree=all