1193641 - (canvas?) text rendering fundamentally broken in latest Developer Edition

Status: RESOLVED FIXED

(Core :: Graphics: Text, defect)

Description

[Katelyn Gadd (:kael)]

As of the latest Developer Edition update, text rendering (in canvas? maybe elsewhere) is completely broken. PDFs in pdf.js have no text, and the same is true for the github network view.

Text in the Firefox UI also looks different (tabs, address bar, etc). Thinner strokes. Looks like the rendering mode changed or something?

Normal DOM text in pages seems to render fine and look the same as it did before the update.

-----------------------

Application Basics
------------------

Name: Firefox
Version: 41.0a2
Build ID: 20150810004008
Update Channel: aurora
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:41.0) Gecko/20100101 Firefox/41.0
Multiprocess Windows: 0/2 (default: false)

Crash Reports for the Last 3 Days
---------------------------------

All Crash Reports

Extensions
----------

Name: Rikaichan
Version: 2.09.1-signed
Enabled: true
ID: {0AA9101C-D3C1-4129-A9B7-D778C6A17F82}

Name: Rikaichan Japanese-English Dictionary File
Version: 2.01.130701.1-signed
Enabled: true
ID: [e-mail]

Name: uBlock Origin
Version: 1.0.0.1
Enabled: true
ID: [e-mail]

Name: User Style Manager
Version: 1.1.1.1-signed
Enabled: true
ID: UserStyleManager@girishsharma

Graphics
--------

Adapter Description: NVIDIA GeForce GTX 970
Adapter Drivers: nvd3dumx,nvwgf2umx,nvwgf2umx,nvwgf2umx nvd3dum,nvwgf2um,nvwgf2um,nvwgf2um
Adapter RAM: 4095
Asynchronous Pan/Zoom: none
Device ID: 0x13c2
Direct2D Enabled: Blocked for your graphics driver version.
DirectWrite Enabled: true (10.0.10240.16390)
Driver Date: 7-22-2015
Driver Version: 10.18.13.5362
GPU #2 Active: false
GPU Accelerated Windows: 2/2 Direct3D 11 (OMTC)
Subsys ID: 39753842
Supports Hardware H264 Decoding: true
Vendor ID: 0x10de
WebGL Renderer: Google Inc. -- ANGLE (NVIDIA GeForce GTX 970 Direct3D11 vs_5_0 ps_5_0)
windowLayerManagerRemote: true
AzureCanvasBackend: skia
AzureContentBackend: cairo
AzureFallbackCanvasBackend: cairo
AzureSkiaAccelerated: 0

Important Modified Preferences
------------------------------

accessibility.typeaheadfind.flashBar: 0
browser.cache.disk.capacity: 358400
browser.cache.disk.filesystem_reported: 1
browser.cache.disk.smart_size.first_run: false
browser.cache.disk.smart_size.use_old_max: false
browser.cache.frecency_experiment: 4
browser.download.importedFromSqlite: true
browser.download.manager.alertOnEXEOpen: false
browser.fixup.domainwhitelist.firehomie: true
browser.fixup.domainwhitelist.rv325: true
browser.places.smartBookmarksVersion: 7
browser.sessionstore.upgradeBackup.latestBuildID: 20150810004008
browser.startup.homepage_override.buildID: 20150810004008
browser.startup.homepage_override.mstone: 41.0a2
browser.tabs.warnOnClose: false
browser.tabs.warnOnOpen: false
dom.apps.reset-permissions: true
dom.mozApps.used: true
extensions.lastAppVersion: 41.0a2
font.internaluseonly.changed: false
font.name.monospace.x-western: Consolas
font.name.sans-serif.x-western: Calibri
font.name.serif.x-western: Cambria
font.size.fixed.x-western: 15
font.size.variable.x-western: 18
gfx.blacklist.direct2d: 3
gfx.blacklist.layers.direct3d11: 3
gfx.direct3d.last_used_feature_level_idx: 0
gfx.driver-init.appVersion: 41.0a2
gfx.driver-init.deviceID: 0x13c2
gfx.driver-init.driverVersion: 10.18.13.5362
gfx.driver-init.feature-d2d: true
gfx.driver-init.feature-d3d11: true
gfx.driver-init.status: 2
gfx.font_rendering.cleartype_params.cleartype_level: 40
gfx.font_rendering.cleartype_params.force_gdi_classic_for_families:
gfx.font_rendering.cleartype.always_use_for_content: true
gfx.font_rendering.directwrite.enabled: true
media.gmp-gmpopenh264.lastUpdate: 1430793755
media.gmp-gmpopenh264.version: 1.4
media.gmp-manager.buildID: 20150810004008
media.gmp-manager.lastCheck: 1439347809
media.hardware-video-decoding.failed: false
network.cookie.prefsMigrated: true
network.predictor.cleaned-up: true
network.prefetch-next: false
places.database.lastMaintenance: 1439126586
places.history.expiration.transient_current_max_pages: 104858
plugin.disable_full_page_plugin_for_types: application/pdf
plugin.importedState: true
plugin.state.flash: 1
plugin.state.npctrl: 2
plugin.state.npgoogleupdate: 0
plugin.state.npnv3dv: 0
plugin.state.npnv3dvstreaming: 0
privacy.cpd.cookies: false
privacy.cpd.downloads: false
privacy.cpd.formdata: false
privacy.cpd.history: false
privacy.cpd.sessions: false
privacy.donottrackheader.enabled: true
privacy.sanitize.migrateFx3Prefs: true
privacy.sanitize.timeSpan: 0
privacy.trackingprotection.enabled: true
security.disable_button.openCertManager: false
storage.vacuum.last.index: 1
storage.vacuum.last.places.sqlite: 1438073345

Important Locked Preferences
----------------------------

JavaScript
----------

Incremental GC: true

Accessibility
-------------

Activated: false
Prevent Accessibility: 0

Library Versions
----------------

NSPR
Expected minimum version: 4.10.8
Version in use: 4.10.8

NSS
Expected minimum version: 3.19.2 Basic ECC
Version in use: 3.19.2 Basic ECC

NSSSMIME
Expected minimum version: 3.19.2 Basic ECC
Version in use: 3.19.2 Basic ECC

NSSSSL
Expected minimum version: 3.19.2 Basic ECC
Version in use: 3.19.2 Basic ECC

NSSUTIL
Expected minimum version: 3.19.2
Version in use: 3.19.2

Experimental Features
---------------------

Comment 1

[Katelyn Gadd (:kael)]

gfx.font_rendering.directwrite.enabled appears to be the 'problem' pref here. I've had it set for multiple versions, but if I 'reset' it, it switches to false and text in the github network view reappears.

Unfortunately this makes text look positively atrocious compared to how it used to look. The UI chrome text still looks weird (thinner strokes).

Comment 2

[u279076]

I cannot reproduce this issue.

1) Please attach a screenshot and any relevant URLs exhibiting this problem
2) Please test Aurora with a new profile
3) Please test with the latest Nightly build from https://nightly.mozilla.org/

If the problem continues to reproduce I'll need to you work on a regression range. I'll provide further instructions after you've reported back on the above.

Comment 3

[Katelyn Gadd (:kael)]

Attachment: Screenshot 2015-08-12 15.58.09.png

https://github.com/sq/JSIL/network has no text. This also completely breaks pdf.js for all PDFs.

This won't reproduce with a new profile because it's caused by directwrite being enabled. I mentioned this in one of the comments already and called out the specific pref.

Comment 4

[Katelyn Gadd (:kael)]

Whatever bug caused this doesn't reproduce in 42.0a1 (2015-08-11).

Comment 5

[Katelyn Gadd (:kael)]

Attachment: mozregression.png

Mozregression hung while bisecting nightlies, but here's how far it got.

It was unable to find a range where the feature broke, but it found a range where it was fixed (which is why it works in nightly).

Maybe this is caused by a windows update? I know my graphics drivers haven't changed, but Win10 did install updates recently.

Comment 6

[Katelyn Gadd (:kael)]

My money's on https://hg.mozilla.org/mozilla-central/rev/b7b4f7c13d92 .

Comment 7

[Katelyn Gadd (:kael)]

In aurora, about:support shows 'Direct2D Enabled	Blocked for your graphics driver version.' but nightly shows 'Direct2D Enabled	true'.

The acceleration mode is different too:
Aurora's is 'GPU Accelerated Windows	2/2 Direct3D 11 WARP (OMTC)'
Nightly's is 'GPU Accelerated Windows	1/1 Direct3D 11 (OMTC)'


Do the driver/device blacklists update separately from builds? That would explain why I couldn't find a regression range for this breaking, only for the fix.

Comment 8

[Katelyn Gadd (:kael)]

My video card/driver/OS configuration isn't in the list on https://wiki.mozilla.org/Blocklisting/Blocked_Graphics_Drivers , but it looks outdated. Despite the guidance on that page, setting gfx.direct2d.force-enabled and layers.acceleration.force-enabled does not actually bypass the driver blacklist.

Spoofing the device info also doesn't bypass it, it just changes the message:
'Blocked for your graphics card because of unresolved driver issues.'

Comment 9

[u279076]

I think your mozregression results are not necessarily reliable given the troubles you had. Best try to bisect this manually by installing and testing Nightly builds from archive.mozilla.org. You can match up the versions in Nightly to the dates here: https://wiki.mozilla.org/RapidRelease/Calendar

In the meantime, David, do you have any leads as to what might be happening here? I wonder if the recent blocklisting we pushed out might be giving :kael variable results.

Comment 10

[u279076]

As an aside, Milan, I'm wondering if you can get a NVIDIA GeForce GTX 970 in the Toronto lab for testing to see if we can get this reproduced. It might help to narrow this down.

Comment 11

[David Anderson [:dvander] - inactive, e-mail if emergency]

I would believe bug 1183910 fixed this. Previously, D2D (and dwrite) were initialized separately from D3D11, so if D3D11 was blocked for driver issues you could still end up using it for content rendering. bug 1183910 changed this to make D2D/DWrite dependent on having a D3D11 compositor.

:kael, do you still get this problem on the current Developer Edition, which should be Firefox 42? (there was an uplift on 8/10, I can't tell but it looks like your build was right before that.)

Milan, it looks like :kael's adapter falls under the Windows 10 blacklist for nVidia. If we're going to keep that, should we uplift a minimal change to beta to fix this?

Comment 12

[Katelyn Gadd (:kael)]

For some reason Developer Edition isn't offering me an update past 41.0a2 (2015-08-10). It says it's up to date. I'll try updating manually...

Comment 13

[Katelyn Gadd (:kael)]

I downloaded the latest x86-64 developer edition build and installed it. It's still 41.0a2 (2015-08-10), but this (temporarily, probably?) cleared the blacklist status on my GPU so I have fully accelerated D3D11 - so everything is fast and text looks right in pdf.js and the github network view, etc.

I assume this will break again once the blacklist kicks in :-)

Comment 14

[Milan Sreckovic [:milan] (needinfo for best results)]

(In reply to David Anderson [:dvander] from comment #11)
> I would believe bug 1183910 fixed this. Previously, D2D (and dwrite) were
> initialized separately from D3D11, so if D3D11 was blocked for driver issues
> you could still end up using it for content rendering. bug 1183910 changed
> this to make D2D/DWrite dependent on having a D3D11 compositor.
> 
> :kael, do you still get this problem on the current Developer Edition, which
> should be Firefox 42? (there was an uplift on 8/10, I can't tell but it
> looks like your build was right before that.)
> 
> Milan, it looks like :kael's adapter falls under the Windows 10 blacklist
> for nVidia. If we're going to keep that, should we uplift a minimal change
> to beta to fix this?

Are we talking about the change from bug 1183910? Doesn't that only help e10s, and we don't have to worry about that for the beta/release?

Comment 15

[Milan Sreckovic [:milan] (needinfo for best results)]

We'll grab a GeForce GT card that's on the list in bug 1189940, probably a 620, 630 or a 640.

Comment 16

[David Anderson [:dvander] - inactive, e-mail if emergency]

(In reply to Milan Sreckovic [:milan] from comment #14)
> (In reply to David Anderson [:dvander] from comment #11)
> > I would believe bug 1183910 fixed this. Previously, D2D (and dwrite) were
> > initialized separately from D3D11, so if D3D11 was blocked for driver issues
> > you could still end up using it for content rendering. bug 1183910 changed
> > this to make D2D/DWrite dependent on having a D3D11 compositor.
> > 
> > :kael, do you still get this problem on the current Developer Edition, which
> > should be Firefox 42? (there was an uplift on 8/10, I can't tell but it
> > looks like your build was right before that.)
> > 
> > Milan, it looks like :kael's adapter falls under the Windows 10 blacklist
> > for nVidia. If we're going to keep that, should we uplift a minimal change
> > to beta to fix this?
> 
> Are we talking about the change from bug 1183910? Doesn't that only help
> e10s, and we don't have to worry about that for the beta/release?

That bug also stopped using DWrite when D3D11 was blocked.

Comment 17

[Milan Sreckovic [:milan] (needinfo for best results)]

(In reply to David Anderson [:dvander] from comment #16)
> ...
> > 
> > Are we talking about the change from bug 1183910? Doesn't that only help
> > e10s, and we don't have to worry about that for the beta/release?
> 
> That bug also stopped using DWrite when D3D11 was blocked.

I'd be interested in seeing if we can get that approved for beta.  Let's put together a patch and try it, either on a new bug or here or on the original 1183910.

Comment 18

[David Anderson [:dvander] - inactive, e-mail if emergency]

Attachment: remove pref

Looking again, this was only a problem because the directwrite pref is really a "force enable" pref, not a soft enable. The gfxWindowsPlatform refactoring on aurora/nightly just removed the pref.

So if we want we can remove it on beta too, seems harmless.

Comment 19

[Milan Sreckovic [:milan] (needinfo for best results)]

Comment on attachment 8652896 [details] [diff] [review]
remove pref

Review of attachment 8652896 [details] [diff] [review]:
-----------------------------------------------------------------

::: gfx/thebes/gfxWindowsPlatform.cpp
@@ -507,5 @@
>      mRenderMode = RENDER_GDI;
>  
>      bool isVistaOrHigher = IsVistaOrLater();
>  
> -    mUseDirectWrite = Preferences::GetBool("gfx.font_rendering.directwrite.enabled", false);

Is it removing, or is it replacing it with:

mUseDirectWrite = false;

Comment 20

[David Anderson [:dvander] - inactive, e-mail if emergency]

Attachment: beta patch

It's just removing the confusing pref. Normally DWrite is pinned to whether or not we're using D2D. The pref really is for force-enabling DWrite which seems unnecessary, and is not allowed anymore on Aurora/Nightly.

Though I just realized this member variable is not initialized in the gfxWindowsPlatform constructor, this patch fixes that.

Comment 21

[David Anderson [:dvander] - inactive, e-mail if emergency]

Attachment: beta patch

Milan pointed out that we need to reset DWrite in UpdateRenderMode().

Comment 22

[David Anderson [:dvander] - inactive, e-mail if emergency]

Attachment: nightly patch

On nightly/aurora this pref doesn't do anything anymore, so we might as well just remove it.

Comment 23

[Milan Sreckovic [:milan] (needinfo for best results)]

Comment on attachment 8653547 [details] [diff] [review]
nightly patch

Review of attachment 8653547 [details] [diff] [review]:
-----------------------------------------------------------------

Removing a pref - do we need a release note for this?  I don't think it was heavily used, so perhaps not, but thought I'd ask.

Comment 24

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/b5e6116756b6

Comment 25

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/e5c83bfca854

Comment 26

[David Anderson [:dvander] - inactive, e-mail if emergency]

(First comment from pulsebot was wrong bug number.)

(In reply to Milan Sreckovic [:milan] from comment #23)
> Removing a pref - do we need a release note for this?  I don't think it was
> heavily used, so perhaps not, but thought I'd ask.

Probably not, this should be an obscure pref.

Comment 27

[David Anderson [:dvander] - inactive, e-mail if emergency]

Comment on attachment 8653079 [details] [diff] [review]
beta patch

Approval Request Comment
[Feature/regressing bug #]: None
[User impact if declined]: If the user sets this pref, they may get broken text rendering on blocked Graphics drivers.
[Describe test coverage new/current, TreeHerder]: Aurora/Nightly removed logic behind this pref already.
[Risks and why]: This patch simply hardcodes the value of the pref that we were shipping, so there is no risk. Users who did change the pref from its default value may get unsupported/buggy graphics behavior.
[String/UUID change made/needed]: None

Comment 28

[Phil Ringnalda (:philor)]

https://hg.mozilla.org/mozilla-central/rev/e5c83bfca854

Comment 29

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

David, if this bug is only an issue with Dev Edition do we still need to uplift to Beta? Please confirm.

Comment 30

[David Anderson [:dvander] - inactive, e-mail if emergency]

(In reply to Ritu Kothari (:ritu) from comment #29)
> David, if this bug is only an issue with Dev Edition do we still need to
> uplift to Beta? Please confirm.

It's not specific to dev edition, sorry if that wasn't clear. This is a patch specifically for beta.

Comment 31

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

David, Milan, another thing I noticed when comparing the Nightly and Beta patch is that the nightly patch removes the pref where as the Beta patch doesn't use/query the pref value? Does that mean Beta users will still see this as a pref but changing it will be a no-op. Is the Beta patch intentionally different than Nightly?

Comment 32

[David Anderson [:dvander] - inactive, e-mail if emergency]

(In reply to Ritu Kothari (:ritu) from comment #31)
> David, Milan, another thing I noticed when comparing the Nightly and Beta
> patch is that the nightly patch removes the pref where as the Beta patch
> doesn't use/query the pref value? Does that mean Beta users will still see
> this as a pref but changing it will be a no-op. Is the Beta patch
> intentionally different than Nightly?

Yes, it is intentionally different. The nightly patch is cleanup of dead code. The actual pref functionality was neutered in bug 1183910. The beta patch is just a minimal backport of that change.

Comment 33

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

We have not removed the pref for 41 as far as I can tell. David mentioned that a very small percentage of our end-users use this pref setting so leaving the pref in for 41 should have minimal negative impact. 

This pref has been removed in FF42 and onwards.

Comment 34

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8653079 [details] [diff] [review]
beta patch

This pref has been removed in 42 on wards. For Beta41, to minimize the risk we are just removing the code to read this pref. Let's uplift to Beta41.

Comment 35

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/releases/mozilla-beta/rev/13e949eca759

Comment 36

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Setting status42 to unaffected to get it out of my uplift queries, and since this doesn't need to land to 42.