1673939 - garbled text display towards end of long page

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, defect, P3)

Description

[Igor Kovalenko]

Attachment: Long page text issue - incomplete text lines at the end of long page

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:82.0) Gecko/20100101 Firefox/82.0

Steps to reproduce:

I'm running into this issue on kde + wayland but also reproduced it in plain X11 session without display manager using latest Nightly build + webrender enabled.

1. Enable webrender via gfx.webrender.all:true

2. Open long text page https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.8.15 and go to end of page, try reading text.

Another example https://gitlab.freedesktop.org/pulseaudio/pulseaudio/-/merge_requests/227 wait for page to load, try reading last comments at the end of page.

Actual results:

Some lines of text are not completely drawn, others are invisible, and some are displayed correctly.
Scrolling up/down and then selecting partially invisible text does change which lines are garbled.
At the top of page all text is visible as expected; problems occur at some place down .

Expected results:

All text on page should be visible.

Comment 1

[Igor Kovalenko]

Attachment: Nightly build support data (raw) firefox-nightly-20201028-1.json

Comment 2

[Igor Kovalenko]

Using mozregression to find first occurrence ~exactly matching screenshot

MOZ_DISABLE_CONTENT_SANDBOX=1 MACH_USE_SYSTEM_PYTHON=1 ./mach mozregression --persist /sources/firefox/mozregression-cache --good 68 --bad 72 --pref gfx.webrender.all:true -a https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.8.15
...
88:22.81 INFO: Narrowed integration regression window from [2d52a5a0, c0f887d8] (3 builds) to [2d52a5a0, b5cfc7ed] (2 builds) (~1 steps left)
88:22.81 INFO: No more integration revisions, bisection finished.
88:22.81 INFO: Last good revision: 2d52a5a00e007ff7f78a053a446e246828cf3d60
88:22.81 INFO: First bad revision: b5cfc7ed22b48babd34b6f1cbd2b1b7d93648a07
88:22.81 INFO: Pushlog: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=2d52a5a00e007ff7f78a053a446e246828cf3d60&tochange=b5cfc7ed22b48babd34b6f1cbd2b1b7d93648a07

Comment 3

[Igor Kovalenko]

Attachment: Screenshot of older problem where text is garbled but appears to be visible

Comment 4

[Igor Kovalenko]

Digging deeper revealed that there was yet older issue bisected like this, for which Screenshot of older problem where text is garbled but appears to be visible is attached. Versions even older than below were displaying text cleanly:

MOZ_DISABLE_CONTENT_SANDBOX=1 MACH_USE_SYSTEM_PYTHON=1 ./mach mozregression --persist /sources/firefox/mozregression-cache --good 6 --pref gfx.webrender.all:true -a https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.8.15
...
15:13.76 INFO: Narrowed nightly regression window from [2019-04-16, 2019-04-18] (2 days) to [2019-04-17, 2019-04-18] (1 days) (~0 steps left)
15:13.76 INFO: Got as far as we can go bisecting nightlies...
15:13.76 INFO: Last good revision: 02b89c29412b6c1444fe32a4847e5261e2bb3d00 (2019-04-17)
15:13.76 INFO: First bad revision: 2ccc6648064315964dd23039ad28ebf7d9f82999 (2019-04-18)
15:13.76 INFO: Pushlog: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=02b89c29412b6c1444fe32a4847e5261e2bb3d00&tochange=2ccc6648064315964dd23039ad28ebf7d9f82999

Comment 5

[Igor Kovalenko]

Attachment: Nightly about:support data with hardware renderer old Radeon card r600 driver

Testing again, issue is reproduced on Firefox Nightly 20201029095639 clean profile + gfx.webrender.all=true, about:support attached.

Comment 6

[Igor Kovalenko]

Attachment: Nightly about:support data with software renderer llvmpipe

Testing again, issue is NOT reproduced on Firefox Nightly 20201029095639 clean profile + gfx.webrender.all=true and software rendering llvmpipe, about:support attached.

Running with software rendering llvmpipe like this:

export MOZ_LAYERS_ALLOW_SOFTWARE_GL=1
export LIBGL_ALWAYS_SOFTWARE=true
./firefox -P Nightly

Comment 7

[Igor Kovalenko]

Looks like the issue is isolated somewhere between webrender and mesa r600 driver running on Radeon HD 4290

Comment 8

[Igor Kovalenko]

Just verified that Google Chrome in the same environment does not have this rendering issue.

Comment 9

[Igor Kovalenko]

Mesa bug report https://gitlab.freedesktop.org/mesa/mesa/-/issues/3720

Comment 10

[Igor Kovalenko]

Resumed bisecting firefox using source build, confirming earlier suggestion by @emilio

The first bad revision is:
changeset:   469949:c513275730f6
user:        Glenn Watson <[e-mail]>
date:        Wed Apr 17 21:06:19 2019 +0000
summary:     Bug 1545006 - Use external scroll offsets to move primitives and clips into local space. r=kats

mach run for changeset 469949:c513275730f6 using test file generated with (for i in $(seq 1 1000000); do echo "0" ${i}; done) > testcase-narrow.txt
with r600 hardware it starts to show visible artifacts near line 587 and end of page is already not visible at all
with llvmpipe software it is much better but in the end of page text lines are misplaced, attached screenshot

Comment 11

[Igor Kovalenko]

Attachment: Screenshot for mach run of changeset 469949:c513275730f6 with llvmpipe software rendeding.

Attached screenshot for mach run of changeset 469949:c513275730f6 with llvmpipe software rendeding and gfx.webrender.all:true

GDK_BACKEND=x11 MOZ_DISABLE_CONTENT_SANDBOX=1 MOZ_LAYERS_ALLOW_SOFTWARE_GL=1 LIBGL_ALWAYS_SOFTWARE=false ./mach run -P dev

Comment 12

[Igor Kovalenko]

(In reply to Igor Kovalenko from comment #9)

Mesa bug report https://gitlab.freedesktop.org/mesa/mesa/-/issues/3720

Attached apitrace running Firefox Nightly 84.0a1 20201030034830 firefox-long-page.trace.xz to mesa bug tracker item.

Comment 13

[Igor Kovalenko]

To double-check this finding I did hg update --rev 469948:a3acb30248ff and then I see no issues with text display with gfx.webrender.all:true for all test cases above, both with hardware r600 and software llvmpipe rendering.

Comment 14

[Jeff Muizelaar [:jrmuizel]]

Andrew, can we block this hardware for the Linux WebRender release?

Comment 15

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1673939 - Block WebRender for AMD R600 GPUs with Mesa drivers due to garbled text.

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0cf49d959929
Block WebRender for AMD R600 GPUs with Mesa drivers due to garbled text. r=jrmuizel

Comment 17

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/0cf49d959929

Comment 18

[Nicolas Silva [:nical]]

Attachment: Bug 1673939 - Remove a few more texture array-related symbols and fix some comments.

Depends on D106790

Comment 19

[Nicolas Silva [:nical]]

Attachment: Bug 1673939 - Remove Device::max_texture_layers.

Depends on D106791

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/848d05610446
Remove a few more texture array-related symbols and fix some comments. r=gfx-reviewers,jrmuizel
https://hg.mozilla.org/integration/autoland/rev/dcda2e73b83b
Remove Device::max_texture_layers. r=gfx-reviewers,jrmuizel

Comment 21

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/848d05610446
https://hg.mozilla.org/mozilla-central/rev/dcda2e73b83b

Comment 22

[Andrew Osmond [:aosmond] (he/him)]

If you force WebRender on, do you still see the issue in the latest nightly? It is possible bug 1694909 fixed in this 88, and we can lift the blocklist restriction we originally added in this bug. Thanks!

Comment 23

[Kestrel]

I still see this with AMD RS880 on latest Nightly 88.0a1 (2021-03-16).

Comment 24

[Igor Kovalenko]

I confirm the issue is still present with Nightly Firefox Version 88.0a1 Build ID 20210317212527
Compositing WebRender on AMD RS880 (DRM 2.50.0 / 5.11.7, LLVM 11.1.0) with mesa/r600 21.0.0.0

Comment 25

[Kestrel]

Can be avoided by setting layers.async-pan-zoom.enabled = false with Fission disabled (fission.autostart = false) (thanks to Bug 1726431 Comment 7).

Regressed by Bug 1545006 as per Comment 10.

Comment 26

[Marco Castelluccio [:marco]]

Set release status flags based on info from the regressing bug 1545006

Comment 27

[Marco Castelluccio [:marco]]

Sorry for the noise, I was debugging the bot and wrongly run it.

Comment 28

[Robert Mader [:rmader]]

(In reply to Igor Kovalenko from comment #24)

I confirm the issue is still present with Nightly Firefox Version 88.0a1 Build ID 20210317212527
Compositing WebRender on AMD RS880 (DRM 2.50.0 / 5.11.7, LLVM 11.1.0) with mesa/r600 21.0.0.0

Hi Igor, can I ask you to test this again?

Comment 29

[Igor Kovalenko]

(In reply to Robert Mader [:rmader] from comment #28)

(In reply to Igor Kovalenko from comment #24)

I confirm the issue is still present with Nightly Firefox Version 88.0a1 Build ID 20210317212527
Compositing WebRender on AMD RS880 (DRM 2.50.0 / 5.11.7, LLVM 11.1.0) with mesa/r600 21.0.0.0

Hi Igor, can I ask you to test this again?
Will take some time as my box with that hardware is currently collecting dust. Will get back when I test it.

Comment 30

[Kestrel]

I can still reproduce this with latest Nightly 97.0a1 (2021-12-14) on Ubuntu 21.10.

Comment 31

[Darkspirit]

(In reply to Kestrel from comment #30)

I can still reproduce this with latest Nightly 97.0a1 (2021-12-14) on Ubuntu 21.10.

There is bug 1726431.

Comment 34

[Kestrel]

Workaround is to set the environment variable R600_DEBUG=notiling and change gfx.webrender.pbo-uploads = false. This avoids the issue with no apparent performance sacrifice.

Comment 35

[Robert Mader [:rmader]]

Clearing needinfo