Closed Bug 1815481 Opened 1 year ago Closed 11 months ago

mesa/vmwgfx: Graphical glitch appears on Wayland,Xwayland&X11 in Ubuntu22.04 VMWare client.

Categories

(Core :: Graphics: WebRender, defect, P3)

Product:
Core
Component:
Graphics: WebRender
Platform:
Desktop
Linux
Type:
defect

Tracking

()

Status:
VERIFIED FIXED
Milestone:
116 Branch
Tracking Flags:
Tracking Status
firefox-esr102 --- disabled
firefox110 --- disabled
firefox111 --- wontfix
firefox112 --- wontfix
firefox113 --- wontfix
firefox114 --- wontfix
firefox115 --- verified
firefox116 --- verified

People

(Reporter: alice0775, Assigned: bradwerth)

References

(Blocks 1 open bug, Regression)

Details

(Keywords: nightly-community, regression)

Attachments

(5 files)

about:support
28.38 KB, text/plain
Details
about:support of host pc(Windows10 x64 Home)
28.87 KB, text/plain
Details
about:support of Nightly 2022-01-08 with MOZ_ENABLE_WAYLAND=0
22.47 KB, text/plain
Details
Bug 1815481: Disable hardware WebRender on Mesa drivers in Virtual Machines.
48 bytes, text/x-phabricator-request
dmeehan
: approval-mozilla-beta+
Details | Review
Screenshot of the bad build
258.16 KB, image/png
Details
Attached file about:support —

Steps to reproduce:
On Ubuntu22.04 VMWare guest. (Host OS is Windows10)

  1. about:preferences
  2. Open Fonts dialog
  3. Mouse hover over the dialog

Actual results:
Graphical glitch appears.
Screencast: https://youtu.be/LiRCT_JEywY

Disabled HWA seems fix this issue.

Regression window:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=89aa2c8696b7b10a4e71f95d4a468171b92bb828&tochange=cc33400f0ff80f0eada6c3aa637f37d247a3ff46

Summary: Graphical glitch appears → Graphical glitch appears on Ubuntu22.04 VMWare client.

Presumably regressed by the activation of Wayland in Bug 1749174. It looks like the graphical glitches are a discoloration of an entire tile, but not a blacking out of the tile as if the texture was lost.

Andrew, can you take a look?

Severity: -- → S3
Flags: needinfo?(aosmond)
Priority: -- → P3

Do you know if this happens with any Windows host, or just particular ones? I am wondering if this is an issue with the underlying graphics card / driver that we are unable to detect that we already have a workaround for. Could you supply the about:support for the host of the VM? Thanks!

Flags: needinfo?(aosmond) → needinfo?(alice0775)

Specifically I am wondering about FEATURE_WEBRENDER_SCISSORED_CACHE_CLEARS which is disabled for Intel devices that maybe is spilling over into the VM:
https://searchfox.org/mozilla-central/rev/d85572c1963f72e8bef2787d900e0a8ffd8e6728/widget/windows/GfxInfo.cpp#1850

On Ubuntu 22.04 LTS: Mesa driver version is 22.2.5: Webrender is used by default, the problem is reproduced.
On Ubuntu 20.04 LTS: Mesa driver version is 21.2.6: Webrender(software) is used by default instead of Webrender, the problem is not reproduced.

So, the problem seems to be caused by Mesa driver version 22.2.5.

Flags: needinfo?(alice0775)

Looks like the underlying card shouldn't have scissored cache issues. Does flipping gfx.webrender.scissored-cache-clears.enabled to false and restarting help in the latest nightly? (I just added the pref)

Flags: needinfo?(alice0775)

(In reply to Andrew Osmond [:aosmond] (he/him) from comment #6)

Looks like the underlying card shouldn't have scissored cache issues. Does flipping gfx.webrender.scissored-cache-clears.enabled to false and restarting help in the latest nightly? (I just added the pref)

In Nightly 112.0a1 (build id 20230222094403),
I changed the preference from true to false and restarted the browser.
The problem is still reproduced.

Flags: needinfo?(alice0775)

(In reply to Alice0775 White from comment #7)

(In reply to Andrew Osmond [:aosmond] (he/him) from comment #6)

Looks like the underlying card shouldn't have scissored cache issues. Does flipping gfx.webrender.scissored-cache-clears.enabled to false and restarting help in the latest nightly? (I just added the pref)

In Nightly 112.0a1 (build id 20230222094403),
I changed the preference from true to false and restarted the browser.
The problem is still reproduced.

Thank you for verifying. A vain hope, but it would have been nice to find a smaller existing lever to fix this over just disabling it on VMs.

From comment #0 and Comment#1.
Tentatively mark Bug 1749174 as "Regressed by".

status-firefox110: --- → disabled
status-firefox112: --- → affected
status-firefox-esr102: --- → unaffected
Regressed by: 1749174

:emilio, since you are the author of the regressor, bug 1749174, could you take a look?

For more information, please visit auto_nag documentation.

Flags: needinfo?(emilio)

Disabled wayland fixes this problem.

Open the file /etc/gdm3/custom.conf:

sudo nano /etc/gdm3/custom.conf

Search for the WaylandEnable config. You should set it to false in order to disable Wayland:

WaylandEnable=false

Save the file and exit. On nano, you do that with CTRL+X then y and ENTER to confirm.

And restart Ubuntu

Please reenable Wayland and start Firefox with: $ MOZ_ENABLE_WAYLAND=0 path/to/firefox
Does the bug still occur or does that fix the problem?

(In reply to Darkspirit from comment #12)

Please reenable Wayland and start Firefox with: $ MOZ_ENABLE_WAYLAND=0 path/to/firefox
Does the bug still occur or does that fix the problem?

The problem is still reproduced.

MOZ_ENABLE_WAYLAND is innocent then.

No longer regressed by: 1749174
Summary: Graphical glitch appears on Ubuntu22.04 VMWare client. → Graphical glitch appears on Xwayland and Wayland in Ubuntu22.04 VMWare client.

Has Xwayland/vmwgfx been qualified for HW WR the day before bug 1749174 or did it get SW WR? It might be the case that MOZ_ENABLE_WAYLAND switched HW WR on.
$ MOZ_ENABLE_WAYLAND=0 mozregression --launch 2022-01-18 -a about:support

In case SW WR has been used, try this:
$ MOZ_ENABLE_WAYLAND=0 mozregression --good 2021-01-01 --bad 2022-01-19 --pref gfx.webrender.all:true

Nightly 2022-01-18 with MOZ_ENABLE_WAYLAND=0 get HW WR.
So, I try mozregression with MOZ_ENABLE_WAYLAND=1.

Regression window with MOZ_ENABLE_WAYLAND=1:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=32443a93b4b7ec8a7cc69dc841b22657956fc805&tochange=1a4d9850780709a65b7a090bfbd8eb48f209b3a7

So, it seems that the blocker is Bug 1736267.
3fba3d5d27c9a3c4c1fde593a458fd90cecf0036 Robert Mader — Bug 1736267 - Generally allow HW-WR on Mesa drivers in nightly, r=gfx-reviewers,jrmuizel

Flags: needinfo?(emilio)

Robert, can you please take a look at this?

Flags: needinfo?(robert.mader)
Regressed by: 1736267

This appears to be a driver bug. @alice: it would be great if you could open a upstream issue at https://gitlab.freedesktop.org/mesa/mesa/-/issues to get it fixed.

In the meantime we can add mesa/vmwgfx to the blocklist and enforce SW-WR.

Flags: needinfo?(robert.mader)

(In reply to Robert Mader [:rmader] from comment #19)

In the meantime we can add mesa/vmwgfx to the blocklist and enforce SW-WR.

Bob, is there someone on the gfx team that could maybe assist with this?

status-firefox113: affected → fix-optional
status-firefox114: --- → affected
status-firefox-esr102: unaffected → disabled
Flags: needinfo?(bhood)

Brad is currently poking around in the blocklist system. Let me see if he can do anything here.

Flags: needinfo?(bhood) → needinfo?(bwerth)

Yeah, I'll take care of this.

Flags: needinfo?(bwerth)
Assignee: nobody → bwerth

Brad, are you still working on this? Thanks

status-firefox113: fix-optional → wontfix
status-firefox114: affected → wontfix
status-firefox115: --- → affected
Flags: needinfo?(bwerth)

I've posted https://phabricator.services.mozilla.com/D179864 but it's not linking up in Bugzilla (yet?)

Flags: needinfo?(bwerth)

We classify anything with 'mesa/vmgfx' as a Virtual Machine driver and
block it for Linux.

(In reply to Alice0775 White from comment #17)

So, it seems that the blocker is Bug 1736267.
3fba3d5d27c9a3c4c1fde593a458fd90cecf0036 Robert Mader — Bug 1736267 - Generally allow HW-WR on Mesa drivers in nightly, r=gfx-reviewers,jrmuizel

Can you run mozregression with WebRender force-enabled to check if a specific WebRender change has introduced the bug?
$ MOZ_DISABLE_CONTENT_SANDBOX=1 mozregression --good 2020-01-01 --bad 2023-01-01 --pref gfx.webrender.all:true

I have encountered a similar bug months ago and today (bug 1836548), but can't reproduce it.

(In reply to Darkspirit from comment #26)

(In reply to Alice0775 White from comment #17)

So, it seems that the blocker is Bug 1736267.
3fba3d5d27c9a3c4c1fde593a458fd90cecf0036 Robert Mader — Bug 1736267 - Generally allow HW-WR on Mesa drivers in nightly, r=gfx-reviewers,jrmuizel

Can you run mozregression with WebRender force-enabled to check if a specific WebRender change has introduced the bug?
$ MOZ_DISABLE_CONTENT_SANDBOX=1 mozregression --good 2020-01-01 --bad 2023-01-01 --pref gfx.webrender.all:true

Regression window:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=1600e73bdd901c5ac4b831c43e29f73d561aa816&tochange=08c29f9d87799463cdf99ab81f08f62339b49328

(In reply to Alice0775 White from comment #27)

(In reply to Darkspirit from comment #26)

Can you run mozregression with WebRender force-enabled to check if a specific WebRender change has introduced the bug?
$ MOZ_DISABLE_CONTENT_SANDBOX=1 mozregression --good 2020-01-01 --bad 2023-01-01 --pref gfx.webrender.all:true

Regression window:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=1600e73bdd901c5ac4b831c43e29f73d561aa816&tochange=08c29f9d87799463cdf99ab81f08f62339b49328

gfx-related ones:

2c899cf4c756451146928db38db5f20c7a4e900a Nicolas Silva — Bug 1654279 - Avoid some allocations in ClipChainStack::push_surface. r=gw
5df699bb72b1c9f9a18094e8c2e1df42bfd7e39b Andrew Osmond — Bug 1653443 - Disable the GPU process on Linux even with hardware compositing. r=nical
45e8dd195f714ebf8f858a9d6a17d80f3b306e13 Robert Mader — Bug 1654687 - Use gtk_window_set_titlebar directly, r=stransky
aa298173d7db9e9c0fb4b7c9b57cd54eb6acde0f Nicolas Silva — Bug 1652743 - Invalidate window origin recursively.
4b66ae956966753221bee1c81cbb141cdbaead47 Dzmitry Malyshau — Bug 1640960 - Remove aPosition from all shaders, except debug ones r=gw,jrmuizel
646a1a717d2b906196ae35155d98412b7d0a26d6 Kartikaya Gupta — Bug 1654353 - Nudge the transform matrix to integers before checking for integer axis alignment. r=mattwoodrow
9c57e70c264dce03f1cb9129fc1f9504b098ff2b Emilio Cobos Álvarez — Bug 1650719 - Don't lose the rect offset from the composition bounds. r=kats
72f4a4d07192ef6dc64f8d3a8fdfe6124db14ae7 Glenn Watson — Bug 1654442 - Add primitives to picture cache slices earlier during scene building r=nical
7abdf136d36588da86c175f13831d79f8d1df01e Dzmitry Malyshau — Bug 1642495 - Switch all WebRender HW-accelerated GPU cache updates to Scatter r=gw

bug 1652743 sounds related to the menu glitch in comment 28.

Does today's glitch from comment 0 mostly look the same each time (and not like comment 28)? Can you run mozregression with force-enabled WebRender to find out since when the glitch started looking like today?

https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=89aa2c8696b7b10a4e71f95d4a468171b92bb828&tochange=cc33400f0ff80f0eada6c3aa637f37d247a3ff46

I can reproduce comment#0 exact same glitch at least from cc33400f0ff80f0eada6c3aa637f37d247a3ff46.

The good build 89a2c8696b7b10a4e71f95d4a468171b92bb828 seems to have occasional screen problems. But I think it is different from this one.

Did the Xwayland glitch from comment 13 look like
a) the one from comment 0
b) the one from comment 28
c) differently
?

Is this still reproducible on Xwayland? $ MOZ_ENABLE_WAYLAND=0 mozregression --launch 2023-06-02 --pref gfx.webrender.all:true

Does Xwayland have the same regression range?
Is this last good? $ MOZ_ENABLE_WAYLAND=0 mozregression --launch 20220118215506 --pref gfx.webrender.all:true
And this first bad? $ MOZ_ENABLE_WAYLAND=0 mozregression --launch 20220119214718 --pref gfx.webrender.all:true

Can Wayland be fixed with these prefs?
$ MOZ_ENABLE_WAYLAND=1 mozregression --launch 2023-06-02 --pref gfx.webrender.all:true gfx.webrender.allow-partial-present-buffer-age:false gfx.webrender.max-partial-present-rects:0 gfx.webrender.pbo-uploads:false gfx.webrender.use-optimized-shaders:false
or
$ MOZ_ENABLE_WAYLAND=1 mozregression --launch 2023-06-02 --pref gfx.webrender.all:true gfx.webrender.allow-partial-present-buffer-age:false gfx.webrender.max-partial-present-rects:0 gfx.webrender.pbo-uploads:false gfx.webrender.use-optimized-shaders:false browser.tabs.inTitlebar:0

Can Xwayland be fixed with these prefs?
$ MOZ_ENABLE_WAYLAND=0 mozregression --launch 2023-06-02 --pref gfx.webrender.all:true gfx.webrender.allow-partial-present-buffer-age:false gfx.webrender.max-partial-present-rects:0 gfx.webrender.pbo-uploads:false gfx.webrender.use-optimized-shaders:false
or
$ MOZ_ENABLE_WAYLAND=0 mozregression --launch 2023-06-02 --pref gfx.webrender.all:true gfx.webrender.allow-partial-present-buffer-age:false gfx.webrender.max-partial-present-rects:0 gfx.webrender.pbo-uploads:false gfx.webrender.use-optimized-shaders:false browser.tabs.inTitlebar:0

See Also: → 1836548

(In reply to Darkspirit from comment #31)

Did the Xwayland gltich from comment 13 look like
a) the one from comment 0
b) the one from comment 28
c) differently
?

It is a) the one from comment 0.

Is this still reproducible on Xwayland? $ MOZ_ENABLE_WAYLAND=0 mozregression --launch 2023-06-02 --pref gfx.webrender.all:true

Yes, it is also the same glitch of comment#0.

(In reply to Darkspirit from comment #31)

Does Xwayland have the same regression range?
Is this last good? $ MOZ_ENABLE_WAYLAND=0 mozregression --launch 20220118215506 --pref gfx.webrender.all:true

Seems Good.

And this first bad? $ MOZ_ENABLE_WAYLAND=0 mozregression --launch 20220119214718 --pref gfx.webrender.all:true

Bad, it is also the same glitch of comment#0.

(In reply to Darkspirit from comment #31)

the same glitch of c) differently: see screencast https://youtu.be/oJk_oCr3jLk

Can Wayland be fixed with these prefs?
$ MOZ_ENABLE_WAYLAND=1 mozregression --launch 2023-06-02 --pref gfx.webrender.all:true gfx.webrender.allow-partial-present-buffer-age:false gfx.webrender.max-partial-present-rects:0 gfx.webrender.pbo-uploads:false gfx.webrender.use-optimized-shaders:false

No. it seems to be the same glitch of c) differently.

or
$ MOZ_ENABLE_WAYLAND=1 mozregression --launch 2023-06-02 --pref gfx.webrender.all:true gfx.webrender.allow-partial-present-buffer-age:false gfx.webrender.max-partial-present-rects:0 gfx.webrender.pbo-uploads:false gfx.webrender.use-optimized-shaders:false browser.tabs.inTitlebar:0

No. it is also the same glitch of comment#0.

Can Xwayland be fixed with these prefs?
$ MOZ_ENABLE_WAYLAND=0 mozregression --launch 2023-06-02 --pref gfx.webrender.all:true gfx.webrender.allow-partial-present-buffer-age:false gfx.webrender.max-partial-present-rects:0 gfx.webrender.pbo-uploads:false gfx.webrender.use-optimized-shaders:false

No. it seems to be the same glitch of c) differently.

or
$ MOZ_ENABLE_WAYLAND=0 mozregression --launch 2023-06-02 --pref gfx.webrender.all:true gfx.webrender.allow-partial-present-buffer-age:false gfx.webrender.max-partial-present-rects:0 gfx.webrender.pbo-uploads:false gfx.webrender.use-optimized-shaders:false browser.tabs.inTitlebar:0

No. it seems to be the same glitch of c) differently.

(In reply to Alice0775 White from comment #33)

(In reply to Darkspirit from comment #31)

Does Xwayland have the same regression range?
Is this last good? $ MOZ_ENABLE_WAYLAND=0 mozregression --launch 20220118215506 --pref gfx.webrender.all:true

Seems Good.

And this first bad? $ MOZ_ENABLE_WAYLAND=0 mozregression --launch 20220119214718 --pref gfx.webrender.all:true

Bad, it is also the same glitch of comment#0.

Does "window protocol" on about:support really contain "xwayland" when trying first bad? Just to be 100% sure.
Is last good really last good?

comment 17 (wayland: bug 1736267) and comment 30 (xwayland) have different results, please try to search with Wayland and force-enabled WR:
$ MOZ_ENABLE_WAYLAND=1 MOZ_DISABLE_CONTENT_SANDBOX=1 mozregression --good 2021-01-01 --bad 2023-01-01 --pref gfx.webrender.all:true

(In reply to Darkspirit from comment #35)

(In reply to Alice0775 White from comment #33)

(In reply to Darkspirit from comment #31)

Does Xwayland have the same regression range?
Is this last good? $ MOZ_ENABLE_WAYLAND=0 mozregression --launch 20220118215506 --pref gfx.webrender.all:true

Seems Good.

And this first bad? $ MOZ_ENABLE_WAYLAND=0 mozregression --launch 20220119214718 --pref gfx.webrender.all:true

Bad, it is also the same glitch of comment#0.

Does "window protocol" on about:support really contain "xwayland" when trying first bad? Just to be 100% sure.

Yes, Window Protocol is "xwayland".

Is last good really last good?

As far as we have tried, yes.
However, I am not 100% sure about this as the result depends on the scroll position of the page, mouse trace and timing.

comment 17 (wayland: bug 1736267) and comment 30 (xwayland) have different results, please try to search with Wayland and force-enabled WR:
$ MOZ_ENABLE_WAYLAND=1 MOZ_DISABLE_CONTENT_SANDBOX=1 mozregression --good 2021-01-01 --bad 2023-01-01 --pref gfx.webrender.all:true

I could not. The build (20210101210516) is also experiencing graphics issues.
See https://youtu.be/KqA72NMb3aI

Duplicate of this bug: 1837128
Blocks: wr-linux
Summary: Graphical glitch appears on Xwayland and Wayland in Ubuntu22.04 VMWare client. → mesa/vmwgfx: Graphical glitch appears on Xwayland and Wayland in Ubuntu22.04 VMWare client.

(In reply to Brad Werth [:bradwerth] from comment #25)

Created attachment 9337304 [details]
Bug 1815481: Disable hardware WebRender on Mesa drivers in Virtual Machines.

We classify anything with 'mesa/vmgfx' as a Virtual Machine driver and
block it for Linux.

Please push. This bug needs to be reported to Mesa. Once it has been fixed, it should be unblocked:
https://www.phoronix.com/search/vmwgfx

vmwgfx is the VMware Linux graphics driver for exposing 3D/OpenGL/3D hardware acceleration to guest virtual machines of VMware's products.

See Also: → 1837167
Duplicate of this bug: 1837167
See Also: 1837167 →
Summary: mesa/vmwgfx: Graphical glitch appears on Xwayland and Wayland in Ubuntu22.04 VMWare client. → mesa/vmwgfx: Graphical glitch appears on Wayland,Xwayland&X11 in Ubuntu22.04 VMWare client.
Duplicate of this bug: 1768690
Duplicate of this bug: 1765218
See Also: → 1830769
Duplicate of this bug: 1830264

:bradwerth any timeline on when this will land? Wondering if we can think of uplifting to beta for 115?

status-firefox116: --- → affected
Flags: needinfo?(bwerth)

(In reply to Donal Meehan [:dmeehan] from comment #44)

:bradwerth any timeline on when this will land? Wondering if we can think of uplifting to beta for 115?

Let's give stransky until tomorrow to review the updated patch (which was changed at stransky's request). If still nothing tomorrow, then I'll remove stransky as reviewer and land it.

Flags: needinfo?(bwerth) → needinfo?(stransky)
Pushed by bwerth@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1ece9296780d
Disable hardware WebRender on Mesa drivers in Virtual Machines. r=rmader

https://hg.mozilla.org/mozilla-central/rev/1ece9296780d

Status: NEW → RESOLVED
Closed: 11 months ago
status-firefox116: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → 116 Branch

Thanks for handling it.

Flags: needinfo?(stransky)

The patch landed in nightly and beta is affected.
:bradwerth, is this bug important enough to require an uplift?

  • If yes, please nominate the patch for beta approval.
  • If no, please set status-firefox115 to wontfix.

For more information, please visit BugBot documentation.

Flags: needinfo?(bwerth)

Comment on attachment 9337304 [details]
Bug 1815481: Disable hardware WebRender on Mesa drivers in Virtual Machines.

Beta/Release Uplift Approval Request

  • User impact if declined: User may see visual anomalies when running Firefox on a Linux Virtual Machine in VMWare. Unknown if the host OS matters.
  • Is this code covered by automated tests?: No
  • Has the fix been verified in Nightly?: No
  • Needs manual test from QE?: Yes
  • If yes, steps to reproduce: STR in the original bug comment. Require a Windows host running VMWare with an Ubuntu22.04 VM.
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): The patch is not well tested, but the patch just disables hardware WebRender and the software path is well tested on Linux.
  • String changes made/needed:
  • Is Android affected?: No
Flags: needinfo?(bwerth)
Attachment #9337304 - Flags: approval-mozilla-beta?

A bit late and just for the record: I think we'll want to delete the mIsAccelerated = false; again in the future as we actually want to use accelerated rendering with this driver - this is just a driver bug, hopefully/likely fixed in some future mesa version. So we'll want to rely on the blocklist with a certain version in the future.

Comment on attachment 9337304 [details]
Bug 1815481: Disable hardware WebRender on Mesa drivers in Virtual Machines.

Approved for 115.0b9.

Attachment #9337304 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
QA Whiteboard: [qa-triaged]
Flags: qe-verify+

I could not reproduce the issue on Ubuntu 20.04 physical machine and neither on VM Ubuntu 22.04 (on Win10) using FF build 111.0a1(20230207043534).
Reporter, can you please confirm the fix on Nightly build 116.0a1(https://archive.mozilla.org/pub/firefox/nightly/2023/06/2023-06-23-09-25-29-mozilla-central/) and on Beta build 115.0b9(https://archive.mozilla.org/pub/firefox/candidates/115.0b9-candidates/build1/linux-x86_64/en-US/). Thank you.

Flags: needinfo?(alice0775)

I can reproduce the issue on Firefox115.0beta8 and Nightly116.0a1(20230616214102) Ubuntu22.04 and Linux Mint 21 cinnamon.
And, I verified fix this on Firefox115.0beta9 and Nightly116.0a1(20230623092529) Ubuntu22.04 and Linux Mint 21 cinnamon.

Status: RESOLVED → VERIFIED
status-firefox115: fixed → verified
status-firefox116: fixed → verified
Flags: needinfo?(alice0775)
QA Whiteboard: [qa-triaged]
Flags: qe-verify+
See Also: → 1830081
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: