1939218 - Graphics corruption while visiting Grafana

Status: UNCONFIRMED

(Core :: Graphics, defect)

Description

[kg4ydw]

I have a very busy custom grafana dashboard (near full screen, 8 visible panels, another 12 folded) that I have constantly open and updating over long periods of time without being reloaded.

After my linux system has been under high memory pressure conditions, the panels become slowly corrupted, occasionally showing parts of other windows and pages in the same browser, or even other panels in the same page, and occasionally patterns that look like raw memory from other uses copied into the graphics pane.

This smells like a use after free bug, possibly with the memory the panels are using being forcibly freed and reused, but with both old and new code now using the same memory. Occasionally I get crashes from this, but usually I can hit the firefox refresh button and it goes away. Note: the grafana refresh button does NOT make it go away, and sometimes makes it worse, with panels going blank or refusing to update or an update from one panel copied to another panel in the same page.

I'm not sure how reproducible this is, but at least I can probably reproduce the tight memory conditions
I'm not interested in a bounty -- having the bug fixed would be reward enough!

I have machines that have less memory and do not exhibit this problem, but they are not running teams and outlook in the same browser either, so the memory may actually be tighter on the machine with more memory. Other web applications may also be misbehaving too but grafana is more visible and obvious.

Let me know if you would like me to try instrumenting this in some way or change versions or something.
I have seen this problem on and off for a long time, this is not a new bug.

Ubuntu 22.04.5 LTS
Firefox browser 133.0.3
Mozilla Firefox Snap for Ubuntu

Comment 1

[Tom Ritter [:tjr]]

It is the holidays, so others may have better ideas when they are back, but you could try using ASAN Nightly to reproduce the crash and submit it to us: https://firefox-source-docs.mozilla.org/tools/sanitizer/asan_nightly.html

See the note about setting your email so we can associate the crash to your account.

Comment 2

[Daniel Veditz [:dveditz]]

Need more information to proceed

Comment 3

[kg4ydw]

Not sure how I can generate a test case.
Is there some data I can collect somewhere that would help?

Comment 4

[Daniel Veditz [:dveditz]]

See comment 1 about ASAN builds. They can make many memory corruption bugs easier to reproduce, and when a crash does happen it captures more context about how the memory was allocated and freed.

If you submitted a crash report to us when it did happen then it should show up in the about:crashes page. If you follow the View button to our crash site (submit the crash if it's in the unsubmitted list) then paste the incident link into this bug.

Given those symptoms and the fact that it's only one of your machines (albeit one under heavy load), I'm a bit suspicious about bad memory. It is definitely worth ruling that out! https://help.ubuntu.com/community/MemoryTest

Obviously reliable "steps to reproduce" or a testcase would be best, but crash logs can sometimes contain enough hints. But we can't do anything with nothing.

Comment 5

[kg4ydw]

Ok, will try ASAN build as soon as the linux download link stops saying ResourceNotFound

Understand you can't debug with nothing.
Just like I'd love to take apart the code that grafana dumps on my browser and make 100 copies of it and run it once a second until the browser crashes, and I'll do that as soon as I have a week of spare time. :-P

Comment 6

[Daniel Veditz [:dveditz]]

what did the memory test say about the affected machine?

Comment 7

[kg4ydw]

I've been unable to try the ASAN build as the linux link doesn't work.

Comment 8

[Daniel Veditz [:dveditz]]

What were the results of the Ubuntu memory test on the specific machine that was having the problem (see link in comment 4)? That's BIOS functionality and does not require any Firefox build to do.

Comment 9

[kg4ydw]

I have to schedule downtime for the memory test.

However, it seems unlikely that there is a memory error, as there is a lot of activity on this system and as far as I can tell, nothing else is affected by any memory errors or corrupted data in memory.

And the issues I am seeing in the browser generally don't look like corrupted memory either -- it looks like two or more panes using the same memory, and it is frequently not even misaligned, like two panes of the same size sharing the exact same memory buffer.

Comment 10

[Daniel Veditz [:dveditz]]

nothing else is affected by any memory errors or corrupted data in memory.

So even when Firefox is showing this problem, if you switch to another app (while system memory usage is high) you don't see problems there? On a linux system this could be a problem in X11 (or, especially, Wayland if you use that instead).

Comment 11

[kg4ydw]

I'm using X11. When the panes in firefox/grafana get corrupted in what looks like overlapping memory use, I only see other data from firefox. Occasionally I see snips of other tabs in the corrupted panel, but usually it's another panel on the same page duplicated.

Hitting the grafana refresh button sometimes swaps which of the two overlapping panels are duplicated. Hitting the firefox refresh button (rebuilding the whole page) typically fixes it for a while. I've only had two full crashes in maybe a year but I see duplicated panels once or twice a week.

I'm not having crashes from any other app (ignoring actual OOM crashes).

I suspect but can't prove that one of the web apps I'm keeping open occasionally gets a run away memory allocation problem that I'm sure exacerbates this issue.