Closed Bug 1660223 (CVE-2024-0746) Opened 5 years ago Closed 1 year ago

Crash in [@ gtk_printer_get_name]

Categories

(Core :: Printing: Output, defect, P3)

Product:
Core
Component:
Printing: Output
Version:
Firefox 81
Platform:
x86_64
Linux
Type:
defect

Tracking

()

Status:
VERIFIED FIXED
Milestone:
122 Branch
Tracking Flags:
Tracking Status
firefox-esr115 122+ verified
firefox81 --- wontfix
firefox120 --- wontfix
firefox121 --- wontfix
firefox122 + verified

People

(Reporter: noni, Assigned: dholbert)

References

(Blocks 1 open bug)

Details

(Keywords: crash, csectype-uaf, sec-moderate, Whiteboard: [print2020][old-ui?][tbird crash][adv-main122+][adv-esr115.7+])

Crash Data

Attachments

(4 files)

Bug 1660223: Let printer enumeration run to completion, to avoid a GTK bug. r?jwatt,AlaskanEmily
48 bytes, text/x-phabricator-request
Details | Review
minimal gtk program (c++ testcase) that triggers the same UAF bug for me
2.85 KB, text/plain
Details
Bug 1660223: Let printer enumeration run to completion, to avoid a GTK bug. r?jwatt,AlaskanEmily
48 bytes, text/x-phabricator-request
phab-bot
: approval-mozilla-esr115+
Details | Review
advisory.txt
137 bytes, text/plain
Details

This bug is for crash report bp-4cb207df-64d5-4ec6-b5be-db1ec0200820.

Top 10 frames of crashing thread:

0 libgtk-3.so.0 gtk_printer_get_name ./debian/build/deb/gtk/../../../../gtk/gtkprinter.c:436
1 libprintbackend-cups.so <name omitted> ../../../../../../modules/printbackends/cups/gtkprintbackendcups.c:1874
2 libglib-2.0.so.0 <name omitted> ../../../glib/glist.c:922
3 libprintbackend-cups.so cups_request_printer_list_cb ../../../../../../modules/printbackends/cups/gtkprintbackendcups.c:3630
4 libprintbackend-cups.so cups_dispatch_watch_dispatch ../../../../../../modules/printbackends/cups/gtkprintbackendcups.c:1581
5 libglib-2.0.so.0 g_main_context_dispatch ../../../glib/gmain.c:3974
6 libglib-2.0.so.0 g_main_context_iterate ../../../glib/gmain.c:4047
7 libglib-2.0.so.0 <name omitted> ../../../glib/gmain.c:4241
8 libgtk-3.so.0 gtk_enumerate_printers ./debian/build/deb/gtk/../../../../gtk/gtkprinter.c:1285
9 libxul.so mozilla::detail::RunnableMethodImpl<nsDeviceContextSpecGTK*, void  xpcom/threads/nsThreadUtils.h:1240

Affected versions

  • 81.0a1 (BuildId:20200819212829)

Affected platforms

  • Ubuntu 20.04 64bit.

Additional notes

  • The drivers for the printers were installed with hplip 3.20.6 (latest available at this time).
  • This could have been an isolated case, I'll look into it in order to get some more specific steps.

Steps to reproduce

  1. Fresh install of the printer (HP MFP M28A in my case)
  2. Open Firefox.
  3. Navigate to any page and print it.

Expected result

  • Page is correctly printed.

Actual result

  • Browser crashed and page was not printed.

Regression Window

  • Not sure if this is a regression or not since I've only ran into it once so far.

[Suggested Severity]

  • S3
Component: Printing → Printing: Output
Product: Toolkit → Core

Something to keep an eye on, but the number of these crashes reported recently is currently no more than we've had going back months.

Severity: -- → S3
Priority: -- → P2
Has Regression Range: --- → no
Has STR: --- → yes

(In reply to Jonathan Watt [:jwatt] from comment #2)

Something to keep an eye on, but the number of these crashes reported recently is currently no more than we've had going back months.

Dropping this to P3 for now, since there doesn't seem to have been any increase in the frequency of this crash since way before the new UI started being developed.

Priority: P2 → P3
Whiteboard: [print2020_v81][old-ui?] → [print2020][old-ui?]

The crashes under this signature all have the poison pattern in one of the registers. This is an use-after-free crash and thus potentially security sensitive.

Group: core-security
Keywords: csectype-uaf

Seems to pretty consistently be rdx which means you should be able to look at the assembly and figure out which object is the freed one.

Calling this sec-moderate on the assumption that we only call this function when the user has explicitly tried to print something, and that it's not something a web site could trigger on its own.

Group: core-security → layout-core-security
Keywords: sec-moderate

Currently all crashes seem to be null-dereference crashes.

Looking at the gtk_printer_get_name source, it seems like printer cannot be null, and therefore printer->priv must be null if we crash in gtk_printer_get_name.

(In reply to Gabriele Svelto [:gsvelto] from comment #4)

The crashes under this signature all have the poison pattern in one of the registers. This is an use-after-free crash and thus potentially security sensitive.

rax contains 0xe5e5e5e5e5e5e5e5. But does that really mean that there is a security issue here? If the poison pattern was in memory, that would be an expected part of normal execution...unless that memory was being read from. I'm possibly showing my ignorance here, but could the pattern be in rax because it was put there in order to copy that value from rax into some freed memory to poison it? Or is there good reason to believe that it being there means that we read that value from freed memory.

Flags: needinfo?(gsvelto)

The addresses reported in Socorro are misleading because the real crashing address - which is 0xe5e5e5e5e5e5e5e5 - causes a general protection fault on x86-64 processors and gets reported as NULL. We're this close to showing the real address in these crashes (bug 1493342). We just need to make a new release of the stack walker and ship it into Socorro, then all these crashes will show up as hitting 0xe5e5e5e5e5e5e5e5.

Flags: needinfo?(gsvelto)
Whiteboard: [print2020][old-ui?] → [print2020][old-ui?][tbird crash]

(In reply to Gabriele Svelto [:gsvelto] (PTO) from comment #7)

The addresses reported in Socorro are misleading because the real crashing address - which is 0xe5e5e5e5e5e5e5e5 - causes a general protection fault on x86-64 processors and gets reported as NULL. We're this close to showing the real address in these crashes (bug 1493342). We just need to make a new release of the stack walker and ship it into Socorro, then all these crashes will show up as hitting 0xe5e5e5e5e5e5e5e5.

As expected, these crashes are now showing up as being on poison values.

Is there a workaround for users to try?

User in bp-515af179-d3a0-420a-aaf1-99aa00230419 cites crashes for both Firefox and Thunderbird when using papercut, which is pretty popular.

Flags: needinfo?(jwatt)

(In reply to Wayne Mery (:wsmwk) from comment #10)

Is there a workaround for users to try?

I don't think so.

User in bp-515af179-d3a0-420a-aaf1-99aa00230419 cites crashes for both Firefox and Thunderbird when using papercut, which is pretty popular.

Yeah, the comment there is interesting (on their system this seems to be specific to that print target, printed to from Firefox only).

However, I'm not sure how far that gets us; the papercut documentation says they don't offer a print driver for Mac or Linux; they just recommend using the platform postscript driver. So that means we aren't e.g. hitting some bug or edge case in a papercut print driver. (Also: it looks like that's the only crash report that mentions papercut.)

Side note, I wondered if maybe bug 1826872 might help here (we're anticipating that that'll help with some other print crashes). However, I don't think it does help here -- bug 1826872's patch landed on the 114 Nightly cycle, and this bug here has two crashes from 114 betas:
bp-0fb8e0c5-74ab-41c9-9a5d-3ac600230520
bp-624919ac-f0b2-4d44-a364-d8c870230518

Added a signature

Crash Signature: [@ gtk_printer_get_name] → [@ gtk_printer_get_name] [@ g_type_check_instance_cast]

There's several old distros for which we don't have symbols which also hit this issue, adding them so we can track the volume.

Crash Signature: [@ gtk_printer_get_name] [@ g_type_check_instance_cast] → [@ gtk_printer_get_name] [@ g_type_check_instance_cast] [@ libgtk-3.so.0@0x3ad666] [@ libgtk-3.so.0@0x3c80cd] [@ libgtk-3.so.0@0x3ad8c6]

Slowly increasing crash rate over the last 6 months. Strongly correlated with day-of-the-week.

#2 e5e5 UAF in the last month, 1387 crashes

Flags: needinfo?(jwatt)
Flags: needinfo?(jwatt)

In reply to Jonathan Watt [:jwatt] from comment #1)

So the gtk_enumerate_printers call made via the runable here:
https://searchfox.org/mozilla-central/rev/6cc48251bb97600fdf11a5b4c5f621bfc8606d55/widget/gtk/nsDeviceContextSpecG.cpp#266

Here's a more current searchfox link; though this particular snippet is unchanged since comment 1 was posted.
https://searchfox.org/mozilla-central/rev/16526d690cccc9c60cacf09cc08e2c3041ef884e/widget/gtk/nsDeviceContextSpecG.cpp#357-360

  NS_DispatchToCurrentThread(
      NewRunnableMethod("nsDeviceContextSpecGTK::EnumeratePrinters", this,
                        &nsDeviceContextSpecGTK::EnumeratePrinters));
}

I'm noticing one red flag here... We're passing a raw this pointer in to a runnable there which gets executed asynchronously, and I'm not sure we have any guarantee that this will still be alive when the runnable gets executed. (this here is a nsDeviceContextSpecGTK which is a refcounted object, owned by the print data for the particular print job, I think).

It seems conceivable that if the print operation fails or is canceled and the nsDeviceContextSpecGTK is torn down before the runnable can be executed, then that would trivially trigger a UAF here, when the runnable is executed.

[EDIT: This actually shouldn't be a problem; see comment 20. NewRunnableMethod takes this into consideration and generates a runnable that maintains a strong reference to the passed-in this pointer.]

Whoops, never mind -- the thing I'm focused on in comment 15 up through this one isn't an issue after all. The NewRunnableMethod utility-code that we're using here does actually maintain a strong refcounted reference to the passed-in object:
https://searchfox.org/mozilla-central/rev/16526d690cccc9c60cacf09cc08e2c3041ef884e/xpcom/threads/nsThreadUtils.h#1352,1358

//  - MyClass must define AddRef and Release methods
...
// The created runnable will take a strong reference to `myObject`.

(I was initially confused since there's another utility-function of the same name here which is documented as being for non-reference-counted objects. But that one's namespaced inside of a class and definitely isn't what we're using here.)

There's a chance this is related to https://gitlab.gnome.org/GNOME/gtk/-/issues/4672 which emilio filed in response to bug 1750768.

In that gitlab issue, emilio described the issue as "a null deref / passing a null name to gtk_printer_get_name from find_printer." Though probably that was mistaken due to our crash reporter mistakenly having shown null instead of e5e5e5 per comment 7, and really it was an instance of this issue (though it still might be an upstream bug).

In any case I agree with emilio's assessment on that bug, that this seems to be a UAF in GTK code rather than in Firefox code.

Here's a recent crash report, so we have a concrete backtrace to reason about:
bp-1b8c13ce-919c-47b9-b49e-3a2290231206

Top 10 frames of crashing thread:

0  libgtk-3.so.0  gtk_printer_get_name  gtk/gtkprinter.c:436
1  libprintbackend-cups.so  find_printer  modules/printbackends/cups/gtkprintbackendcups.c:1874
2  libglib-2.0.so.0  g_list_find_custom  glib/glist.c:922
3  libprintbackend-cups.so  cups_request_printer_list_cb  modules/printbackends/cups/gtkprintbackendcups.c:3630
4  libprintbackend-cups.so  cups_dispatch_watch_dispatch  modules/printbackends/cups/gtkprintbackendcups.c:1581
5  libglib-2.0.so.0  g_main_dispatch  glib/gmain.c:3309
5  libglib-2.0.so.0  g_main_context_dispatch  glib/gmain.c:3974
6  libglib-2.0.so.0  g_main_context_iterate  glib/gmain.c:4047
7  libglib-2.0.so.0  g_main_loop_run  glib/gmain.c:4241
8  libgtk-3.so.0  gtk_enumerate_printers  gtk/gtkprinter.c:1285

I'm using the Linux Mint source on github as a reference for this GTK/cups code, since it's one of the first easily-linkable copies that I could find that has many/most of these source files (though line numbers might not match up exactly due to version differences).

The innermost function in the backtrace is https://github.com/linuxmint/gtk/blob/158a2b0e1e8d582bc041acc7fe323922747d7787/gtk/gtkprinter.c#L434-L436

const gchar *
gtk_printer_get_name (GtkPrinter *printer)
{
  g_return_val_if_fail (GTK_IS_PRINTER (printer), NULL);

  return printer->priv->name;
}

The only object that could conceivably cause a UAF there is the GtkPrinter *printer parameter itself, or possibly its priv member (i.e. if printer-priv is pointing at an object that's been deleted and filled with garbage).

Going up a few stack levels, this printer arg seems to come from a linked-list, removed_printer_checklist, that CUPS code gets and iterates-across in cups_request_printer_list_cb.

That list gets assigned here:

removed_printer_checklist = gtk_print_backend_get_printer_list (backend);

https://github.com/linuxmint/gtk/blob/158a2b0e1e8d582bc041acc7fe323922747d7787/modules/printbackends/cups/gtkprintbackendcups.c#L3244C1-L3248C76

...and then cups iterates over it here (and we crash during this iteration; note g_list_find_custom in the backtrace, calling the passed-in function pointer find_printer):

      /* remove name from checklist if it was found */
      node = g_list_find_custom (removed_printer_checklist,
				 info->printer_name,
				 (GCompareFunc) find_printer);
      removed_printer_checklist = g_list_delete_link (removed_printer_checklist,
						      node);

https://github.com/linuxmint/gtk/blob/158a2b0e1e8d582bc041acc7fe323922747d7787/modules/printbackends/cups/gtkprintbackendcups.c#L3326-L3331

That iteration does seem to be deleting things as it goes. So a UAF here would suggest that there's a cycle in that list, or perhaps two printer objects that somehow share the same priv pointer (if printer->priv is the deleted thing rather than printer). Or just an entry in that list that's got a bogus pointer-to-deleted-data for other reasons.

Anyway: Gecko doesn't manage the lifetime of any of these objects, so I think it's pretty clear that the UAF here is a cups or gtk bug, and not a Gecko bug.

Nice work, thanks, Daniel!!

(In reply to Daniel Holbert [:dholbert] from comment #22)

I'm using the Linux Mint source on github as a reference for this GTK/cups code, since it's one of the first easily-linkable copies that I could find that has many/most of these source files (though line numbers might not match up exactly due to version differences).

FWIW if we were to know which version of gtk is in use, the upstream gtk repo allows you to find the source by tag. For example, on Fedora 38 the current gtk4 package that's installed is 4.10.5:

https://github.com/GNOME/gtk/blob/4.10.5/gtk/gtkprinter.c

Flags: needinfo?(jwatt)
Flags: needinfo?(jwatt)

I should be able to figure out the GTK versions by looking at the proc maps in a few minidumps.

Good news, I just repro'd locally. I'll try to capture in rr and submit to pernosco later on today.

This seems to require opening multiple print-preview dialogs at once (which normally isn't something that can happen, but apparently is possible if you get stuck waiting on a response from a remote cups server and you hit Ctrl+P again while the dialog is waiting)

My STR, using Ubuntu 22.04:
(1) Set up a machine as a CUPS print server to share its printers:

sudo cupsctl --share-printers

Not sure if it's necessary, but I also went to Ubuntu print-settings, Additional Printer Settings, "Add", and then chose "Generic CUPS-BRF" as the printer type and then chose "Generic" and "Postscript" at the driver prompts. This just gave me a shared printer that doesn't actually spit out paper, which is nice.

The rest of my STR happens on my laptop (a second Ubuntu 22.04 machine):
(2) Configure laptop to use the above machine as its CUPS print server, by creating $HOME/.cups/client.conf with contents ServerName IP-ADDRESS-OF-SERVER
(3) Throttle the laptop bandwidth using wondershaper: sudo apt install wondershaper and then sudo wondershaper [interface-name] 200 200 (Here, interface-name comes from e.g. ifconfig)
(4) Start Firefox (e.g. at about:home; the content doesn't matter).
(5) Hold Ctrl and rapidly press P several times.
This will spawn several layers of stacked print dialogs (you can't really tell, except that the shadow-border gets a little darker as they stack. You'll uncover them in subsequent steps). The foreground one will probably show a spinner for a while.
(6) (optional) at this point you can remove the wondershaper throttling, sudo wondershaper clear [interface-name])
(7) When the print preview is ready to interact with, click "Print".
(8) After a second, you'll find yourself looking at another print-preview dialog (the one that was covered up). Hit "Print" on that one as well.

ACTUAL RESULTS:
After I hit Print on that second (initially-covered-up) dialog in step 8, it says "Printing" for a few seconds and then I crash.

I added some markers in the pernosco trace, as guideposts.

So it looks like one bit of interestingness here is that gtk_enumerate_printers spins the event loop (g_main_loop_run) -- that's shown in the crash reports here as well. That callout happens here:
https://github.com/GNOME/gtk/blob/f5516845d451ddc1286d6c9c0d7a5a8ce1dd8bc3/gtk/gtkprinter.c#L1336-L1339

That means we end up in a reentrant event-loop, nested two levels deep in nsThread::ProcessNextEvent (shown in pernosco, but not shown in our crash report backtrace for some reason; I'm not sure why).

During that event loop, the GTK printer data gets destroyed, after (because?) we find the printer that we're looking for -- i.e. we get a callback to nsDeviceContextSpecGTK::PrinterEnumerator from which we return TRUE to indicate "yes, that's the one". And then the GTK printer objects get freed (gtk_print_backend_destroy), and then we unwind the stack up to a stack level that's just reported as
/usr/lib/x86_64-linux-gnu/gtk-3.0/3.0.0/printbackends/libprintbackend-cups.so+[offset]
...and we crash before exiting that stack level, inside of a call to g_list_find_custom (as can be seen in the crash report backtraces).

So it sort of looks like in our reentrant event loop, CUPS is getting confused about whether or not we're enumerating. I'm not yet clear if it's doing a separate enumeration over the same data, or failing to notice that our enumeration completed, or what.

I probably need to try to re-record with more symbols packages installed, or possibly with gtk/cups sources available if I can figure out how to wire that up. It might be a few days before I can do that, though -- I have to context-switch to some other tasks at the moment.

Flags: needinfo?(jwatt) → needinfo?(dholbert)

Note, there are two distinct usability issues in comment 26 that we might want to spin off here, once we're comfortable doing so:

  • Issue #1: the print dialog takes a long time to populate, if you have CUPS configured to use a remote cups server ($HOME/.cups/client.conf with contents ServerName IP-ADDRESS-OF-SERVER), particularly if your network traffic to that server is slow (e.g. via wondershaper throttling in my STR, or just bad network conditions).

  • Issue #2: when that happens (and maybe other times too?), we allow ourselves to spawn multiple print-preview dialogs for the same page. (Possibly just because the dialog is blocked on gtk_enumerate_printers which spins the event loop when it's waiting, as discussed above).

Some thoughts on these:
I wonder if Issue #1 might be related to some of the print hangs we've seen on other bugs (often folks tell us they're using a network printer, but I'm not sure if we've considered/asked whether they're using a remote cups server as well). In any case, we need to fix things to account for the possibility that cups can be arbitrarily slow-to-respond.

I imagine/hope we can fix Issue #2 in a relatively straightforward way by adding some sort of RAII guard (or JS equivalent) when the print dialog opens, to prevent further dialogs from opening in nested event loops. This is worth doing for its own sake, and there's a chance that might avoid this crash entirely.

One other quick note, https://chromium.googlesource.com/chromium/src/+/main/docs/linux/debugging_gtk.md has some info on how Chromium folks build gtk for local debugging -- I might doing try that for the sake of a more-understandable pernosco trace.

The "multiple superimposed print dialogs" thing doesn't seem to be required, actually; I was just able to trigger this crash with two entirely separate print operations (i.e. my comment 26 STR, but without the "rapidly press ctrl+p" thing; just printing once with ctrl+p, completing that operation, and then printing again with ctrl+p).

I also was able to build libgtk locally, with debug info/symbols, and run Firefox with that locally-built library. For whatever reason, my locally-built library doesn't trigger the crash -- but at the moment when I'd expect a crash, I do get a GTK nonfatal assertion-failure that seems to be a symptom of the same trouble:

Gtk-CRITICAL **: 11:41:26.372: gtk_printer_get_name: assertion 'GTK_IS_PRINTER (printer)' failed

That's probably a signal that we're passing a bogus object of some sort into gtk_printer_get_name, and for whatever reason it hasn't been poisoned yet, but it's still a landmine.

Here's a pernosco trace where we hit that gtk assertion failure, with my local debug build of gtk (so hopefully fairly-debuggable), and with GTK_DEBUG=printing to add various bits of logging around gtk printing internals:
https://pernos.co/debug/DUwm21zoKTPK5aRy37dSfg/index.html#f{m[A0G2,GcCZ_,t[KA,ENHV_,f{e[A0G2,GcCZ_,s{af6RSwAAA,bGw,uvBEL,ovHil___,v[{wiLg,v[{f'list',q'stdouterr',p{_,xAYag_,{f'container',q'stack',p{_,xAYag___,{w/eg,v[{f'source',q'source',p{_,xAYag_,{f'list',q'execution',p{'symbol''gtk_printer_get_name'_,xAYag____/

Looking at that pernosco trace, I'm more confident that this is a bug in GTK (specifically in gtkprintbackendcups.c)

The short version is that we're in some gtk/cups code that updates a collection of printers, while also maintaining a separate list of those printers; and one of the updates triggers the whole collection to be deleted (through a bit of indirection), and then we go to check our separate list and we crash.

Here's roughly what happens:

Flags: needinfo?(dholbert)

Unfortunately a lot of this GTK code has git-blame dating back ~17 years or so, so for better or for worse, this may have been a problem for quite a while.

Coincidentally, our doomed removed_printer_checklist search (the last bullet in my previous-comment explanation) seems to have been last touched in a commit from 10 years ago titled Revert "Fix gtkprintbackendcups crash": https://github.com/GNOME/gtk/commit/9d36dbb44ba14adeb29135de01a936eb623375f2

But the crash referenced there seems to have been something different from the crash that we're hitting here... maybe. (The reverted patch was a null-check for removed_printer_checklist which wouldn't help in this case.)

I'm pretty sure it's possible to write a small GTK program that triggers this same issue, to assist in reporting upstream and seeing if the GTK folks can do anything about this. Based on how ancient the code is and how fiddly printer things are, I'm not entirely sure a fix will be trivial/forthcoming.

I have an untested idea for a workaround on our end -- I think we can avoid around this by changing our comparison callback to pretend it never finds the thing it's looking for, i.e. make nsDeviceContextSpecGTK::PrinterEnumerator unconditionally return FALSE. That should prevent list_added_cb from calling stop_enumeration and deleting data out from under the cups_request_printer_list_cb loop. It's possible there'll be a perf cost (even in the general weren't-gonna-crash case) to forcing ourselves to traverse the whole printer list, but hopefully it's minimal...

Assignee: nobody → dholbert
Status: NEW → ASSIGNED
Attachment #9367750 - Attachment description: Bug 1660223: Let printer enumeration run to completion, to avoid a GTK bug. r?jwatt → Bug 1660223: Let printer enumeration run to completion, to avoid a GTK bug. r?jwatt,AlaskanEmily

Workaround posted. Early next week, I'll also try to write a small demo GTK program that triggers the issue, to assist with reporting upstream so the GTK folks can maybe/hopefully try to fix the root issue here, so that we don't have to work around it.

This is a great idea to solve this issue, thanks for taking the time to look into this.

BTW, the reason we're doing this printer enumeration in the first place is explained in bug 1090448 comment 0 (though things have changed somewhat since then, I think).

(Basically, we've got a printer name stored as part of the print settings, but not an actual GtkPrinter* handle. To get that handle, we have to enumerate the printers to find the one with the matching name.)

Pushed by dholbert@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/833e0bfa6d81 Let printer enumeration run to completion, to avoid a GTK bug. r=jwatt

https://hg.mozilla.org/mozilla-central/rev/833e0bfa6d81

Group: layout-core-security → core-security-release
Status: ASSIGNED → RESOLVED
Closed: 1 year ago
status-firefox122: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 122 Branch
Duplicate of this bug: 1750768

Good news: in bug 1750768 comment 14, we have confirmation from a user that the patch (landed in comment 37) has addressed the crash.

(I've now marked that bug as a dupe of this one.)

Here's a minimal gtk testcase that triggers the underlying GTK bug.

As noted in the comment at the top of this C++ source code: one of the requirements to trigger the issue with this testcase is the environmental variable G_DEBUG=fatal-criticals. Without that, gtk_printer_get_name's graceful GTK_IS_PRINTER assertion can handle this gracefully pretty reliably and will return when it detects that it's been given a garbage-object rather than something that it can recognize as a GtkPrinter.

The G_DEBUG=fatal-criticals environmental variable makes that assertion-failure fatal, though (which is useful, since in this case it's a sign of UAF.)

Attachment #9368606 - Attachment mime type: text/x-c++src → text/plain

I've just reported this to the Gnome folks at https://security.gnome.org (just a web form; if a ticket results from it, I'll post a link to the ticket here). They might just bump https://gitlab.gnome.org/GNOME/gtk/-/issues/4672 to be sec-sensitive, too; I mentioned in my form-submission that this is essentially that exact bug, which we now know to be a UAF rather than a null-deref.

(In reply to Randell Jesup [:jesup] (needinfo me) from comment #43)

Would this be part of the same issue? https://crash-stats.mozilla.org/signature/?address=~e5e5e&product=Firefox&version=122.0a1&version=121.0&version=121.0b&version=120.0.1&version=120.0&date=%3E%3D2023-11-15T02%3A34%3A00.000Z&date=%3C2023-12-15T02%3A34%3A00.000Z&_sort=-date&signature=libgtk-3.so.0%400x3c80cd

Yes, that's for libgtk-3.so.0@0x3c80cd -- it looks like we've already got that tracked as a known one of the crash signatures here (added in comment 13).

Double-checking though: the most recent crash there, bp-369737f8-0648-4b03-a5ab-96c290231215 , seems to be exactly this -- it's got symbols for nearly all stack levels except the level 0 and level 21. Comparing the the stack levels that it does have against the crash report that I linked in comment 22, they match up pretty closely.

(In reply to Randell Jesup [:jesup] (needinfo me) from comment #44)

and https://crash-stats.mozilla.org/signature/?address=~e5e5e&product=Firefox&version=122.0a1&version=121.0&version=121.0b&version=120.0.1&version=120.0&signature=libgtk-3.so.0%400x39bcf6&date=%3E%3D2023-11-15T02%3A34%3A00.000Z&date=%3C2023-12-15T02%3A34%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_columns=startup_crash&_sort=-date&page=1

Very likely yes. Hard to say for certain there, since I'm not seeing any crashes that have symbols for any of the non-Gecko stacktrace levels, but the fact that we're in a RunnableMethodImpl<nsDeviceContextSpecGTK*, ...> and the pattern of libgtk/libglib/libprintbackend-cups below that are a pretty strong indicator that it's this same crash.

--> Tentatively adding the crash signature from comment 44.

Crash Signature: [@ gtk_printer_get_name] [@ g_type_check_instance_cast] [@ libgtk-3.so.0@0x3ad666] [@ libgtk-3.so.0@0x3c80cd] [@ libgtk-3.so.0@0x3ad8c6] → [@ gtk_printer_get_name] [@ g_type_check_instance_cast] [@ libgtk-3.so.0@0x3ad666] [@ libgtk-3.so.0@0x3c80cd] [@ libgtk-3.so.0@0x3ad8c6] [@ libgtk-3.so.0@0x39bcf6]
Attachment #9369132 - Flags: approval-mozilla-esr115?

Uplift Approval Request

  • Risk associated with taking this patch: Low.
  • String changes made/needed: None
  • Is Android affected?: no
  • Code covered by automated testing: no
  • Steps to reproduce for manual QE testing: See comment 26 on the bug ( https://bugzilla.mozilla.org/show_bug.cgi?id=1660223#c26 )
  • Explanation of risk level: The patch just has us complete a walk through the list of printers (which we might have to do in general), rather than interrupting the walk partway through. In many cases, we already would have been doing a walthrough of the full list of printers, so this isn't a new behavior. See end of comment 32.
  • Fix verified in Nightly: yes
  • Needs manual QE test: no
  • User impact if declined: Crashes when printing in certain configurations

(In reply to Daniel Holbert [:dholbert] from comment #42)

I've just reported this to the Gnome folks at https://security.gnome.org (just a web form; if a ticket results from it, I'll post a link to the ticket here).

My report spawned a Gnome security-bug at https://gitlab.gnome.org/GNOME/gtk/-/issues/6265 (which shows up as 404 since it's hidden to the public). It's essentially a dupe of their public bug, but they're keeping them separate for now since 4672 is already public.

I'm also adding a whiteboard note here to remind ourselves to keep this bug hidden until those bugs are fixed/openable (particularly when 6265 can be made public.) Don't want to zero-day the gnome folks with the details discussed here.

See Also: → https://gitlab.gnome.org/GNOME/gtk/-/issues/6265
Whiteboard: [print2020][old-ui?][tbird crash] → [keep hidden until linked GTK bugs are public/fixed][print2020][old-ui?][tbird crash]
QA Whiteboard: [post-critsmash-triage]
Flags: qe-verify+
QA Whiteboard: [post-critsmash-triage] → [post-critsmash-triage] [qa-triaged]

I have attempted to perform the steps in comment 26 to trigger this crash, but I am unable due to various reasons: the technical knowledge needed to properly complete them is quite high for my level (creating client.conf file with specific content and server names, throttling bandwidth, creating and sharing printers) and 2 different Ubuntu systems are required to complete the steps. This being said I cannot reproduce this issue.

Thanks for trying! I think I can do verification here, since I've got a setup that reproduces the crash.

Which builds are we trying to verify as good/bad at this stage? Just nightlies before/after the patch landed? Or current release 121 vs. current beta 122?

Flags: needinfo?(dbodea)

(2) Configure laptop to use the above machine as its CUPS print server, by creating $HOME/.cups/client.conf with contents ServerName IP-ADDRESS-OF-SERVER

Side note, partly for my own reference later: instead of this client.conf thing, you can also just run the following in the terminal (importantly, the same terminal that you launch Firefox from):
export CUPS_SERVER=10.0.30.5
(where 10.0.30.5 is just an example; that's meant to be the IP address of the other machine that you're using as a remote CUPS server.)

Right now at least, in my test environment (using my laptop, with desktop as CUPS server via CUPS_SERVER environmental variable as noted in previous comment), I'm hitting this crash immediately on the first print operation, in builds known-to-be affected by the bug, without even needing the wondershaper bandwidth-throttling or repeated prints described in comment 26.

NIGHTLY BEFORE/AFTER THE FIX:
Reproduced the bug in Nightly 122.0a1 (2023-12-12). Crash report: bp-48910ebe-a33d-4180-90ab-38dcd0231222
Unable to reproduce in Nightly 122.0a1 (2023-12-14)

RELEASE 121 VS BETA 122:
Reproduced the bug in Firefox 121.0 (64-bit). Crash report: bp-bc0a3026-0fc9-4a41-8de1-6fdf70231222
Unable to reproduce in Firefox 122.0b3

NOTES:
In these comparisons, the reproducing builds (121. and the 2023-12-12 nightly) reproduced the crash for me on my very first attempt to print.
The not-reproducing builds did not; I tried again several times, including the measures in comment 26 (wondershaper, repeated print operations with inadvertently-stacked print dialogs), and was still unable to repro, so I'm pretty confident that the not-repreoducing in these builds is "real" rather than just luck.

So I'm happy calling this "verified fixed" in Nightly (at least in 2023-12-14 and newer), and "verified fixed" in 122.0b3

danibodea, let me know if there are any other comparisons/verification that we need here. I'll be happy to restest with ESR builds once the fix gets uplifted there.

Thanks!

Status: RESOLVED → VERIFIED
status-firefox122: fixed → verified
Attachment #9369132 - Flags: approval-mozilla-esr115? → approval-mozilla-esr115+

(In reply to Daniel Holbert [:dholbert] from comment #51)

Thanks for trying! I think I can do verification here, since I've got a setup that reproduces the crash.

Which builds are we trying to verify as good/bad at this stage? Just nightlies before/after the patch landed? Or current release 121 vs. current beta 122?

I'm sorry for the late reply. I normally attempt to reproduce in builds immediately before the fix and then attempt verification in the latest versions of respective channels. Thank you for the verification! I'll keep this bug in a backlog and remind you to verify ESR channel as well, when it lands, however, it's worth knowing that verification in a treeherder build with the fix is also acceptable.

Flags: needinfo?(dbodea)

Thanks! Looks like this hit ESR already, actually (on treeherder at least). I'll do verification for the before/after builds on TreeHerder for that tree, using Linux x64 Shippable row, "Bs" task, choosing the "target.tar.bz2" artifact.

Unpatched: https://treeherder.mozilla.org/jobs?repo=mozilla-esr115&searchStr=linux%2C64%2Cshippable%2CBs&revision=0f158c2767db4d0e5f486c70fa83ba6cec4f1287&selectedTaskRun=S24v2lXbSGKEDckUXvxRPg.0
Patched: https://treeherder.mozilla.org/jobs?repo=mozilla-esr115&searchStr=linux%2C64%2Cshippable%2CBs&revision=0168b085c45b39f0f2ddb3d671e2e3a6f6631ff6&selectedTaskRun=RWNWoLcUSkOLMqll4HpWCg.0

As in comment 53, the "Unpatched" build here crashed on the first print operation when running in my test environment with CUPS_SERVER set to use my remote print server. Crash reports:
bp-891c047c-ecdb-4965-92c6-142420240106
bp-5bd4c0d5-7f3e-4948-bcf3-0c0b90240106

And I was unable to get the "Patched" build to crash, after a few minutes of similar print jobs that crashed the unpatched build (and trying with wondershaper as well for good measure).

--> Verified on ESR115 with treeherder builds.

status-firefox-esr115: fixed → verified

Thank you for the help!

Flags: qe-verify+
Whiteboard: [keep hidden until linked GTK bugs are public/fixed][print2020][old-ui?][tbird crash] → [keep hidden until linked GTK bugs are public/fixed][print2020][old-ui?][tbird crash][adv-main122+][adv-esr115.7+]

Note, the associated [@ g_type_check_instance_cast ] crash signature happens to have two new crash reports today (here and here, from Nightly 123.0a1 -- but they're not related to this bug (nothing printing-related in the backtrace). I filed bug 1875370 on those.

Disregarding those two crashes discussed in comment 59, we've still got zero crash volume for any Firefox versions beyond 121 (i.e. any versions with the fix), for any of the signatures associated with this bug.

Alias: CVE-2024-0746

FWIW the associated GTK bugs here are unhidden, so I'm not sure we need to keep this closed anymore.

The two GTK bug reports were
https://gitlab.gnome.org/GNOME/gtk/-/issues/4672
https://gitlab.gnome.org/GNOME/gtk/-/issues/6265

The older one was always open (it was filed before we realized there was a sec issue or knew what was going on), and has since been duped to the newer one.
The newer one has been fixed and was made "visible to everyone" 3 months ago [March 13th]

dveditz, I think we're OK to un-hide this if you agree.

Flags: needinfo?(dveditz)

Thanks

Group: core-security-release
Flags: needinfo?(dveditz)
Keywords: keep-hidden
Whiteboard: [keep hidden until linked GTK bugs are public/fixed][print2020][old-ui?][tbird crash][adv-main122+][adv-esr115.7+] → [print2020][old-ui?][tbird crash][adv-main122+][adv-esr115.7+]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: