478073 - We need some changes to Gecko to get modal dialogs working properly on OS X

Status: RESOLVED INCOMPLETE

(Core :: Widget: Cocoa, defect)

Description

[Steven Michaud [:smichaud] (Retired)]

Gecko's implementation of modal dialogs currently doesn't work very
well on OS X.  This is largely because, up til now, we've tried to
make Cocoa widgets' modal dialogs behave exactly as they do on Windows
and Linux.  But modal dialogs behave significantly differently in
"native" Cocoa (or Carbon) apps than they do on Windows or Linux.

We've implemented hacked (ersatz) modal windows in Cocoa widgets that
behave more like those on Windows and Linux (see bug 395465).  But
these have never worked perfectly.  And we've had to continue using
native modal dialogs (like Print, Page Setup, Open File and Save As),
which don't work at all well with Gecko code -- necessitating further
hacks (see bug 436473, bug 442442, bug 468393 and bug 476541).

This strategy is wrong-headed -- it requires code in Cocoa widgets
that's over-complex and error-prone, and still doesn't work quite
correctly.

We need to acknowledge that modal windows don't work the same way on
OS X as they do on Windows and Linux, and to allow Cocoa widgets'
implementation of them to be different.  We also need to make some
changes to Gecko code to allow it to accomodate certain quirks of
Cocoa modal dialogs (particularly of modal windows).

In following comments I'll outline the differences between how modal
dialogs behave on Windows and Linux (on the one hand) and OS X (on the
other).

I'll also describe (in a general way) the changes that are needed in
Gecko code to make its modal dialogs work well on OS X.  These changes
won't be large, and won't require any modifications to how modal
dialogs behave on other platforms (besides OS X).

Comment 1

[Steven Michaud [:smichaud] (Retired)]

I) How Firefox Uses Modal Dialogs on Windows and Linux

On Windows and Linux (in Windows and GTK2 widgets), all modal dialogs
are window-modal windows.  And each top-level window has its own menu
(displayed just below the titlebar).

1) Each modal dialog is a separate window (as opposed to a sheet).

2) Each modal dialog opens above another window (its parent), and (in
   principle) prevents access to the parent until the modal dialog has
   been dismissed.  (On Windows this breaks down if modal windows are
   nested.  But nested modal windows do work correctly on Linux.  I'll
   have more to say about this in a later comment.)

   On neither platform does a modal window stop you from accessing a
   window that's not its parent.  In other words, Gecko never uses
   app-modal dialogs on either Windows or Linux.

3) No modal dialog has a menu.  And none of the parent window's menu
   items are accessible while the modal dialog is open.  (With most
   modal dialogs, even keyboard shortcuts don't work.)

4) Modal dialogs can be nested (I'll have more to say about this in a
   later comment).

II) How "Native" Cocoa and Carbon Apps Use Modal Dialogs on OS X

On OS X (in "native" apps) there are two kinds of modal dialog --
window-modal sheets and app-modal windows.  And there is a single,
app-level menu (displayed at the top of the screen) that's shared by
all windows (though sometimes this menu changes as different windows
get the focus).

A) Window-Modal Sheets (Apple calls them document-modal)

   A sheet is attached to the (top-level) window above which it opens
   (its parent).  While it's open, the sheet prevents most access to
   its parent (though you're still allowed to minimize and zoom the
   parent, and to move it around).

   A sheet doesn't prevent access to windows other than its parent.

   Sheets (like all windows) have access to the shared app-level menu.
   But sometimes parts of the menu are disabled while the sheet has
   the focus.

   Examples of sheets are the Print dialog in Safari and JavaScript
   alerts in Firefox.

   Sheets can be nested, though Firefox currently never allows this,
   and Safari usually prevents it from happening.

   1) In Safari, when you nest sheets, a new window is created and the
      nested sheet gets attached to that.  For an example of this,
      choose File : Print, then choose Safari : Report Bugs to Apple.

      In other cases, though, nothing happens when you try to nest
      sheets.  For example choose File : Save As (which opens a sheet)
      and then File : Print (which normally opens a different sheet)
      -- nothing happens, even after you dismiss the Save As dialog.

   2) In Firefox, when you try to nest a sheet over another sheet, the
      second sheet doesn't get displayed until the first sheet has
      been dismissed.  (I'll have more to say about this in a later
      comment.)

B) App-Modal Windows

   An app-modal dialog is a separate window that prevents most access
   to all the app's open windows (it generally stops you from doing
   anything but moving those windows around).

   An app-modal dialog (like all windows) does have access to the
   shared app-level menu.  But many items in that menu are normally
   disabled when it has the focus.  (For example, for obvious reasons
   it shouldn't be possible to use the menu, or any keyboard shortcut,
   to open new windows while an app-modal dialog is open.)

   Examples of app-modal dialogs are the Open File dialog in Safari
   and Firefox, and JavaScript alerts in Safari.

   App-modal dialogs can be nested.  For an example (in Safari), load
   attachment 351645 [details] (from bug 468393 comment #0), then quickly do
   File : Open.

   App-modal dialogs can be nested over sheets.  For an example (in
   Safari) choose File : Print, then File : Open File.  For obvious
   reasons, sheets can't be nested over app-modal dialogs.

III) Changes Needed To Accommodate These Differences

Currently this list is short (just two items).  It may need to be
expanded.

1) Currently showModalDialog() is implemented using a (more or less)
   window-modal window on OS X (per my hacks in bug 395465), as it is
   on Windows and Linux.

   But in native apps, modal windows (as opposed to sheets) must be
   app-modal on OS X.  So we should either go back to using a sheet
   for showModalDialog(), or accept that it will be app-modal.

2) If a window-modal sheet wants to open above an app-modal dialog, it
   will need either to be suppressed/postponed or opened as an
   app-modal dialog.

Comment 2

[Steven Michaud [:smichaud] (Retired)]

I) Low-level Quirks in Standard OS X Modal Dialogs

Above I mentioned several high-level (and user-visible) differences in
how modal dialogs are implemented in Firefox on Windows and Linux, and
how they're implemented in native apps on OS X.

But there are also a couple of low-level (user-invisible) quirks to
how modal dialogs are implemented on OS X (ones not found on Windows
or Linux) that make it very difficult to use them in Gecko (as it's
currently written):

1) Native app-modal dialogs run their own (nested) event loops.  But
   Gecko (in nsXULWindow::ShowModal()) insists on running its own
   (nested) event loop for the modal dialogs that it spawns.

   This is fine for sheets (which don't run their own event loops).
   But it makes it impossible to run a native app-modal dialog from
   nsXULWindow::ShowModal() (as it's now written).

2) OS X provides a number of canned modal dialogs for common tasks
   (like Open File, Save As, Print and Page Setup).  These are used by
   all standard apps -- which means that (practically speaking)
   they're not optional.  And in fact Firefox currently uses them.

   But none of these is displayed via nsPromptService::Alert() (and
   friends) or nsGlobalWindow::Alert() (and friends).  So there's
   currently no way for Gecko to tell that one of these dialogs is
   being displayed (for example by calling
   nsGlobalWindow::IsInModalState()).  And so Gecko doesn't have the
   option (in nsGlobalWindow::SetTimeoutOrInterval() and
   nsGlobalWindow::RunTimeout()) to postpone showing (i.e. nesting)
   another modal dialog while one of these canned modal dialogs is
   open.

II) Changes Needed to Accomodate These Quirks

1) nsXULWindow::ShowModal() should be able to "delegate" the running
   of its nested event loop.

   In other words, there should be the option to have
   nsXULWindow::ShowModal() call a callback, which runs a "native" (or
   semi-native) event loop.  This callback would be responsible for
   processing both Gecko events (using NS_ProcessNextEvent()) and
   native events.  The callback could (as nsXULWindow::ShowModal())
   now does) break out of the event loop (and return) when
   NS_ProcessNextEvent() returns FALSE.

2) There should be some way to notify Gecko that a native modal dialog
   is running that it didn't spawn in the usual way (i.e. using
   nsGlobalWindow::Alert() and friends).

   There should probably also be some way to tell Gecko what kind of
   modal dialog is running -- either a window-modal dialog running
   above a given parent window or an app-modal dialog running above
   all other windows.

Comment 3

[Steven Michaud [:smichaud] (Retired)]

Nesting of Modal Dialogs

Gecko has code (in nsGlobalWindow.cpp and nsPromptService.cpp) to
postpone a modal dialog opening over a given parent window if another
modal dialog is already open over that parent window.  But this code
currently only works if both dialogs are opened from the same tab.  In
other words, if a window has two tabs and a modal dialog has been
opened from "tab 1", another modal dialog opened from "tab 2" will be
nested over the modal dialog from "tab 1" (it's display won't be
postponed).

This may or may not be a bug.  But even if it's considered a bug, this
behavior probably won't be easy to change.  So I think it makes sense
to ensure that the nesting of modal windows works properly, on all
platforms.  Currently there are problems with modal dialog nesting on
both OS X and Windows.

The following examples show how "modal dialog postponing" works when
both dialogs are spawned from the same tab:

  javascript:setTimeout( function() {alert('second');}, 0); alert('first');
  javascript:setTimeout( function() {showModalDialog('data:text/html,second');}, 0); alert('first');

But try invoking either of the following examples from one tab (each
launches a modal dialog on a 5-second timer):

  attachment 351645 [details] (from bug 468393 comment #0)
  attachment 356070 [details] (from bug 468393 comment #21)

And then (quickly, before 5 seconds are up) either of the following
examples from another tab:

  javascript:alert('first');
  javascript:showModalDialog('data:text/html,first');

On Windows and Linux this will always result in nested modal dialogs
(though you may have to fiddle with your popup blocking settings to
get the showModalDialog() examples to work at all).

On OS X this will usually result in nested modal dialogs.  In the case
where alert() is used in both tabs, though, the "second" alert will
only appear after the "first" has been dismissed.  But this has
nothing to do with Gecko.  Instead it's the result of "magic"
performed in nsCocoaWindow::Show().  (The second alert is a sibling of
the first alert, and nsCocoaWindow::Show() sets mSheetNeedsShow in the
second alert, so that it can be displayed (on a subsequent call to
nsCocoaWindow::Show()) after the first alert has been dismissed.)

Problems Nesting Modal Dialogs on Windows

1) On Linux, if two modal dialogs are nested, you can't get access to
   "lower" dialogs until the topmost one has been dismissed.  This is
   correct.

   But on Windows you can access nested modal dialogs in any order.

Problems Nesting Modal Dialogs on OS X

1) Like on Windows (and unlike on Linux), nested modal windows (as
   opposed to sheets) can be accessed in any order.

2) Severe problems can result if you nest a sheet spawned from
   nsXULWindow::ShowModal() over a native app-modal dialog (for
   example the dialogs Firefox uses for Open File, Save As, Print and
   Page Setup).

   One example is the hang reported at bug 436473 and bug 442442.  I
   worked around this with my (hackish) patch for bug 436473.

   Another example is the "hang" reported at bug 476541.  Another
   (hackish) patch exists for this (at bug 436473 comment #24), but it
   hasn't been landed.

Comment 4

[Boris Zbarsky [:bzbarsky]]

> So we should either go back to using a sheet for showModalDialog(), or accept
> that it will be app-modal.

The problem is that option 1 is not really very web-compatible, while option 2 is just terrible UI...  I just checked, and Safari seems to make it app-modal, so I suppose that's the way to go.  Might still not be quite web-compatible, but if that's the case we'll find out.