At the moment it is possible to create a new window via messenger.windows.create() and then wait for the window to be closed - like it is done in the awaitPopup Example. But unfortunately the user can still interact with the message compose window in the meantime.
A locking mechanism, which locks the compose window completely, is also needed for Thunderbird itself in various places - see Bug 1758469 and Bug 1758470. Making this mechanism available via a WebExtension API, e.g. messenger.compose.lock(tab) and messenger.compose.unlock(tab), would be very welcome and should be easy to implement.
A viable alternative - possibly even better than modal dialogs - could be to create a "modal" overlay, in which an iframe with the popup.html of the add-on can be loaded into, e.g. via messenger.compose.overlay.create(). (Like it is done for the new print dialog.)
The additional benefit of creating an overlay compared to opening a new window or tab is, that the connection - from a user perspective - between the compose window and the popup is very strong in case of an overlay. Of course the overlay should also have the possibility to lock the compose window.
With the two additions - a locking mechanism and a "modal" overlay - a "modal" dialog would no longer be needed for my own add-on Mail Merge.